My OpenIndiana machine recently emailed me the following error message, with the subject line Cron <root@DellOptiPlex390MT> [ -x /usr/lib/fs/nfs/nfsfind ] && /usr/lib/fs/nfs/nfsfind:

find: cannot open /znapzend/DellOptiPlex390MT/ROOT/openindiana: No such file or directory
find: cannot open /znapzend/DellOptiPlex390MT/export/home/jdrch: No such file or directory
find: cannot open /znapzend/DellOptiPlex390MT/export: No such file or directory
find: cannot open /znapzend/DellOptiPlex390MT/export/home: No such file or directory
find: cannot open /znapzend/DellOptiPlex390MT/ROOT: No such file or directory

Here's how I fixed it

1st, let's interpret the error message. Each line is saying that the find command cannot open a certain path, because there is no file or directory at that path. Both those paths exist, so how can find not locate them? More on that later.

The Cron <root@DellOptiPlex390MT>email subject line tells us that this error message is coming from the root crontab (or, more accurately, the cron daemon running as root), and that it's occurring at the -x /usr/lib/fs/nfs/nfsfind ] && /usr/lib/fs/nfs/nfsfind line.

Let's look at the root crontab to see if we can find a matching line. Sure enough, there it is:

15 3 * * 0 [ -x /usr/lib/fs/nfs/nfsfind ] && /usr/lib/fs/nfs/nfsfind

The above line means "at 0315 every Sunday, run -x /usr/lib/fs/nfs/nfsfind ] && /usr/lib/fs/nfs/nfsfind." So what is nfsfind? You can find the full description in the Solaris docs (note that the svc:/network/nfs/cleanup and associated details do not apply to Illumos). nfsfind cleans stale temporary files out of your NFS shares once a week, presumably to prevent the shared filesystems from running out of space.

Now that we know what nfsfind does, let's take a look at it using our editor of choice. I prefer nano, invoked here under my own user account as I do not want to accidentally edit a system script:

$ nano /usr/lib/fs/nfs/nfsfind

if [ ! -s /etc/dfs/sharetab ]; then exit ; fi

# Get all NFS filesystems exported with read-write permission.

DIRS=`/usr/bin/nawk '($3 != "nfs") { next }
        ($4 ~ /^rw$|^rw,|^rw=|,rw,|,rw=|,rw$/) { print $1; next }
        ($4 !~ /^ro$|^ro,|^ro=|,ro,|,ro=|,ro$/) { print $1 }' /etc/dfs/sharetab`

for dir in $DIRS
do
        find $dir -type f -name .nfs\* -mtime +7 -mount -exec rm -f {} \;
done

The penultimate line of the nfsfind script has the script's only find command. By process of elimination, this would be where the error message is coming from. It's safe to assume find isn't malfunctioning and its options are syntactically correct, so the error message is probably showing up because it's being fed the wrong input ($dir).

find $dir tells us the find command is operating on a variable $dir, which from the for dir in $DIRS line is each successive value in $DIRS. From the DIRS= line we see that DIRS comes from whatever is found in /etc/dfs/sharetab.

Let's look at /etc/dfs/sharetab, invoking nano with the same privileges as before:

$ nano /etc/dfs/sharetab

/znapzend/DellOptiPlex390MT/ROOT/openindiana    -       nfs     sec=sys,rw=@192.168.0.107/32,root=@192.168.0.107/32
/znapzend/DellOptiPlex390MT/export/home/jdrch   -       nfs     sec=sys,rw=@192.168.0.107/32,root=@192.168.0.107/32
/znapzend/DellOptiPlex390MT     -       nfs     sec=sys,rw=@192.168.0.107/32,root=@192.168.0.107/32
/rpool1 -       nfs     sec=sys,rw=@192.168.0.107/32,root=@192.168.0.107/32
/znapzend       -       nfs     sec=sys,rw=@192.168.0.107/32,root=@192.168.0.107/32
/znapzend/DellOptiPlex390MT/export      -       nfs     sec=sys,rw=@192.168.0.107/32,root=@192.168.0.107/32
/znapzend/DellOptiPlex390MT/export/home -       nfs     sec=sys,rw=@192.168.0.107/32,root=@192.168.0.107/32
/znapzend/DellOptiPlex390MT/ROOT        -       nfs     sec=sys,rw=@192.168.0.107/32,root=@192.168.0.107/32

Now, some background:

• /znapzend is the mountpoint for rpool1/znapzend/, a ZFS fiesystem I created as a destination for znapzend
• rpool1 itself is mounted at /rpool1. I shared it via NFS using # zfs set sharenfs=on long before I created rpool1/znapzend

Clearly, all of rpool1's child datasets inherited its sharenfs=on property upon their creation. Besides that, I had mounted rpool1/znapzend at /znapzend, which resulted in all rpool1/znapzend's child datasets being mounted as described below:

$ mount | grep znapzend
/znapzend on rpool1/znapzend read/write/setuid/devices/nonbmand/exec/xattr/atime/dev=4c10008 on Fri Apr 30 21:59:04 2021
/znapzend/DellOptiPlex390MT on rpool1/znapzend/DellOptiPlex390MT read/write/setuid/devices/nonbmand/exec/xattr/atime/dev=4c10034 on Sat May  1 10:00:03 2021
/znapzend/DellOptiPlex390MT/ROOT on rpool1/znapzend/DellOptiPlex390MT/ROOT read/write/setuid/devices/nonbmand/exec/xattr/atime/dev=4c1003a on Sat May  1 10:00:04 2021
/znapzend/DellOptiPlex390MT/ROOT/openindiana on rpool1/znapzend/DellOptiPlex390MT/ROOT/openindiana read/write/setuid/devices/nonbmand/exec/xattr/atime/dev=4c1003c on Sat May  1 10:03:00 2021
/znapzend/DellOptiPlex390MT/export on rpool1/znapzend/DellOptiPlex390MT/export read/write/setuid/devices/nonbmand/exec/xattr/atime/dev=4c1003d on Sat May  1 10:03:22 2021
/znapzend/DellOptiPlex390MT/export/home on rpool1/znapzend/DellOptiPlex390MT/export/home read/write/setuid/devices/nonbmand/exec/xattr/atime/dev=4c10040 on Sat May  1 10:03:33 2021
/znapzend/DellOptiPlex390MT/export/home/jdrch on rpool1/znapzend/DellOptiPlex390MT/export/home/jdrch read/write/setuid/devices/nonbmand/exec/xattr/atime/dev=4c10042 on Sat May  1 10:03:48 2021

It seems I made 2 mistakes here:

1. I forgot to # zfs set sharenfs=off the child datasets
2. I probably unnecessarily mounted the child datasets. As you can see from my znapzend tutorial, znapzend uses ZFS zpool/dataset paths, not filesystem paths (created by mount operations)

(As a corollary, this is probably why ZFS uses the term dataset and not filesystem. All ZFS filesystems are datasets, but not all ZFS datasets are filesystems. A dataset becomes a filesystem only when it is mounted.)

But that still doesn't explain why find chokes on those paths. Let's try to navigate to them ourselves using cd:

# cd /znapzend/DellOptiPlex390MT/ROOT/openindiana
-bash: cd: /znapzend/DellOptiPlex390MT/ROOT/openindiana: No such file or directory

Wait, what? How can there be no file or directory at that path? The answer lies in the sequence of sequence of events that led to that location being considered a filesystem (note the emphasis) path to begin with. 1st, rpool1 was created with sharenfs=on. Much later rpool1/znapzend and rpool1/znapzend/DellOptiPlex390MT were created. Both those datasets inherited the sharenfs=on.

rpool1/znapzend was then mounted at /znapzend, which also mounted all of its current and future datasets. All of the above became filesystems by virtue of being mounted and NFS shares by virtue of inheriting their parent dataset(s)' sharenfs=on setting.

The future datasets came into being when znapzend created them recursively as zfs receive destinations. However, because each dataset is actually a family of snapshots, each has no actual corresponding filesystem, despite the apparent path! This is why # cd fails to find anything.

We can fix this problem by 1st unsharing the "problematic" dataset (actually this alone is sufficient to solve the problem):

# zfs set sharenfs=off rpool1/znapzend

and then also unmounting it (good practice, since the location isn't intended to be generally accessible to non-ZFS operations anyway):

# zfs unmount rpool1/znapzend

For those who may be confused about the continued accessibility of the destination dataset to znapzend after unmount, remember that all datasets on a zpool are accessible via ZFS (note the emphasis) commands as long as that zpool has been imported (automatic on OpenIndiana and FreeBSD for zpools created on the same machine) and has not been exported.

Checking the contents of /etc/dfs/sharetab again:

$ nano /etc/dfs/sharetab

gives:

/rpool1 -       nfs     sec=sys,rw=@192.168.0.107/32,root=@192.168.0.107/32

Both find and 'cdwork on/rpool1`, so we can be sure there will be no further errors of the kind detailed at the outset.

Thanks to Joshua M. Clulow & Alan Coopersmith on the openindiana-discuss mailing list for helping me resolve this problem.