Re: [nfs-utils PATCH] exports: Fix referrals when --enable-junction=no

[Scott Mayhew <smayhew@redhat.com>]

On Mon, 02 Dec 2024, Steve Dickson wrote:

> Hey,
> 
> On 12/2/24 3:30 PM, Scott Mayhew wrote:
> > Commit 15dc0bea ("exportd: Moved cache upcalls routines into
> > libexport.a") caused write_fsloc() to be elided when junction support is
> > disabled.  Get rid of the bogus #ifdef HAVE_JUNCTION_SUPPORT blocks so
> > that referrals work again (the only #ifdef HAVE_JUNCTION_SUPPORT should
> > be around actual junction code).
> Why not just take the enable_junction config variable
> out of configure.ac as well?
> 
> If we want junctions/referrals (which are the same)
> IMHO... on all the time... Lets not be able to
> turn them off at all?
> 
> Point being... if we are going remove 3 of the 4
> HAVE_JUNCTION_SUPPORT ifdefs... let get ride of
> all of them.

Junctions and referrals are _not_ the same.  A junction is one mechanism
that can be used to generate a referral.  The other way to generate a
referral is with an export entry, and that is the method that stopped
working after 15dc0bea.

When you set up a referral via an export entry you use the refer=
export option, and the directory must be a mountpoint so that nfsd will
consult the export cache when the client tries to access the directory.

For example, I set up the following in /etc/exports:

/export *(rw,insecure,no_root_squash)
/export/ref     *(rw,insecure,no_root_squash,refer=/export@192.168.124.66)

After the client tries to access /export/ref, this is what I see when I
dump the export cache without my fix:

[root@rawhide ~]# cat /proc/net/rpc/nfsd.export/content
#path domain(flags)
/       *(ro,insecure,no_root_squash,sync,no_wdelay,no_subtree_check,v4root,fsid=0,sec=1)
/export *(rw,insecure,no_root_squash,sync,wdelay,no_subtree_check,uuid=c4eeda84:ea1a4dcd:a043fdc1:372d7878,sec=1)
/export/ref     *(rw,insecure,no_root_squash,sync,wdelay,no_subtree_check,uuid=c4eeda84:ea1a4dcd:a043fdc1:372d7878,sec=1)

Notice there's no refer= option.  So when the client does a LOOKUP of
/export/ref, the server treats it as a normal directory... it doesn't
return NFS4ERR_MOVED and so the client doesn't know to query
fs_locations.

Here is what the export cache looks like with my fix:

[root@rawhide ~]# cat /proc/net/rpc/nfsd.export/content
#path domain(flags)
/export *(rw,insecure,no_root_squash,sync,wdelay,no_subtree_check,uuid=c4eeda84:ea1a4dcd:a043fdc1:372d7878,sec=1)
/       *(ro,insecure,no_root_squash,sync,no_wdelay,no_subtree_check,v4root,fsid=0,sec=1)
/export/ref     *(rw,insecure,no_root_squash,sync,wdelay,no_subtree_check,refer=/export@192.168.124.66,uuid=c4eeda84:ea1a4dcd:a043fdc1:372d7878,sec=1)

Note the refer= option is present, and the referral works normally.

A junction is basically a fancy directory that has the user/group/other
mode bits set to 0 and the sticky bit turned on.  The original mode bits
are stored in the trusted.junction.mode extended attribute and the
referral information is stored in the trusted.junction.nfs extended
attribute.

Continuing with my previous example, I have this in my /etc/exports

[root@rawhide ~]# cat /etc/exports
/export *(rw,insecure,no_root_squash)
/export/ref     *(rw,insecure,no_root_squash,refer=/export@192.168.124.66)

Let's add a referral using a junction.

[root@rawhide ~]# nfsref add /export/junc 192.168.124.66 /export
Created junction /export/junc

In this case, /export/junc didn't previously exist, so the nfsref tool
created it.  If /export/junc did already exist, then the original mode
would be stored in the trusted.junction.mode and the original contents
of the directory would be hidden from the client (as well as
non-privileged users on the server).

You can look up the referral info using 'nfsref lookup':

[root@rawhide ~]# nfsref lookup /export/junc
192.168.124.66:/export

        NFS port:       2049
        Valid for:      0
        Currency:       -1
        Flags:          varsub(false)
        GenFlags:       writable(false), going(false), split(true)
        TransFlags:     rdma(true)
        Class:          simul(0), handle(0), fileid(0)
        Class:          writever(0), change(0), readdir(0)
        Read:           rank(0), order(0)
        Write:          rank(0), order(0)

Or you can just use getfattr if you want to see the raw xml:

[root@rawhide ~]# getfattr --only-values -d -m trusted.junction.nfs /export/junc
getfattr: Removing leading '/' from absolute path names
<?xml version="1.0" encoding="UTF-8"?>
<junction>
  <savedmode bits="755"/>
  <fileset>
    <location>
      <host name="192.168.124.66"/>
      <path>
        <component>export</component>
      </path>
      <currency>-1</currency>
      <genflags writable="false" going="false" split="true"/>
      <transflags rdma="true"/>
      <class simul="0" handle="0" fileid="0" writever="0" change="0" readdir="0"/>
      <read rank="0" order="0"/>
      <write rank="0" order="0"/>
      <flags varsub="false"/>
      <validfor>0</validfor>
    </location>
  </fileset>
</junction>

Note that since the /export/junc referral is stored in a junction, it
doesn't appear in the export info:

[root@rawhide ~]# exportfs -v
/export         <world>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,insecure,no_root_squash,no_all_squash)
/export/ref     <world>(sync,wdelay,hide,no_subtree_check,refer=/export@192.168.124.66,sec=sys,rw,insecure,no_root_squash,no_all_squash)

From the client's standpoint, both style referrals work the same:

root@aion:~# mount 192.168.124.26:/export /mnt/t
root@aion:~# ls /mnt/t
junc  ref
root@aion:~# ls /mnt/t/ref
file
root@aion:~# cat /mnt/t/ref/file
I am on the referral server.
root@aion:~# grep nfs4 /proc/mounts
192.168.124.26:/export /mnt/t nfs4 rw,relatime,vers=4.2,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.124.1,local_lock=none,addr=192.168.124.26 0 0
192.168.124.66:/export /mnt/t/ref nfs4 rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.124.1,local_lock=none,addr=192.168.124.66 0 0
root@aion:~# ls /mnt/t/junc
file
root@aion:~# cat /mnt/t/junc/file
I am on the referral server.
root@aion:~# grep nfs4 /proc/mounts
192.168.124.26:/export /mnt/t nfs4 rw,relatime,vers=4.2,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.124.1,local_lock=none,addr=192.168.124.26 0 0
192.168.124.66:/export /mnt/t/ref nfs4 rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.124.1,local_lock=none,addr=192.168.124.66 0 0
192.168.124.66:/export /mnt/t/junc nfs4 rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.124.1,local_lock=none,addr=192.168.124.66 0 0


So if you want to get rid of that last #ifdef HAVE_JUNCTION_SUPPORT
then you have 2 options:

a) get rid of junctions entirely, leaving users with only the old
(relatively speaking) method for configuring referrals
b) force all packagers of nfs-utils to pull in the extra dependencies
needed to support junctions, which is the exact opposite of what the
Debian folks are requesting.

Or you can take the patch and we can continue to have both style
referrals.

-Scott
> 
> steved.
> > 
> > Fixes: 15dc0bea ("exportd: Moved cache upcalls routines into libexport.a")
> > Signed-off-by: Scott Mayhew <smayhew@redhat.com>
> > ---
> >   support/export/cache.c | 7 -------
> >   1 file changed, 7 deletions(-)
> > 
> > diff --git a/support/export/cache.c b/support/export/cache.c
> > index 6c0a44a3..3a8e57cf 100644
> > --- a/support/export/cache.c
> > +++ b/support/export/cache.c
> > @@ -34,10 +34,7 @@
> >   #include "pseudoflavors.h"
> >   #include "xcommon.h"
> >   #include "reexport.h"
> > -
> > -#ifdef HAVE_JUNCTION_SUPPORT
> >   #include "fsloc.h"
> > -#endif
> >   #ifdef USE_BLKID
> >   #include "blkid/blkid.h"
> > @@ -999,7 +996,6 @@ static void nfsd_retry_fh(struct delayed *d)
> >   	*dp = d;
> >   }
> > -#ifdef HAVE_JUNCTION_SUPPORT
> >   static void write_fsloc(char **bp, int *blen, struct exportent *ep)
> >   {
> >   	struct servers *servers;
> > @@ -1022,7 +1018,6 @@ static void write_fsloc(char **bp, int *blen, struct exportent *ep)
> >   	qword_addint(bp, blen, servers->h_referral);
> >   	release_replicas(servers);
> >   }
> > -#endif
> >   static void write_secinfo(char **bp, int *blen, struct exportent *ep, int flag_mask, int extra_flag)
> >   {
> > @@ -1120,9 +1115,7 @@ static int dump_to_cache(int f, char *buf, int blen, char *domain,
> >   		qword_addint(&bp, &blen, exp->e_anongid);
> >   		qword_addint(&bp, &blen, fsidnum);
> > -#ifdef HAVE_JUNCTION_SUPPORT
> >   		write_fsloc(&bp, &blen, exp);
> > -#endif
> >   		write_secinfo(&bp, &blen, exp, flag_mask, do_fsidnum ? NFSEXP_FSID : 0);
> >   		if (exp->e_uuid == NULL || different_fs) {
> >   			char u[16];
>