nfs4.1+: workaround for defunct clientaddr?

nfs4.1+: workaround for defunct clientaddr?

[Manfred Schwarb]

Hi,

I have 2 boxes connected with 2 network cards each, one
crossover connection and one connection via LAN.
I want to use the crossover connection for backup,
so I want to be able to select exactly this wire when
doing my NFS backup transfers. Everything interconnected via NFS4.1
and automount.

Now the thing is, if there is an already existing connection
via LAN, I am not able to select the crossover connection,
there is some session reuse against my will.

automount config:
/net/192.168.99.1  -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.99.100   /  192.168.99.1:/
/net2/192.168.98.1 -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.98.100   /  192.168.98.1:/

mount -l:
192.168.99.1:/data on /net/192.168.99.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
192.168.99.1:/data on /net2/192.168.98.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)

As you see, both connections are on "192.168.99.1:/data", and the backup runs
over the same wire as all user communication, which is not desired.
This even happens if I explicitly set some clientaddr= option.

Now I found two workarounds:
- downgrade to NFS 4.0, clientaddr seems to work with it
- choose different NFS versions, i.e. one connection with
  minorversion=1 and the other with minorversion=2

Both possibilities seem a bit lame to me.
Are there some other (recommended) variants which do what I want?

It seems different minor versions result in different "nfs4_unique_id" values,
and therefore no session sharing occurs. But why do different network
interfaces (via explicitly set clientaddr= by user) not result in different
"nfs4_unique_id" values?

Thanks for any comments and advice,
Manfred

Re: nfs4.1+: workaround for defunct clientaddr?

[Jeff Layton]

On Sun, 2022-10-02 at 14:35 +0200, Manfred Schwarb wrote:
> Hi,
> 
> I have 2 boxes connected with 2 network cards each, one
> crossover connection and one connection via LAN.
> I want to use the crossover connection for backup,
> so I want to be able to select exactly this wire when
> doing my NFS backup transfers. Everything interconnected via NFS4.1
> and automount.
> 
> Now the thing is, if there is an already existing connection
> via LAN, I am not able to select the crossover connection,
> there is some session reuse against my will.
> 
> automount config:
> /net/192.168.99.1  -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.99.100   /  192.168.99.1:/
> /net2/192.168.98.1 -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.98.100   /  192.168.98.1:/
> 
> mount -l:
> 192.168.99.1:/data on /net/192.168.99.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
> 192.168.99.1:/data on /net2/192.168.98.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
> 
> As you see, both connections are on "192.168.99.1:/data", and the backup runs
> over the same wire as all user communication, which is not desired.
> This even happens if I explicitly set some clientaddr= option.
> 
> Now I found two workarounds:
> - downgrade to NFS 4.0, clientaddr seems to work with it
> - choose different NFS versions, i.e. one connection with
>   minorversion=1 and the other with minorversion=2
> 
> Both possibilities seem a bit lame to me.
> Are there some other (recommended) variants which do what I want?
> 
> It seems different minor versions result in different "nfs4_unique_id" values,
> and therefore no session sharing occurs. But why do different network
> interfaces (via explicitly set clientaddr= by user) not result in different
> "nfs4_unique_id" values?
> 
> Thanks for any comments and advice,
> Manfred

That sounds like a bug. We probably need to compare the clientaddr
values in nfs_compare_super or nfs_compare_mount_options so that it
doesn't match if the clientaddrs are different.

As a workaround, you can probably mount the second mount with
-o nosharecache and get what you want.
-- 
Jeff Layton <jlayton@kernel.org>

Re: nfs4.1+: workaround for defunct clientaddr?

[Manfred Schwarb]

Am 03.10.22 um 13:39 schrieb Jeff Layton:
> On Sun, 2022-10-02 at 14:35 +0200, Manfred Schwarb wrote:
>> Hi,
>>
>> I have 2 boxes connected with 2 network cards each, one
>> crossover connection and one connection via LAN.
>> I want to use the crossover connection for backup,
>> so I want to be able to select exactly this wire when
>> doing my NFS backup transfers. Everything interconnected via NFS4.1
>> and automount.
>>
>> Now the thing is, if there is an already existing connection
>> via LAN, I am not able to select the crossover connection,
>> there is some session reuse against my will.
>>
>> automount config:
>> /net/192.168.99.1  -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.99.100   /  192.168.99.1:/
>> /net2/192.168.98.1 -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.98.100   /  192.168.98.1:/
>>
>> mount -l:
>> 192.168.99.1:/data on /net/192.168.99.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
>> 192.168.99.1:/data on /net2/192.168.98.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
>>
>> As you see, both connections are on "192.168.99.1:/data", and the backup runs
>> over the same wire as all user communication, which is not desired.
>> This even happens if I explicitly set some clientaddr= option.
>>
>> Now I found two workarounds:
>> - downgrade to NFS 4.0, clientaddr seems to work with it
>> - choose different NFS versions, i.e. one connection with
>>   minorversion=1 and the other with minorversion=2
>>
>> Both possibilities seem a bit lame to me.
>> Are there some other (recommended) variants which do what I want?
>>
>> It seems different minor versions result in different "nfs4_unique_id" values,
>> and therefore no session sharing occurs. But why do different network
>> interfaces (via explicitly set clientaddr= by user) not result in different
>> "nfs4_unique_id" values?
>>
>> Thanks for any comments and advice,
>> Manfred
>
> That sounds like a bug. We probably need to compare the clientaddr
> values in nfs_compare_super or nfs_compare_mount_options so that it
> doesn't match if the clientaddrs are different.
>
> As a workaround, you can probably mount the second mount with
> -o nosharecache and get what you want.

Indeed, nosharecache works. But the man page has some scary words for it:
  "This is considered a data risk".

Thanks,
Manfred

Re: nfs4.1+: workaround for defunct clientaddr?

[Jeff Layton]

On Mon, 2022-10-03 at 13:55 +0200, Manfred Schwarb wrote:
> Am 03.10.22 um 13:39 schrieb Jeff Layton:
> > On Sun, 2022-10-02 at 14:35 +0200, Manfred Schwarb wrote:
> > > Hi,
> > > 
> > > I have 2 boxes connected with 2 network cards each, one
> > > crossover connection and one connection via LAN.
> > > I want to use the crossover connection for backup,
> > > so I want to be able to select exactly this wire when
> > > doing my NFS backup transfers. Everything interconnected via NFS4.1
> > > and automount.
> > > 
> > > Now the thing is, if there is an already existing connection
> > > via LAN, I am not able to select the crossover connection,
> > > there is some session reuse against my will.
> > > 
> > > automount config:
> > > /net/192.168.99.1  -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.99.100   /  192.168.99.1:/
> > > /net2/192.168.98.1 -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.98.100   /  192.168.98.1:/
> > > 
> > > mount -l:
> > > 192.168.99.1:/data on /net/192.168.99.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
> > > 192.168.99.1:/data on /net2/192.168.98.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
> > > 
> > > As you see, both connections are on "192.168.99.1:/data", and the backup runs
> > > over the same wire as all user communication, which is not desired.
> > > This even happens if I explicitly set some clientaddr= option.
> > > 
> > > Now I found two workarounds:
> > > - downgrade to NFS 4.0, clientaddr seems to work with it
> > > - choose different NFS versions, i.e. one connection with
> > >   minorversion=1 and the other with minorversion=2
> > > 
> > > Both possibilities seem a bit lame to me.
> > > Are there some other (recommended) variants which do what I want?
> > > 
> > > It seems different minor versions result in different "nfs4_unique_id" values,
> > > and therefore no session sharing occurs. But why do different network
> > > interfaces (via explicitly set clientaddr= by user) not result in different
> > > "nfs4_unique_id" values?
> > > 
> > > Thanks for any comments and advice,
> > > Manfred
> > 
> > That sounds like a bug. We probably need to compare the clientaddr
> > values in nfs_compare_super or nfs_compare_mount_options so that it
> > doesn't match if the clientaddrs are different.
> > 


Actually, I take it back, clientaddr is specifically advertised as being
for NFSv4.0 only. The workaround for you is "nosharecache", which will
force the mount under /net2 to get a new superblock altogether.

> > As a workaround, you can probably mount the second mount with
> > -o nosharecache and get what you want.
> 
> Indeed, nosharecache works. But the man page has some scary words for it:
>   "This is considered a data risk".
> 

Yeah, it does sound scary but it's not a huge issue unless you're doing
I/O to the same files at the same time via both mounts. With
"sharecache" (the default) you get better cache coherency in that
situation since the inode and its pagecache are the same.

With "nosharecache" you need to be more careful to flush caches, etc. if
you are doing reads and writes to the same files via different paths. If
you need careful coordination there, then you probably want to use file
locking.
--
Jeff Layton <jlayton@kernel.org>

Re: nfs4.1+: workaround for defunct clientaddr?

[Manfred Schwarb]

Am 03.10.22 um 14:26 schrieb Jeff Layton:
> On Mon, 2022-10-03 at 13:55 +0200, Manfred Schwarb wrote:
>> Am 03.10.22 um 13:39 schrieb Jeff Layton:
>>> On Sun, 2022-10-02 at 14:35 +0200, Manfred Schwarb wrote:
>>>> Hi,
>>>>
>>>> I have 2 boxes connected with 2 network cards each, one
>>>> crossover connection and one connection via LAN.
>>>> I want to use the crossover connection for backup,
>>>> so I want to be able to select exactly this wire when
>>>> doing my NFS backup transfers. Everything interconnected via NFS4.1
>>>> and automount.
>>>>
>>>> Now the thing is, if there is an already existing connection
>>>> via LAN, I am not able to select the crossover connection,
>>>> there is some session reuse against my will.
>>>>
>>>> automount config:
>>>> /net/192.168.99.1  -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.99.100   /  192.168.99.1:/
>>>> /net2/192.168.98.1 -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.98.100   /  192.168.98.1:/
>>>>
>>>> mount -l:
>>>> 192.168.99.1:/data on /net/192.168.99.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
>>>> 192.168.99.1:/data on /net2/192.168.98.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
>>>>
>>>> As you see, both connections are on "192.168.99.1:/data", and the backup runs
>>>> over the same wire as all user communication, which is not desired.
>>>> This even happens if I explicitly set some clientaddr= option.
>>>>
>>>> Now I found two workarounds:
>>>> - downgrade to NFS 4.0, clientaddr seems to work with it
>>>> - choose different NFS versions, i.e. one connection with
>>>>   minorversion=1 and the other with minorversion=2
>>>>
>>>> Both possibilities seem a bit lame to me.
>>>> Are there some other (recommended) variants which do what I want?
>>>>
>>>> It seems different minor versions result in different "nfs4_unique_id" values,
>>>> and therefore no session sharing occurs. But why do different network
>>>> interfaces (via explicitly set clientaddr= by user) not result in different
>>>> "nfs4_unique_id" values?
>>>>
>>>> Thanks for any comments and advice,
>>>> Manfred
>>>
>>> That sounds like a bug. We probably need to compare the clientaddr
>>> values in nfs_compare_super or nfs_compare_mount_options so that it
>>> doesn't match if the clientaddrs are different.
>>>
>
>
> Actually, I take it back, clientaddr is specifically advertised as being
> for NFSv4.0 only. The workaround for you is "nosharecache", which will
> force the mount under /net2 to get a new superblock altogether.

But clientaddr is silently accepted on NFS4.1+, and seemingly silently does nothing.

The point is, RFC5661 explicitely tells
"NFS minor version 1 is deemed superior to NFS minor version 0 with no loss of functionality".

So this behavior comes as a surprise.

>
>>> As a workaround, you can probably mount the second mount with
>>> -o nosharecache and get what you want.
>>
>> Indeed, nosharecache works. But the man page has some scary words for it:
>>   "This is considered a data risk".
>>
>
> Yeah, it does sound scary but it's not a huge issue unless you're doing
> I/O to the same files at the same time via both mounts. With
> "sharecache" (the default) you get better cache coherency in that
> situation since the inode and its pagecache are the same.
>

So I guess this is equivalent to the minorversion=1/minorversion=2 trick
cache coherency wise then?


> With "nosharecache" you need to be more careful to flush caches, etc. if
> you are doing reads and writes to the same files via different paths. If
> you need careful coordination there, then you probably want to use file
> locking.

Thanks for these explanations, it is appreciated!
Manfred

> --
> Jeff Layton <jlayton@kernel.org>

Re: nfs4.1+: workaround for defunct clientaddr?

[Olga Kornievskaia]

 Hi Manfred,

What's the purpose of segregating your connections? You don't want
your backup traffic "interfering" with your regular operation.
However, the assumption is that between 2 NICs the backup traffic and
regular traffic could co-exists, correct? In that case why not use
session trunking?  What you are correctly experiencing is that with
4.1+ the 2nd mount discovers that in the 2nd mount it's the same
server the client is talking to (even if it's thru a different IPs)
and the client will drop the new connection.

For session trunking, you can configure your linux server (I'm
assuming it is, if not that might be a problem) to support session
trunking (by using replica=<> option). Then you can also add
"discovertrunking" option to your mount command and then the client
will discover the 2 available paths to the server. You wouldn't need 2
mounts and you'd have both NICs available to serve your combined
regular and backup traffic. This would be the solution to utilize both
of the NICs (network paths) you have available between the client and
the server.


On Mon, Oct 3, 2022 at 9:27 AM Manfred Schwarb <manfred99@gmx.ch> wrote:
>
> Am 03.10.22 um 14:26 schrieb Jeff Layton:
> > On Mon, 2022-10-03 at 13:55 +0200, Manfred Schwarb wrote:
> >> Am 03.10.22 um 13:39 schrieb Jeff Layton:
> >>> On Sun, 2022-10-02 at 14:35 +0200, Manfred Schwarb wrote:
> >>>> Hi,
> >>>>
> >>>> I have 2 boxes connected with 2 network cards each, one
> >>>> crossover connection and one connection via LAN.
> >>>> I want to use the crossover connection for backup,
> >>>> so I want to be able to select exactly this wire when
> >>>> doing my NFS backup transfers. Everything interconnected via NFS4.1
> >>>> and automount.
> >>>>
> >>>> Now the thing is, if there is an already existing connection
> >>>> via LAN, I am not able to select the crossover connection,
> >>>> there is some session reuse against my will.
> >>>>
> >>>> automount config:
> >>>> /net/192.168.99.1  -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.99.100   /  192.168.99.1:/
> >>>> /net2/192.168.98.1 -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.98.100   /  192.168.98.1:/
> >>>>
> >>>> mount -l:
> >>>> 192.168.99.1:/data on /net/192.168.99.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
> >>>> 192.168.99.1:/data on /net2/192.168.98.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
> >>>>
> >>>> As you see, both connections are on "192.168.99.1:/data", and the backup runs
> >>>> over the same wire as all user communication, which is not desired.
> >>>> This even happens if I explicitly set some clientaddr= option.
> >>>>
> >>>> Now I found two workarounds:
> >>>> - downgrade to NFS 4.0, clientaddr seems to work with it
> >>>> - choose different NFS versions, i.e. one connection with
> >>>>   minorversion=1 and the other with minorversion=2
> >>>>
> >>>> Both possibilities seem a bit lame to me.
> >>>> Are there some other (recommended) variants which do what I want?
> >>>>
> >>>> It seems different minor versions result in different "nfs4_unique_id" values,
> >>>> and therefore no session sharing occurs. But why do different network
> >>>> interfaces (via explicitly set clientaddr= by user) not result in different
> >>>> "nfs4_unique_id" values?
> >>>>
> >>>> Thanks for any comments and advice,
> >>>> Manfred
> >>>
> >>> That sounds like a bug. We probably need to compare the clientaddr
> >>> values in nfs_compare_super or nfs_compare_mount_options so that it
> >>> doesn't match if the clientaddrs are different.
> >>>
> >
> >
> > Actually, I take it back, clientaddr is specifically advertised as being
> > for NFSv4.0 only. The workaround for you is "nosharecache", which will
> > force the mount under /net2 to get a new superblock altogether.
>
> But clientaddr is silently accepted on NFS4.1+, and seemingly silently does nothing.
>
> The point is, RFC5661 explicitely tells
> "NFS minor version 1 is deemed superior to NFS minor version 0 with no loss of functionality".
>
> So this behavior comes as a surprise.
>
> >
> >>> As a workaround, you can probably mount the second mount with
> >>> -o nosharecache and get what you want.
> >>
> >> Indeed, nosharecache works. But the man page has some scary words for it:
> >>   "This is considered a data risk".
> >>
> >
> > Yeah, it does sound scary but it's not a huge issue unless you're doing
> > I/O to the same files at the same time via both mounts. With
> > "sharecache" (the default) you get better cache coherency in that
> > situation since the inode and its pagecache are the same.
> >
>
> So I guess this is equivalent to the minorversion=1/minorversion=2 trick
> cache coherency wise then?
>
>
> > With "nosharecache" you need to be more careful to flush caches, etc. if
> > you are doing reads and writes to the same files via different paths. If
> > you need careful coordination there, then you probably want to use file
> > locking.
>
> Thanks for these explanations, it is appreciated!
> Manfred
>
> > --
> > Jeff Layton <jlayton@kernel.org>
>

Re: nfs4.1+: workaround for defunct clientaddr?

[Manfred Schwarb]

Am 03.10.22 um 16:18 schrieb Olga Kornievskaia:
>  Hi Manfred,
>
> What's the purpose of segregating your connections? You don't want
> your backup traffic "interfering" with your regular operation.
> However, the assumption is that between 2 NICs the backup traffic and
> regular traffic could co-exists, correct? In that case why not use
> session trunking?  What you are correctly experiencing is that with
> 4.1+ the 2nd mount discovers that in the 2nd mount it's the same
> server the client is talking to (even if it's thru a different IPs)
> and the client will drop the new connection.
>

I did not know that I'm supposed to want this ;-)  Sounds interesting.
I searched for "replica" and "discovertrunking" but did not find anything.
I'm on Linux 5.14 / nfs-kernel-server-2.1.1. Did you mean "nconnect"
or is my Linux installation simply too old? Probably. 2.1.1 is 5 years old.
I'm on openSUSE Leap 15.4, on idea why they ship such old userspace components.


> For session trunking, you can configure your linux server (I'm
> assuming it is, if not that might be a problem) to support session
> trunking (by using replica=<> option). Then you can also add
> "discovertrunking" option to your mount command and then the client
> will discover the 2 available paths to the server. You wouldn't need 2
> mounts and you'd have both NICs available to serve your combined
> regular and backup traffic. This would be the solution to utilize both
> of the NICs (network paths) you have available between the client and
> the server.
>
>
> On Mon, Oct 3, 2022 at 9:27 AM Manfred Schwarb <manfred99@gmx.ch> wrote:
>>
>> Am 03.10.22 um 14:26 schrieb Jeff Layton:
>>> On Mon, 2022-10-03 at 13:55 +0200, Manfred Schwarb wrote:
>>>> Am 03.10.22 um 13:39 schrieb Jeff Layton:
>>>>> On Sun, 2022-10-02 at 14:35 +0200, Manfred Schwarb wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I have 2 boxes connected with 2 network cards each, one
>>>>>> crossover connection and one connection via LAN.
>>>>>> I want to use the crossover connection for backup,
>>>>>> so I want to be able to select exactly this wire when
>>>>>> doing my NFS backup transfers. Everything interconnected via NFS4.1
>>>>>> and automount.
>>>>>>
>>>>>> Now the thing is, if there is an already existing connection
>>>>>> via LAN, I am not able to select the crossover connection,
>>>>>> there is some session reuse against my will.
>>>>>>
>>>>>> automount config:
>>>>>> /net/192.168.99.1  -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.99.100   /  192.168.99.1:/
>>>>>> /net2/192.168.98.1 -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.98.100   /  192.168.98.1:/
>>>>>>
>>>>>> mount -l:
>>>>>> 192.168.99.1:/data on /net/192.168.99.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
>>>>>> 192.168.99.1:/data on /net2/192.168.98.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
>>>>>>
>>>>>> As you see, both connections are on "192.168.99.1:/data", and the backup runs
>>>>>> over the same wire as all user communication, which is not desired.
>>>>>> This even happens if I explicitly set some clientaddr= option.
>>>>>>
>>>>>> Now I found two workarounds:
>>>>>> - downgrade to NFS 4.0, clientaddr seems to work with it
>>>>>> - choose different NFS versions, i.e. one connection with
>>>>>>   minorversion=1 and the other with minorversion=2
>>>>>>
>>>>>> Both possibilities seem a bit lame to me.
>>>>>> Are there some other (recommended) variants which do what I want?
>>>>>>
>>>>>> It seems different minor versions result in different "nfs4_unique_id" values,
>>>>>> and therefore no session sharing occurs. But why do different network
>>>>>> interfaces (via explicitly set clientaddr= by user) not result in different
>>>>>> "nfs4_unique_id" values?
>>>>>>
>>>>>> Thanks for any comments and advice,
>>>>>> Manfred
>>>>>
>>>>> That sounds like a bug. We probably need to compare the clientaddr
>>>>> values in nfs_compare_super or nfs_compare_mount_options so that it
>>>>> doesn't match if the clientaddrs are different.
>>>>>
>>>
>>>
>>> Actually, I take it back, clientaddr is specifically advertised as being
>>> for NFSv4.0 only. The workaround for you is "nosharecache", which will
>>> force the mount under /net2 to get a new superblock altogether.
>>
>> But clientaddr is silently accepted on NFS4.1+, and seemingly silently does nothing.
>>
>> The point is, RFC5661 explicitely tells
>> "NFS minor version 1 is deemed superior to NFS minor version 0 with no loss of functionality".
>>
>> So this behavior comes as a surprise.
>>
>>>
>>>>> As a workaround, you can probably mount the second mount with
>>>>> -o nosharecache and get what you want.
>>>>
>>>> Indeed, nosharecache works. But the man page has some scary words for it:
>>>>   "This is considered a data risk".
>>>>
>>>
>>> Yeah, it does sound scary but it's not a huge issue unless you're doing
>>> I/O to the same files at the same time via both mounts. With
>>> "sharecache" (the default) you get better cache coherency in that
>>> situation since the inode and its pagecache are the same.
>>>
>>
>> So I guess this is equivalent to the minorversion=1/minorversion=2 trick
>> cache coherency wise then?
>>
>>
>>> With "nosharecache" you need to be more careful to flush caches, etc. if
>>> you are doing reads and writes to the same files via different paths. If
>>> you need careful coordination there, then you probably want to use file
>>> locking.
>>
>> Thanks for these explanations, it is appreciated!
>> Manfred
>>
>>> --
>>> Jeff Layton <jlayton@kernel.org>
>>

Re: nfs4.1+: workaround for defunct clientaddr?

[Olga Kornievskaia]

On Mon, Oct 3, 2022 at 12:14 PM Manfred Schwarb <manfred99@gmx.ch> wrote:
>
> Am 03.10.22 um 16:18 schrieb Olga Kornievskaia:
> >  Hi Manfred,
> >
> > What's the purpose of segregating your connections? You don't want
> > your backup traffic "interfering" with your regular operation.
> > However, the assumption is that between 2 NICs the backup traffic and
> > regular traffic could co-exists, correct? In that case why not use
> > session trunking?  What you are correctly experiencing is that with
> > 4.1+ the 2nd mount discovers that in the 2nd mount it's the same
> > server the client is talking to (even if it's thru a different IPs)
> > and the client will drop the new connection.
> >
>
> I did not know that I'm supposed to want this ;-)  Sounds interesting.
> I searched for "replica" and "discovertrunking" but did not find anything.
> I'm on Linux 5.14 / nfs-kernel-server-2.1.1. Did you mean "nconnect"
> or is my Linux installation simply too old? Probably. 2.1.1 is 5 years old.
> I'm on openSUSE Leap 15.4, on idea why they ship such old userspace components.

Trunking support is rather new on the client, so you'd need something
that's 5.18+ (that's where "trunkdiscovery" (apologizes for
mislabeling it) option went in but main code support went into 5.17).
"replica=" option on the server I'm not sure how old but not "new".

> > For session trunking, you can configure your linux server (I'm
> > assuming it is, if not that might be a problem) to support session
> > trunking (by using replica=<> option). Then you can also add
> > "discovertrunking" option to your mount command and then the client
> > will discover the 2 available paths to the server. You wouldn't need 2
> > mounts and you'd have both NICs available to serve your combined
> > regular and backup traffic. This would be the solution to utilize both
> > of the NICs (network paths) you have available between the client and
> > the server.
> >
> >
> > On Mon, Oct 3, 2022 at 9:27 AM Manfred Schwarb <manfred99@gmx.ch> wrote:
> >>
> >> Am 03.10.22 um 14:26 schrieb Jeff Layton:
> >>> On Mon, 2022-10-03 at 13:55 +0200, Manfred Schwarb wrote:
> >>>> Am 03.10.22 um 13:39 schrieb Jeff Layton:
> >>>>> On Sun, 2022-10-02 at 14:35 +0200, Manfred Schwarb wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> I have 2 boxes connected with 2 network cards each, one
> >>>>>> crossover connection and one connection via LAN.
> >>>>>> I want to use the crossover connection for backup,
> >>>>>> so I want to be able to select exactly this wire when
> >>>>>> doing my NFS backup transfers. Everything interconnected via NFS4.1
> >>>>>> and automount.
> >>>>>>
> >>>>>> Now the thing is, if there is an already existing connection
> >>>>>> via LAN, I am not able to select the crossover connection,
> >>>>>> there is some session reuse against my will.
> >>>>>>
> >>>>>> automount config:
> >>>>>> /net/192.168.99.1  -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.99.100   /  192.168.99.1:/
> >>>>>> /net2/192.168.98.1 -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.98.100   /  192.168.98.1:/
> >>>>>>
> >>>>>> mount -l:
> >>>>>> 192.168.99.1:/data on /net/192.168.99.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
> >>>>>> 192.168.99.1:/data on /net2/192.168.98.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
> >>>>>>
> >>>>>> As you see, both connections are on "192.168.99.1:/data", and the backup runs
> >>>>>> over the same wire as all user communication, which is not desired.
> >>>>>> This even happens if I explicitly set some clientaddr= option.
> >>>>>>
> >>>>>> Now I found two workarounds:
> >>>>>> - downgrade to NFS 4.0, clientaddr seems to work with it
> >>>>>> - choose different NFS versions, i.e. one connection with
> >>>>>>   minorversion=1 and the other with minorversion=2
> >>>>>>
> >>>>>> Both possibilities seem a bit lame to me.
> >>>>>> Are there some other (recommended) variants which do what I want?
> >>>>>>
> >>>>>> It seems different minor versions result in different "nfs4_unique_id" values,
> >>>>>> and therefore no session sharing occurs. But why do different network
> >>>>>> interfaces (via explicitly set clientaddr= by user) not result in different
> >>>>>> "nfs4_unique_id" values?
> >>>>>>
> >>>>>> Thanks for any comments and advice,
> >>>>>> Manfred
> >>>>>
> >>>>> That sounds like a bug. We probably need to compare the clientaddr
> >>>>> values in nfs_compare_super or nfs_compare_mount_options so that it
> >>>>> doesn't match if the clientaddrs are different.
> >>>>>
> >>>
> >>>
> >>> Actually, I take it back, clientaddr is specifically advertised as being
> >>> for NFSv4.0 only. The workaround for you is "nosharecache", which will
> >>> force the mount under /net2 to get a new superblock altogether.
> >>
> >> But clientaddr is silently accepted on NFS4.1+, and seemingly silently does nothing.
> >>
> >> The point is, RFC5661 explicitely tells
> >> "NFS minor version 1 is deemed superior to NFS minor version 0 with no loss of functionality".
> >>
> >> So this behavior comes as a surprise.
> >>
> >>>
> >>>>> As a workaround, you can probably mount the second mount with
> >>>>> -o nosharecache and get what you want.
> >>>>
> >>>> Indeed, nosharecache works. But the man page has some scary words for it:
> >>>>   "This is considered a data risk".
> >>>>
> >>>
> >>> Yeah, it does sound scary but it's not a huge issue unless you're doing
> >>> I/O to the same files at the same time via both mounts. With
> >>> "sharecache" (the default) you get better cache coherency in that
> >>> situation since the inode and its pagecache are the same.
> >>>
> >>
> >> So I guess this is equivalent to the minorversion=1/minorversion=2 trick
> >> cache coherency wise then?
> >>
> >>
> >>> With "nosharecache" you need to be more careful to flush caches, etc. if
> >>> you are doing reads and writes to the same files via different paths. If
> >>> you need careful coordination there, then you probably want to use file
> >>> locking.
> >>
> >> Thanks for these explanations, it is appreciated!
> >> Manfred
> >>
> >>> --
> >>> Jeff Layton <jlayton@kernel.org>
> >>
>

Re: nfs4.1+: workaround for defunct clientaddr?

[Tom Talpey]

On 10/3/2022 9:24 AM, Manfred Schwarb wrote:
> Am 03.10.22 um 14:26 schrieb Jeff Layton:
>> On Mon, 2022-10-03 at 13:55 +0200, Manfred Schwarb wrote:
>>> Am 03.10.22 um 13:39 schrieb Jeff Layton:
>>>> On Sun, 2022-10-02 at 14:35 +0200, Manfred Schwarb wrote:
>>>>> Hi,
>>>>>
>>>>> I have 2 boxes connected with 2 network cards each, one
>>>>> crossover connection and one connection via LAN.
>>>>> I want to use the crossover connection for backup,
>>>>> so I want to be able to select exactly this wire when
>>>>> doing my NFS backup transfers. Everything interconnected via NFS4.1
>>>>> and automount.
>>>>>
>>>>> Now the thing is, if there is an already existing connection
>>>>> via LAN, I am not able to select the crossover connection,
>>>>> there is some session reuse against my will.
>>>>>
>>>>> automount config:
>>>>> /net/192.168.99.1  -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.99.100   /  192.168.99.1:/
>>>>> /net2/192.168.98.1 -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.98.100   /  192.168.98.1:/
>>>>>
>>>>> mount -l:
>>>>> 192.168.99.1:/data on /net/192.168.99.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
>>>>> 192.168.99.1:/data on /net2/192.168.98.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
>>>>>
>>>>> As you see, both connections are on "192.168.99.1:/data", and the backup runs
>>>>> over the same wire as all user communication, which is not desired.
>>>>> This even happens if I explicitly set some clientaddr= option.
>>>>>
>>>>> Now I found two workarounds:
>>>>> - downgrade to NFS 4.0, clientaddr seems to work with it
>>>>> - choose different NFS versions, i.e. one connection with
>>>>>    minorversion=1 and the other with minorversion=2
>>>>>
>>>>> Both possibilities seem a bit lame to me.
>>>>> Are there some other (recommended) variants which do what I want?
>>>>>
>>>>> It seems different minor versions result in different "nfs4_unique_id" values,
>>>>> and therefore no session sharing occurs. But why do different network
>>>>> interfaces (via explicitly set clientaddr= by user) not result in different
>>>>> "nfs4_unique_id" values?
>>>>>
>>>>> Thanks for any comments and advice,
>>>>> Manfred
>>>>
>>>> That sounds like a bug. We probably need to compare the clientaddr
>>>> values in nfs_compare_super or nfs_compare_mount_options so that it
>>>> doesn't match if the clientaddrs are different.
>>>>
>>
>>
>> Actually, I take it back, clientaddr is specifically advertised as being
>> for NFSv4.0 only. The workaround for you is "nosharecache", which will
>> force the mount under /net2 to get a new superblock altogether.
> 
> But clientaddr is silently accepted on NFS4.1+, and seemingly silently does nothing.
> 
> The point is, RFC5661 explicitely tells
> "NFS minor version 1 is deemed superior to NFS minor version 0 with no loss of functionality".

NFSv4.1 doesn't have a clientaddr in the protocol, only 4.0.

I believe the reason that the clientaddr option is accepted by mount is
to handle multi-protocol negotiate. If the admin requests "any NFSv4"
and provides an explicit clientaddr, it is only needed when falling
all the way down to 4.0, but ignoring it would be incorrect. So perhaps
the mount command could use a tweak.

> So this behavior comes as a surprise.

Well, it wouldn't have been an issue if the missing comparison bug that
Jeff mentions was fixed... I think that's the real issue.

Tom.


>>>> As a workaround, you can probably mount the second mount with
>>>> -o nosharecache and get what you want.
>>>
>>> Indeed, nosharecache works. But the man page has some scary words for it:
>>>    "This is considered a data risk".
>>>
>>
>> Yeah, it does sound scary but it's not a huge issue unless you're doing
>> I/O to the same files at the same time via both mounts. With
>> "sharecache" (the default) you get better cache coherency in that
>> situation since the inode and its pagecache are the same.
>>
> 
> So I guess this is equivalent to the minorversion=1/minorversion=2 trick
> cache coherency wise then?
> 
> 
>> With "nosharecache" you need to be more careful to flush caches, etc. if
>> you are doing reads and writes to the same files via different paths. If
>> you need careful coordination there, then you probably want to use file
>> locking.
> 
> Thanks for these explanations, it is appreciated!
> Manfred
> 
>> --
>> Jeff Layton <jlayton@kernel.org>
> 
>