NFS4: ssh + unlink(~/.Xauthority) delays

NFS4: ssh + unlink(~/.Xauthority) delays

[Sven Geggus]

Hello,

I'm about to setup a Linux based fileserver for home directories and a
couple of Linux clients with kerberos and NFS4 (no NFS3 for security
reasons).

The whole stuff is currently based on debian stable (with a backported
current NFS userland 1.2.5) and a recent vanilla kernel (3.5.2).

So far I have the NFS4 Server up and running as well as a couple of clients
with NFS4+autofs mounted home directories.

All this stuff mostly works now, but unfortunately I ran into some strange
bahaviour now.

When I ssh from one machine to another the system hangs (a delay of up to 60
seconds) while running xauth.

Replacing xauth by a wrapper script I have been able to trace this behaviour
to a hang of an unlink("/home/<user>/.Xauthority") system call.

So the question is what cases hangs in NFS4 based Linux systems in general
an in this particular case?

Any hint on how to debug this?

Sven

-- 
"C Is Quirky, Flawed, And An Enormous Success."
(Dennis M. Ritchie)

/me is giggls@ircnet, http://sven.gegg.us/ on the Web

NFS4ERR_DELAY (was: NFS4: ssh + unlink(~/.Xauthority) delays)

[Sven Geggus]

Sven Geggus <lists@fuchsschwanzdomain.de> wrote:

> So the question is what cases hangs in NFS4 based Linux systems in general
> an in this particular case?

Digging this down a little bit further using wireshark I figured out that
the server responds to the REMOVE .Xauthority call with an NFS4ERR_DELAY for
whatever reason.

This is a currently a test-machine with one single user and next to zero
load.

So I will probably need a hint on how to debug the server behaviour, as a
NFS4ERR_DELAY response is definitely not expected in this case.

Sven

-- 
"Das Einzige wovor wir Angst haben müssen ist die Angst selbst"
						(Franklin D. Roosevelt)

/me is giggls@ircnet, http://sven.gegg.us/ on the Web

Re: NFS4ERR_DELAY (was: NFS4: ssh + unlink(~/.Xauthority) delays)

[Jeff Layton]

On Tue, 21 Aug 2012 09:19:03 +0000 (UTC)
Sven Geggus <lists@fuchsschwanzdomain.de> wrote:

> Sven Geggus <lists@fuchsschwanzdomain.de> wrote:
> 
> > So the question is what cases hangs in NFS4 based Linux systems in general
> > an in this particular case?
> 
> Digging this down a little bit further using wireshark I figured out that
> the server responds to the REMOVE .Xauthority call with an NFS4ERR_DELAY for
> whatever reason.
> 
> This is a currently a test-machine with one single user and next to zero
> load.
> 
> So I will probably need a hint on how to debug the server behaviour, as a
> NFS4ERR_DELAY response is definitely not expected in this case.
> 
> Sven
> 

It's often the case that this indicates a problem communicating over
the callback channel. For instance, the server is trying to recall a
delegation but the client isn't responding, so the server has to wait
until the recall attempt times out before proceeding.

-- 
Jeff Layton <jlayton@redhat.com>

Re: NFS4ERR_DELAY

[Sven Geggus]

Jeff Layton <jlayton@redhat.com> wrote:

> It's often the case that this indicates a problem communicating over
> the callback channel. For instance, the server is trying to recall a
> delegation but the client isn't responding, so the server has to wait
> until the recall attempt times out before proceeding.

Hm I'm not shure if I understand this correctly.

I am talking about exactly 3 machines (and one single user for now) here:
clientA, clientB and the NFS-server.

"user" is logged in on clientA any now opens a shell to ssh to clientB.

The NFS-server is providing the home for the user on both machines.

The ssh is now getting delayed for up to 1 minute because the NFS server
does not allow for the .Xauthority file to be deleted immediately.

It is probably worth to mention, that I'm currently experimenting with
btrfs on the server. Is there a chance that this bug will disapper when I
change the underlaying filesystem of the server to ext4?

Sven

-- 
Trotz der zunehmenden Verbreitung von Linux erfreut sich der Bär,
und - dank Knut - insbesondere der Eisbär, deutlich größerer
Beliebtheit als der Pinguin. (Gefunden bei http://telepolis.de/)
/me is giggls@ircnet, http://sven.gegg.us/ on the Web

Re: NFS4ERR_DELAY

[Jeff Layton]

On Tue, 21 Aug 2012 12:52:16 +0000 (UTC)
Sven Geggus <lists@fuchsschwanzdomain.de> wrote:

> Jeff Layton <jlayton@redhat.com> wrote:
> 
> > It's often the case that this indicates a problem communicating over
> > the callback channel. For instance, the server is trying to recall a
> > delegation but the client isn't responding, so the server has to wait
> > until the recall attempt times out before proceeding.
> 
> Hm I'm not shure if I understand this correctly.
> 
> I am talking about exactly 3 machines (and one single user for now) here:
> clientA, clientB and the NFS-server.
> 
> "user" is logged in on clientA any now opens a shell to ssh to clientB.
> 

Right, so you probably opened ~/Xauthority on clientA and got a
delegation. Then you ssh'ed to clientB and opened the file there. At
that point, the server has to recall the delegation. Usually that's
pretty quick, but if the server can't talk to clientA on the callback
port then it has to wait and eventually time out before it can allow
the open on clientB to proceed.

> The NFS-server is providing the home for the user on both machines.
> 
> The ssh is now getting delayed for up to 1 minute because the NFS server
> does not allow for the .Xauthority file to be deleted immediately.
> 
> It is probably worth to mention, that I'm currently experimenting with
> btrfs on the server. Is there a chance that this bug will disapper when I
> change the underlaying filesystem of the server to ext4?
> 
> Sven
> 

You asked for hints on how to debug it, and I gave one. The server will
often return NFS4ERR_DELAY when it's waiting for a delegation recall to
complete. I'd make sure that that's all working as expected.

-- 
Jeff Layton <jlayton@redhat.com>

Re: NFS4ERR_DELAY

[J. Bruce Fields]

On Tue, Aug 21, 2012 at 09:07:06AM -0400, Jeff Layton wrote:
> On Tue, 21 Aug 2012 12:52:16 +0000 (UTC)
> Sven Geggus <lists@fuchsschwanzdomain.de> wrote:
> > The NFS-server is providing the home for the user on both machines.
> > 
> > The ssh is now getting delayed for up to 1 minute because the NFS server
> > does not allow for the .Xauthority file to be deleted immediately.
> > 
> > It is probably worth to mention, that I'm currently experimenting with
> > btrfs on the server. Is there a chance that this bug will disapper when I
> > change the underlaying filesystem of the server to ext4?
> > 
> > Sven
> > 
> 
> You asked for hints on how to debug it, and I gave one. The server will
> often return NFS4ERR_DELAY when it's waiting for a delegation recall to
> complete. I'd make sure that that's all working as expected.

Yep.  There's a recent regression which could cause this; could you try:

	https://lkml.org/lkml/2012/8/16/531

?

--b.

Re: NFS4ERR_DELAY

[Sven Geggus]

J. Bruce Fields <bfields@fieldses.org> wrote:

> Yep.  There's a recent regression which could cause this; could you try:
> 
>        https://lkml.org/lkml/2012/8/16/531

Patch applied, problem solved.

Thanks

Sven

-- 
Um Kontrolle Ihres Kontos wiederzugewinnen, klicken Sie bitte auf das
Verbindungsgebrüll. (aus einer Ebay fishing Mail)

/me is giggls@ircnet, http://sven.gegg.us/ on the Web

Re: NFS4: ssh + unlink(~/.Xauthority) delays

[Jim Rees]

Sven Geggus wrote:

  Hello,
  
  I'm about to setup a Linux based fileserver for home directories and a
  couple of Linux clients with kerberos and NFS4 (no NFS3 for security
  reasons).
  
  The whole stuff is currently based on debian stable (with a backported
  current NFS userland 1.2.5) and a recent vanilla kernel (3.5.2).
  
  So far I have the NFS4 Server up and running as well as a couple of clients
  with NFS4+autofs mounted home directories.
  
  All this stuff mostly works now, but unfortunately I ran into some strange
  bahaviour now.
  
  When I ssh from one machine to another the system hangs (a delay of up to 60
  seconds) while running xauth.
  
  Replacing xauth by a wrapper script I have been able to trace this behaviour
  to a hang of an unlink("/home/<user>/.Xauthority") system call.
  
  So the question is what cases hangs in NFS4 based Linux systems in general
  an in this particular case?

No, the question is why did X put that file in your home directory.  That's
not where it belongs.  I've got this in my .xinitrc:

setenv XAUTHORITY /tmp/Xauthority`id -u`

Re: NFS4: ssh + unlink(~/.Xauthority) delays

[Sven Geggus]

Jim Rees <rees@umich.edu> wrote:

> No, the question is why did X put that file in your home directory.  That's
> not where it belongs.  I've got this in my .xinitrc:
> 
> setenv XAUTHORITY /tmp/Xauthority`id -u`

I agree, that ~/.Xauthority is probably not the best place for this file,
but it is the default (at least in Debian).

If it is expected behaviour that this will not work correctly with shared
Homedirectories, I wonder why this did not cause me any trouble with NFS3
for many years.

Anyway I did not yet figure out how to globally change the XAUTHORITY
Variable in kdm and ssh :(

Sven

-- 
Microsoft ist offenbar die einzige Firma, die in der Lage ist, ein mit
Office nicht kompatibles Bürosoftwarepaket einzuführen.
                            (Florian Weimer in de.alt.sysadmin.recovery)
/me is giggls@ircnet, http://sven.gegg.us/ on the Web

Re: NFS4: ssh + unlink(~/.Xauthority) delays

[Jim Rees]

Sven Geggus wrote:

  Jim Rees <rees@umich.edu> wrote:
  
  > No, the question is why did X put that file in your home directory.  That's
  > not where it belongs.  I've got this in my .xinitrc:
  > 
  > setenv XAUTHORITY /tmp/Xauthority`id -u`
  
  I agree, that ~/.Xauthority is probably not the best place for this file,
  but it is the default (at least in Debian).
  
  If it is expected behaviour that this will not work correctly with shared
  Homedirectories, I wonder why this did not cause me any trouble with NFS3
  for many years.

Are you sure it didn't cause you any trouble?  You were giving root access
to anyone who could snoop the Xauthority file off the wire.  That's the main
reason you don't want it in your home dir.  As to why that's still the
default, laziness I guess.

  Anyway I did not yet figure out how to globally change the XAUTHORITY
  Variable in kdm and ssh :(

I don't use kdm.  For ssh:

% cat ~/.ssh/environment
XAUTHORITY=/tmp/Xauthority1234

You might also have to fix /etc/ssh/sshd_config.

Re: NFS4: ssh + unlink(~/.Xauthority) delays

[Malahal Naineni]

> Sven Geggus wrote:
> 
>   Jim Rees <rees@umich.edu> wrote:
>   
>   > No, the question is why did X put that file in your home directory.  That's
>   > not where it belongs.  I've got this in my .xinitrc:
>   > 
>   > setenv XAUTHORITY /tmp/Xauthority`id -u`
>   
>   I agree, that ~/.Xauthority is probably not the best place for this file,
>   but it is the default (at least in Debian).
>   
>   If it is expected behaviour that this will not work correctly with shared
>   Homedirectories, I wonder why this did not cause me any trouble with NFS3
>   for many years.

NFS3 doesn't have delegations and this file placement should not be an
issue in NFSv4 either if it is working correctly!

Regards, Malahal.

Re: NFS4: ssh + unlink(~/.Xauthority) delays

[Jim Rees]

Malahal Naineni wrote:

  NFS3 doesn't have delegations and this file placement should not be an
  issue in NFSv4 either if it is working correctly!

Xauthority doesn't belong in $HOME, it belongs on the local disk.  That has
nothing to do with NFS v3/v4.  It has to do with limiting distribution and
exposure of sensitive information.

Re: NFS4: ssh + unlink(~/.Xauthority) delays

[J. Bruce Fields]

On Tue, Aug 21, 2012 at 02:42:30PM -0400, Jim Rees wrote:
> Malahal Naineni wrote:
> 
>   NFS3 doesn't have delegations and this file placement should not be an
>   issue in NFSv4 either if it is working correctly!
> 
> Xauthority doesn't belong in $HOME, it belongs on the local disk.  That has
> nothing to do with NFS v3/v4.  It has to do with limiting distribution and
> exposure of sensitive information.

Yah.  Unlinking a file shouldn't take a minute, though, so we do have a
bug, however it was found.  Let me know if the patch I referred to fixes
it.

--b.

nfs4err_delay

[Kazi Anwar]

Hi,
We are running nfs v 4.1 on centos 7.6.
We are seeing an NFS issue where when files/dirs are deleted from a
client they are occasionally stuck at unlinkat system call(according
to strace its stuck for 100.5 secs every time). Can anyone explain
this behavior?
Running tcp dump shows nfs4err_delay status sent from the server to
the stuck client.

--
Kazi Anwar

Re: nfs4err_delay

[J. Bruce Fields]

On Wed, Sep 22, 2021 at 03:56:23PM -0500, Kazi Anwar wrote:
> We are running nfs v 4.1 on centos 7.6.
> We are seeing an NFS issue where when files/dirs are deleted from a
> client they are occasionally stuck at unlinkat system call(according
> to strace its stuck for 100.5 secs every time). Can anyone explain
> this behavior?
> Running tcp dump shows nfs4err_delay status sent from the server to
> the stuck client.

Client and server are both centos 7.6?

Is the NFS4ERR_DELAY a reponse to a REMOVE?

Does /proc/locks show a delegation held on the file the client's trying
to remove?

--b.

Re: nfs4err_delay

[Kazi Anwar]

Yes, both clients and server are centos 7.6. And the NFS4ERR_DELAY is
a response to a REMOVE.
I will need to check on the locks the next time it happens. Can you
share what you are thinking?

thanks,
Kazi

On Fri, Sep 24, 2021 at 11:39 AM J. Bruce Fields <bfields@fieldses.org> wrote:
>
> On Wed, Sep 22, 2021 at 03:56:23PM -0500, Kazi Anwar wrote:
> > We are running nfs v 4.1 on centos 7.6.
> > We are seeing an NFS issue where when files/dirs are deleted from a
> > client they are occasionally stuck at unlinkat system call(according
> > to strace its stuck for 100.5 secs every time). Can anyone explain
> > this behavior?
> > Running tcp dump shows nfs4err_delay status sent from the server to
> > the stuck client.
>
> Client and server are both centos 7.6?
>
> Is the NFS4ERR_DELAY a reponse to a REMOVE?
>
> Does /proc/locks show a delegation held on the file the client's trying
> to remove?
>
> --b.

Re: nfs4err_delay

[J. Bruce Fields]

On Fri, Sep 24, 2021 at 12:54:30PM -0500, Kazi Anwar wrote:
> Yes, both clients and server are centos 7.6. And the NFS4ERR_DELAY is
> a response to a REMOVE.
> I will need to check on the locks the next time it happens. Can you
> share what you are thinking?

Offhand the only reason I can think a server would return DELAY is that
there's a delegation on the file being removed, and the delegation
recall and return isn't working for some reason.

If that's the case, it should succeed after about 90 seconds.  Also, you
can workaround the problem by turning of delegations and leases with
"echo 0 >/proc/sys/fs/leases_enable".

--b.

> 
> thanks,
> Kazi
> 
> On Fri, Sep 24, 2021 at 11:39 AM J. Bruce Fields <bfields@fieldses.org> wrote:
> >
> > On Wed, Sep 22, 2021 at 03:56:23PM -0500, Kazi Anwar wrote:
> > > We are running nfs v 4.1 on centos 7.6.
> > > We are seeing an NFS issue where when files/dirs are deleted from a
> > > client they are occasionally stuck at unlinkat system call(according
> > > to strace its stuck for 100.5 secs every time). Can anyone explain
> > > this behavior?
> > > Running tcp dump shows nfs4err_delay status sent from the server to
> > > the stuck client.
> >
> > Client and server are both centos 7.6?
> >
> > Is the NFS4ERR_DELAY a reponse to a REMOVE?
> >
> > Does /proc/locks show a delegation held on the file the client's trying
> > to remove?
> >
> > --b.