Move (mv) hangs with NFSv4

Move (mv) hangs with NFSv4

[Pierre Ossman]

I've been trying out NFSv4 instead of NFSv3 on the systems here and have
had mixed success. First of there was an uid/gid mapping bug which was
solved in 2.6.9.

Now I've run into another serious bug; I cannot move files to the NFS
filesystem. It stalls when the file is finished. I've tried with big and
small files (even 0 bytes) and the effect is the same.

I started guessing that it sets some attributes during the final stage
so i tried cp -a, which has the same effect. So this seems to support 
that theory. chmod and chown work fine. I don't know if there is some 
other command I can try that only messes with attributes.

There is no error message on the server. On the client I get "RPC: 
rpciod waiting on sync task!" now and then. Most of the time it doesn't 
say a thing though so I'm not sure it's related.

The server is a FC2 machine running 2.6.9-1.6_FC2 (also tried 2.6.8.1 +
uid/gid patch) and the clients are a FC2 machine with the same kernel
and a FC3 machine with a clean 2.6.9.

Rgds
Pierre


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Move (mv) hangs with NFSv4

[Trond Myklebust]

fr den 03.12.2004 Klokka 18:51 (+0100) skreiv Pierre Ossman:

> There is no error message on the server. On the client I get "RPC: 
> rpciod waiting on sync task!" now and then. Most of the time it doesn't 
> say a thing though so I'm not sure it's related.

The "rpciod waiting on sync task" is a known problem that should
normally only occur if you ^C out. I'm still working on a complete fix
for that.

As for the other hang: I'm not reproducing it against non-linux servers.
Bruce, any ideas?

Cheers,
  Trond

Cheers,
  Trond

-- 
Trond Myklebust <trond.myklebust@fys.uio.no>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Move (mv) hangs with NFSv4

[J. Bruce Fields]

On Fri, Dec 03, 2004 at 01:18:51PM -0500, Trond Myklebust wrote:
> fr den 03.12.2004 Klokka 18:51 (+0100) skreiv Pierre Ossman:
> 
> > There is no error message on the server. On the client I get "RPC: 
> > rpciod waiting on sync task!" now and then. Most of the time it doesn't 
> > say a thing though so I'm not sure it's related.
> 
> The "rpciod waiting on sync task" is a known problem that should
> normally only occur if you ^C out. I'm still working on a complete fix
> for that.
> 
> As for the other hang: I'm not reproducing it against non-linux servers.
> Bruce, any ideas?

Hm.  With our latest kernel and user utilities I actually get a segfault
on cp -a from the getxattr to get the acl on the source file.  Probably
a bug our modified libacl, ugh.  In any case, probably also a different
bug from what he's seeing.  Maybe I could see a pcap file or an strace
of the cp -a showing where it hangs?

--b.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Move (mv) hangs with NFSv4

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 609 bytes --]

J. Bruce Fields wrote:

>
>Hm.  With our latest kernel and user utilities I actually get a segfault
>on cp -a from the getxattr to get the acl on the source file.  Probably
>a bug our modified libacl, ugh.  In any case, probably also a different
>bug from what he's seeing.  Maybe I could see a pcap file or an strace
>of the cp -a showing where it hangs?
>
>  
>
Here's a tcpdump of the transfer (I figure that's what you mean with a 
pcap file). The last couple of packets keep repeating themselves so the 
line never goes quiet (until I ^C).
The transfer is a mv of a empty file named 'foo'.

Rgds
Pierre


[-- Attachment #2: nfsbug.dump --]
[-- Type: application/octet-stream, Size: 9604 bytes --]

Re: Move (mv) hangs with NFSv4

[J. Bruce Fields]

On Tue, Dec 07, 2004 at 10:11:37PM +0100, Pierre Ossman wrote:
> Here's a tcpdump of the transfer (I figure that's what you mean with a 
> pcap file). The last couple of packets keep repeating themselves so the 
> line never goes quiet (until I ^C).
> The transfer is a mv of a empty file named 'foo'.

Thanks.  It's a problem with the idmapping on the server, argh.  Is
rpc.idmapd still running after this, or does it die?--b.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Move (mv) hangs with NFSv4

[Pierre Ossman]

J. Bruce Fields wrote:

>On Tue, Dec 07, 2004 at 10:11:37PM +0100, Pierre Ossman wrote:
>  
>
>>Here's a tcpdump of the transfer (I figure that's what you mean with a 
>>pcap file). The last couple of packets keep repeating themselves so the 
>>line never goes quiet (until I ^C).
>>The transfer is a mv of a empty file named 'foo'.
>>    
>>
>
>Thanks.  It's a problem with the idmapping on the server, argh.  Is
>rpc.idmapd still running after this, or does it die?--b.
>  
>
It's still running. The effect is only local to the mv process. Other 
traffic to the same mount continue to work perfectly.  There has been a 
few incidents where the entire mount hangs. But in these cases I have to 
restart the client, not the server.

Can I get some debug output from idmapd that will pinpoint where the 
problem is?

Rgds
Pierre



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Move (mv) hangs with NFSv4

[Pierre Ossman]

Pierre Ossman wrote:
> J. Bruce Fields wrote:
> 
>>
>> Thanks.  It's a problem with the idmapping on the server, argh.  Is
>> rpc.idmapd still running after this, or does it die?--b.
>>  
>>
> It's still running. The effect is only local to the mv process. Other 
> traffic to the same mount continue to work perfectly.  There has been a 
> few incidents where the entire mount hangs. But in these cases I have to 
> restart the client, not the server.
> 
> Can I get some debug output from idmapd that will pinpoint where the 
> problem is?
> 

Any updates to this issue?

Rgds
Pierre


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Move (mv) hangs with NFSv4

[J. Bruce Fields]

On Sun, Jan 02, 2005 at 07:48:54PM +0100, Pierre Ossman wrote:
> >J. Bruce Fields wrote:
> >>Thanks.  It's a problem with the idmapping on the server, argh.  Is
> >>rpc.idmapd still running after this, or does it die?--b.
> >> 
> >>
> >It's still running. The effect is only local to the mv process. Other 
> >traffic to the same mount continue to work perfectly.  There has been a 
> >few incidents where the entire mount hangs. But in these cases I have to 
> >restart the client, not the server.
> >
> >Can I get some debug output from idmapd that will pinpoint where the 
> >problem is?
> >
> 
> Any updates to this issue?

Sorry for the overly long delay on the response.  Cleaning out my inbox
prompted me to look back at this issue.  I did find one bug in the ACL
code which could cause a "mv" with the target on an NFSv4 partition to
fail.  Other than that I can't reproduce any problems with move.

It's possible we've fixed something since the beginning of the year, or
it's possible something else is going on.  If you get the chance to
retest with our latest patches, let me know.

--b.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Move (mv) hangs with NFSv4

[Pierre Ossman]

J. Bruce Fields wrote:

>On Sun, Jan 02, 2005 at 07:48:54PM +0100, Pierre Ossman wrote:
>  
>
>>>J. Bruce Fields wrote:
>>>      
>>>
>>>>Thanks.  It's a problem with the idmapping on the server, argh.  Is
>>>>rpc.idmapd still running after this, or does it die?--b.
>>>>
>>>>
>>>>        
>>>>
>>>It's still running. The effect is only local to the mv process. Other 
>>>traffic to the same mount continue to work perfectly.  There has been a 
>>>few incidents where the entire mount hangs. But in these cases I have to 
>>>restart the client, not the server.
>>>
>>>Can I get some debug output from idmapd that will pinpoint where the 
>>>problem is?
>>>
>>>      
>>>
>>Any updates to this issue?
>>    
>>
>
>Sorry for the overly long delay on the response.  Cleaning out my inbox
>prompted me to look back at this issue.  I did find one bug in the ACL
>code which could cause a "mv" with the target on an NFSv4 partition to
>fail.  Other than that I can't reproduce any problems with move.
>
>It's possible we've fixed something since the beginning of the year, or
>it's possible something else is going on.  If you get the chance to
>retest with our latest patches, let me know.
>
>--b.
>  
>

Ehm... where would these patches be located? I didn't see anything
obvious on the web page for any recent kernel. Or perhaps you meant the
lats nfs-utils? or both? =)

Rgds
Pierre


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Move (mv) hangs with NFSv4

[J. Bruce Fields]

On Sun, Mar 13, 2005 at 05:36:37PM +0100, Pierre Ossman wrote:
> Ehm... where would these patches be located? I didn't see anything
> obvious on the web page for any recent kernel. Or perhaps you meant the
> lats nfs-utils? or both? =)

Sorry, I meant the latest citi kernel patch:

http://www.citi.umich.edu/projects/nfsv4/linux/kernel-patches/2.6.11-4/linux-2.6.11-CITI_NFS4_ALL-4.dif

--b.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Move (mv) hangs with NFSv4

[Pierre Ossman]

J. Bruce Fields wrote:

>On Sun, Mar 13, 2005 at 05:36:37PM +0100, Pierre Ossman wrote:
>  
>
>>Ehm... where would these patches be located? I didn't see anything
>>obvious on the web page for any recent kernel. Or perhaps you meant the
>>lats nfs-utils? or both? =)
>>    
>>
>
>Sorry, I meant the latest citi kernel patch:
>
>http://www.citi.umich.edu/projects/nfsv4/linux/kernel-patches/2.6.11-4/linux-2.6.11-CITI_NFS4_ALL-4.dif
>
>--b.
>  
>
I've tried the patch and the problem remains.

[root@apollo home]# ~/mv ~/freedesktop.org.xml .
Updating timestamps...done
Updating ownership...

(this is my hacked mv which lists each operation as it is performed)

Rgds
Pierre



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Move (mv) hangs with NFSv4

[J. Bruce Fields]

On Mon, Mar 14, 2005 at 05:03:16PM +0100, Pierre Ossman wrote:
> I've tried the patch and the problem remains.
> 
> [root@apollo home]# ~/mv ~/freedesktop.org.xml .
> Updating timestamps...done
> Updating ownership...
> 
> (this is my hacked mv which lists each operation as it is performed)

Hm.  Maybe my "mv" doesn't work quite the same as yours, but I'm not
seeing this.  Could I see a raw packet trace?  (tcpdump s0 -wtmp, send
me tmp.pcap).  Are you having any problems with rpc.idmapd dying on the
server?

--b.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Move (mv) hangs with NFSv4

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 1143 bytes --]

J. Bruce Fields wrote:

>On Mon, Mar 14, 2005 at 05:03:16PM +0100, Pierre Ossman wrote:
>  
>
>>I've tried the patch and the problem remains.
>>
>>[root@apollo home]# ~/mv ~/freedesktop.org.xml .
>>Updating timestamps...done
>>Updating ownership...
>>
>>(this is my hacked mv which lists each operation as it is performed)
>>    
>>
>
>Hm.  Maybe my "mv" doesn't work quite the same as yours, but I'm not
>seeing this.  Could I see a raw packet trace?  (tcpdump s0 -wtmp, send
>me tmp.pcap).  Are you having any problems with rpc.idmapd dying on the
>server?
>  
>

No change in behaviour from the last attempt. idmapd keeps running just
fine.
Dump from new run included.

Previous test:
http://marc.theaimsgroup.com/?l=linux-nfs&m=110245477428548&w=4
http://marc.theaimsgroup.com/?l=linux-nfs&m=110250639332016&w=4

As I was playing around with idmapd (after running the test) it started
to work just fine. I am unable to reproduce the error now, even though
idmapd is started the same way as originally. A reboot might get the
error back but I can't test that on this machine. I'll see if I can find
another machine to torture.

Rgds
Pierre

[-- Attachment #2: tmp.pcap --]
[-- Type: application/octet-stream, Size: 8166 bytes --]

Re: Move (mv) hangs with NFSv4

[J. Bruce Fields]

On Wed, Mar 16, 2005 at 01:32:50PM +0100, Pierre Ossman wrote:
> No change in behaviour from the last attempt. idmapd keeps running just
> fine.
> Dump from new run included.

OK, thanks for the data.  It still looks like a server-side idmapping
problem.  Next time it gets into this state, it would be interesting to
know whether sending rpc.idmapd a SIGHUP (or just restarting it) makes
any difference.  I'll try to find the time to investigate further
later this week....--b.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs