NFSv3, 2.6, ext3 and dir_index

NFSv3, 2.6, ext3 and dir_index

[Tobias Diedrich]

[-- Attachment #1: Type: text/plain, Size: 2246 bytes --]

Hi,

I recently switched from a local homedir to a NFS mounted homedir
and noticed some strange things:

I have a Cronjob that goes over my mp3 directory and after the
switch find would print "foo/bar/blubb: No such file or directory"
for a few (approx. 2-3 out of 1928) directories, but not always the same
ones.  Also, my mp3 player program would sometimes stop short in the
middle of a song and skip to the next one, because it got ENOENT on
a read of an open file.

Now first I thought it might just be a problem with my Kernel
version or one of the patches I'm using and upgraded to a more
recent one (currently 2.6.10-ac8-nfsacl on the client and
2.6.10-ac8-imq-nfsacl on the server, I also had the same problem
without the nfsacl patches).

Then I read an older thread about trouble with ext3/dir_index and
NFS in older 2.6 versions and tried disabling dir_index (Which was
enabled on all my ext3 filesystems).  With that the problem
vanished.

Are there any known problems with dir_index and NFS, or is this
maybe a new bug?

Getting a tcpdump of the client<->server traffic proved difficult,
because it is a quite sporadic bug and reproducing it would involve
quite a lot of NFS traffic.  I could reliably trigger this with my
CD/DVD burning script, which generates md5sums for each file and
puts them both into the file MD5SUMS in the current directory, as
well as into another file in my Homedir (But the files being
md5summed also live on another NFS export, so there is a _lot_ of
traffic).

I _do_ have a traffic capture of ls returning "No such file or
directory" on the current directory, then cding up a level, down into the
directory again and then a working ls.

Client fstab entry:
nukunuku:/mnt/space1/ranma /home/ranma nfs hard,intr,bg,udp,rsize=4096,wsize=4096 0 0

/proc/mounts entry:
nukunuku:/mnt/space1/ranma /home/ranma nfs rw,v3,rsize=4096,wsize=4096,hard,intr,udp,lock,addr=nukunuku 0 0

Server export file:
/ melchior.yamamaya.is-a-geek.org(sync,rw,no_root_squash)
/mnt/space1 melchior.yamamaya.is-a-geek.org(sync,rw,no_root_squash)
/mnt/space2 melchior.yamamaya.is-a-geek.org(sync,rw,no_root_squash)
/mnt/space3 melchior.yamamaya.is-a-geek.org(sync,rw,no_root_squash)

-- 
Tobias						PGP: http://9ac7e0bc.uguu.de

[-- Attachment #2: nfs.tcpdump.gz --]
[-- Type: application/octet-stream, Size: 5397 bytes --]

Re: NFSv3, 2.6, ext3 and dir_index

[Michael Haverkamp]

I have had the same problem with LVM2 and reiserfs using kernel 2.6.9. 
Whenever I have a large IO load on the server, these error crop up.  I 
copy everything on /space (/dev/hda5) to /space_bak (/dev/hdc5) using cp 
-a and that always makes the problem happen for me.  I have not tried 
removing LVM2 yet, so I don't know if that is a contributor.

I don't think that this problem is limited to ext3 and dir_index.

Tobias Diedrich wrote:
> Hi,
> 
> I recently switched from a local homedir to a NFS mounted homedir
> and noticed some strange things:
> 
> I have a Cronjob that goes over my mp3 directory and after the
> switch find would print "foo/bar/blubb: No such file or directory"
> for a few (approx. 2-3 out of 1928) directories, but not always the same
> ones.  Also, my mp3 player program would sometimes stop short in the
> middle of a song and skip to the next one, because it got ENOENT on
> a read of an open file.
> 
> Now first I thought it might just be a problem with my Kernel
> version or one of the patches I'm using and upgraded to a more
> recent one (currently 2.6.10-ac8-nfsacl on the client and
> 2.6.10-ac8-imq-nfsacl on the server, I also had the same problem
> without the nfsacl patches).
> 
> Then I read an older thread about trouble with ext3/dir_index and
> NFS in older 2.6 versions and tried disabling dir_index (Which was
> enabled on all my ext3 filesystems).  With that the problem
> vanished.
> 
> Are there any known problems with dir_index and NFS, or is this
> maybe a new bug?
> 
> Getting a tcpdump of the client<->server traffic proved difficult,
> because it is a quite sporadic bug and reproducing it would involve
> quite a lot of NFS traffic.  I could reliably trigger this with my
> CD/DVD burning script, which generates md5sums for each file and
> puts them both into the file MD5SUMS in the current directory, as
> well as into another file in my Homedir (But the files being
> md5summed also live on another NFS export, so there is a _lot_ of
> traffic).
> 
> I _do_ have a traffic capture of ls returning "No such file or
> directory" on the current directory, then cding up a level, down into the
> directory again and then a working ls.
> 
> Client fstab entry:
> nukunuku:/mnt/space1/ranma /home/ranma nfs hard,intr,bg,udp,rsize=4096,wsize=4096 0 0
> 
> /proc/mounts entry:
> nukunuku:/mnt/space1/ranma /home/ranma nfs rw,v3,rsize=4096,wsize=4096,hard,intr,udp,lock,addr=nukunuku 0 0
> 
> Server export file:
> / melchior.yamamaya.is-a-geek.org(sync,rw,no_root_squash)
> /mnt/space1 melchior.yamamaya.is-a-geek.org(sync,rw,no_root_squash)
> /mnt/space2 melchior.yamamaya.is-a-geek.org(sync,rw,no_root_squash)
> /mnt/space3 melchior.yamamaya.is-a-geek.org(sync,rw,no_root_squash)
> 

-- 
Michael Haverkamp


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

More Stale NFS handles

[Eiwe Lingefors]

I have trawled the nfs mailing list archives a bit and judging by  
recent posts there is undoubtedly several others with the same  
problem.   The problems started happening when I installed a new  
Fedora Core 3 server and migrated home directories to it. Like Jim  
Farley mentioned in a recent thread on the list, our environment has  
been stable for years prior.

The server:
Dell PowerEdge 2850
1 x PowerVault 220S 14x300GB SCSI
Fedora Core 3
Kernel 2.6.9-1.667smp
Several exported LVM volumes all formatted with reiserfs v3.6

The clients:
64 x IBM x335 cluster nodes
RedHat 7.3
Kernel 2.4.26

40 x Sun Blade 150
Solaris 9

Things I have done:
x Upgraded automount on all linux clients to 4.1.4
x Tuned rsize,wsize,timeo,retrans
x Increased number of nfsd processes to 128

So far nothing I have done has helped reduce the amount of stale NFS  
file handles.

I'm not sure what additional information might be helpful. The  
problems I'm having essentially mirror those of others who have  
posted regarding stale NFS handles in the past few months on this  
list. I'm at my wits end after having fought with this problem for  
weeks. Any insight or pointers would be deeply appreciated. I'll be  
happy to provide additional information if needed.

Thanks,
Eiwe Lingefors

PS. I have quoted Jim Farley's message below since the problems are  
essentially identical to mine.

Jim Farley wrote:

 > Hi,
 >
 > We have been running an nfs environment for a couple of years now  
without
 > difficulties. Unfortunately when we upgraded the nfs server from  
RedHat9 to Fedora
 > Core 3 (2.6.9-1.667smp), clients (10 RedHat 9 systems,  
2.4.20-8smp) started getting
 > stale NFS file handles. I used tcpdump and verified that the  
messages are exchanged
 > between the server and client without delay which would indicate  
the network is not a
 > problem. I have also verified that the server is not under heavy  
load (cpu, memory,
 > network), nor are the clients.
 > We basically take the defaults on the client and on the server:
 >
 > Client:
 > cat /proc/mounts:
 > 192.168.11.10:/home /home nfs
 > rw,v3,rsize=4096,wsize=4096,hard,intr,udp,lock,addr=192.168.11.10 0 0
 >
 > Server:
 > cat /etc/exports
 > /home 192.168.11.0/255.255.255.0(rw,no_subtree_check)
 >
 > What I have done:
 > - increase number of nfsd's to 32
 > - disabled caching on one client (noac)
 > - reverted the server from the ext3 filesystem to ext2
 >
 > None of this has helped. Is there anything else I can try?
 >
 > Thanks,
 > Jimmy



-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: More Stale NFS handles

[Kris Vassallo]

[-- Attachment #1: Type: text/plain, Size: 1814 bytes --]

On Thu, 2005-06-23 at 13:55, Eiwe Lingefors wrote:

> The server:
> Dell PowerEdge 2850
> 1 x PowerVault 220S 14x300GB SCSI
> Fedora Core 3
> Kernel 2.6.9-1.667smp
> Several exported LVM volumes all formatted with reiserfs v3.6
> So far nothing I have done has helped reduce the amount of stale NFS  
> file handles.

Bah! I have the same problem, I've had it for months now. I was about to
build yet another server and use reiserfs but thanks to you I am spared
the agony of finding out it wont work. 

> 
> I'm not sure what additional information might be helpful. The  
> problems I'm having essentially mirror those of others who have  
> posted regarding stale NFS handles in the past few months on this  
> list. I'm at my wits end after having fought with this problem for  
> weeks.

YES YES YES!!! It almost brings a state of insanity with it!! Plus, atop
my insanity I have a group of 30 developers who are complaining about
not being able to do work because of stale file handles. 

>  Any insight or pointers would be deeply appreciated. I'll be  
> happy to provide additional information if needed.

I was using ext3, upgraded to core 3 from core 1 and had the exact same
problems. What did it for me, until someone fixes this problem, was to
turn off journaling (so basically its back to ext2). Since reverting
back to ext 2 the problem has gone away. Now this is going to be a huge
problem if the machine crashes because fscking a 1.5 TB disk array is
going to suck! I experimented with the way the journaling gets done
(data goes to disk first or journal first) and I wasn't able to fix the
problem. 

Aaaaaaargh! 
I just thought I would share my ongoing battle story, hopefully someone
will figure out what's causing this and will come up with a fix.

-Kris


> 
> Thanks,
> Eiwe Lingefors
> 



[-- Attachment #2: Type: text/html, Size: 2368 bytes --]

NFSv3, 2.6, ext3 and dir_index

[Tobias Diedrich]

[-- Attachment #1: Type: text/plain, Size: 2246 bytes --]

Hi,

I recently switched from a local homedir to a NFS mounted homedir
and noticed some strange things:

I have a Cronjob that goes over my mp3 directory and after the
switch find would print "foo/bar/blubb: No such file or directory"
for a few (approx. 2-3 out of 1928) directories, but not always the same
ones.  Also, my mp3 player program would sometimes stop short in the
middle of a song and skip to the next one, because it got ENOENT on
a read of an open file.

Now first I thought it might just be a problem with my Kernel
version or one of the patches I'm using and upgraded to a more
recent one (currently 2.6.10-ac8-nfsacl on the client and
2.6.10-ac8-imq-nfsacl on the server, I also had the same problem
without the nfsacl patches).

Then I read an older thread about trouble with ext3/dir_index and
NFS in older 2.6 versions and tried disabling dir_index (Which was
enabled on all my ext3 filesystems).  With that the problem
vanished.

Are there any known problems with dir_index and NFS, or is this
maybe a new bug?

Getting a tcpdump of the client<->server traffic proved difficult,
because it is a quite sporadic bug and reproducing it would involve
quite a lot of NFS traffic.  I could reliably trigger this with my
CD/DVD burning script, which generates md5sums for each file and
puts them both into the file MD5SUMS in the current directory, as
well as into another file in my Homedir (But the files being
md5summed also live on another NFS export, so there is a _lot_ of
traffic).

I _do_ have a traffic capture of ls returning "No such file or
directory" on the current directory, then cding up a level, down into the
directory again and then a working ls.

Client fstab entry:
nukunuku:/mnt/space1/ranma /home/ranma nfs hard,intr,bg,udp,rsize=4096,wsize=4096 0 0

/proc/mounts entry:
nukunuku:/mnt/space1/ranma /home/ranma nfs rw,v3,rsize=4096,wsize=4096,hard,intr,udp,lock,addr=nukunuku 0 0

Server export file:
/ melchior.yamamaya.is-a-geek.org(sync,rw,no_root_squash)
/mnt/space1 melchior.yamamaya.is-a-geek.org(sync,rw,no_root_squash)
/mnt/space2 melchior.yamamaya.is-a-geek.org(sync,rw,no_root_squash)
/mnt/space3 melchior.yamamaya.is-a-geek.org(sync,rw,no_root_squash)

-- 
Tobias						PGP: http://9ac7e0bc.uguu.de

[-- Attachment #2: nfs.tcpdump.gz --]
[-- Type: application/octet-stream, Size: 5397 bytes --]

Re: NFSv3, 2.6, ext3 and dir_index

[Chip Salzenberg]

According to Tobias Diedrich:
> Are there any known problems with dir_index and NFS, or is this
> maybe a new bug?

Did you ever get any replies on dir_index vs. nfs?

I'm just setting up a new server and I'm wondering if there's
something still out there that might make me sorry to use dir_index.
-- 
Chip Salzenberg            - a.k.a. -            <chip@pobox.com>
         Open Source is not an excuse to write fun code
            then leave the actual work to others.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFSv3, 2.6, ext3 and dir_index

[Tobias Diedrich]

Sorry for the late reply.

Chip Salzenberg wrote:

> Did you ever get any replies on dir_index vs. nfs?

Not really.

> I'm just setting up a new server and I'm wondering if there's
> something still out there that might make me sorry to use dir_index.

For now I've just disabled dir_index and I haven't had any problems
since, but I'm not completely sure if that really was the cause of
the problem I was seeing.  At least disabling dir_index is rather
easy (remove dir_index featureflag with tune2fs and run e2fsck to
get rid of the remaining hashtrees).
Maybe I'll try again to get a proper traffic trace, but I guess I'd
have to set up a test client first to separate the unrelated nfs traffic.

-- 
Tobias						PGP: http://9ac7e0bc.uguu.de


-------------------------------------------------------
This SF.net email is sponsored by Microsoft Mobile & Embedded DevCon 2005
Attend MEDC 2005 May 9-12 in Vegas. Learn more about the latest Windows
Embedded(r) & Windows Mobile(tm) platforms, applications & content.  Register
by 3/29 & save $300 http://ads.osdn.com/?ad_id=6883&alloc_id=15149&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFSv3, 2.6, ext3 and dir_index

[Tobias Diedrich]

Followup for people looking at this thread in the archive:

The patch mentioned in Bills Message seems to be in 2.6.13 release candidates,
however my server is still running 2.6.11.12 (without the patch),
but I'm about to upgrade to 2.6.12.4 (with the patch applied).
Hopefully this will fix both errors I was seeing. :-)

Tobias Diedrich wrote on 20 Mar 2005 in
<20050320212106.GA5580@yamamaya.is-a-geek.org>:
> Sorry for the late reply.
> 
> Chip Salzenberg wrote:
> 
> > Did you ever get any replies on dir_index vs. nfs?
> 
> Not really.
> 
> > I'm just setting up a new server and I'm wondering if there's
> > something still out there that might make me sorry to use dir_index.
> 
> For now I've just disabled dir_index and I haven't had any problems
> since, but I'm not completely sure if that really was the cause of
> the problem I was seeing.  At least disabling dir_index is rather
> easy (remove dir_index featureflag with tune2fs and run e2fsck to
> get rid of the remaining hashtrees).
> Maybe I'll try again to get a proper traffic trace, but I guess I'd
> have to set up a test client first to separate the unrelated nfs traffic.
> 
> -- 
> Tobias						PGP: http://9ac7e0bc.uguu.de

Bill Rugolsky Jr. wrote on 29 Jun 2005 in
<20050629205056.GB29952@ti64.telemetry-investments.com>:
> On Wed, Jun 29, 2005 at 01:28:07PM -0700, Kris Vassallo wrote:
> > I was using ext3, upgraded to core 3 from core 1 and had the exact same
> > problems. What did it for me, until someone fixes this problem, was to
> > turn off journaling (so basically its back to ext2). Since reverting
> > back to ext 2 the problem has gone away. Now this is going to be a huge
> > problem if the machine crashes because fscking a 1.5 TB disk array is
> > going to suck! I experimented with the way the journaling gets done
> > (data goes to disk first or journal first) and I wasn't able to fix the
> > problem. 
> > 
> > Aaaaaaargh! 
> > I just thought I would share my ongoing battle story, hopefully someone
> > will figure out what's causing this and will come up with a fix.
> 
> We're you using ext3 directory indexing? [Look for feature "dir_index" on
> the "Filesystem features:" line in "/sbin/tune2fs -l <device>" output.]
> 
> If so, was the following patch in the kernel that you were using?
> 
>    http://marc.theaimsgroup.com/?l=ext2-devel&m=111568753527230&w=2
> 
> Regards,
> 
> 	Bill Rugolsky

-- 
Tobias						PGP: http://9ac7e0bc.uguu.de


-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: More Stale NFS handles

[Bill Rugolsky Jr.]

On Wed, Jun 29, 2005 at 01:28:07PM -0700, Kris Vassallo wrote:
> I was using ext3, upgraded to core 3 from core 1 and had the exact same
> problems. What did it for me, until someone fixes this problem, was to
> turn off journaling (so basically its back to ext2). Since reverting
> back to ext 2 the problem has gone away. Now this is going to be a huge
> problem if the machine crashes because fscking a 1.5 TB disk array is
> going to suck! I experimented with the way the journaling gets done
> (data goes to disk first or journal first) and I wasn't able to fix the
> problem. 
> 
> Aaaaaaargh! 
> I just thought I would share my ongoing battle story, hopefully someone
> will figure out what's causing this and will come up with a fix.

We're you using ext3 directory indexing? [Look for feature "dir_index" on
the "Filesystem features:" line in "/sbin/tune2fs -l <device>" output.]

If so, was the following patch in the kernel that you were using?

   http://marc.theaimsgroup.com/?l=ext2-devel&m=111568753527230&w=2

Regards,

	Bill Rugolsky


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs