Strange XFS issue on tiny-NAS ARM NFS server

Strange XFS issue on tiny-NAS ARM NFS server

[Talpey, Thomas]

This might be pilot error, but a *very* strange thing happens with
an XFS filesystem on an NFS server I'm experimenting with. This is
an NSLU2 ARM-based machine, running 2.6.20.1 and an XFS filesystem
freshly built on a usb-attached 2.5" drive.

Running Connectathon 04 basic tests against the server, things are
fine with an EXT-formatted filesystem. However, reformatting the
export as a default XFS filesystem (mkfs.xfs -f /dev/sda3), the
following occurs:

>[tmt@tmt2 cthon04]$ ./server -b -p /mnt/export -m /mnt 192.168.1.77
>Start tests on path /mnt/tmt2.test [y/n]? y
>
>sh ./runtests  -b -t /mnt/tmt2.test
>
>Starting BASIC tests: test directory /mnt/tmt2.test (arg: -t)
>
>./test1: File and directory creation test
>        created 155 files 62 directories 5 levels deep in 0.78 seconds
>        ./test1 ok.
>
>./test2: File and directory removal test
>        removed 155 files 62 directories 5 levels deep in 0.60 seconds
>        ./test2 ok.
>
>./test3: lookups across mount point
>        500 getcwd and stat calls in 0.0  seconds
>        ./test3 ok.
>
>./test4: setattr, getattr, and lookup
>        1000 chmods and stats on 10 files in 0.80 seconds
>        ./test4 ok.
>
>./test5: read and write
>rm: cannot remove `/mnt/tmt2.test/file.7': No such file or directory
>rm: cannot remove `/mnt/tmt2.test/file.8': No such file or directory
>rm: cannot remove `/mnt/tmt2.test/file.9': No such file or directory
>        ./test5: (/home/tmt/nfs/cthon04/basic) can't remove old test directory /mnt/tmt2.test
>basic tests failed
>Tests failed, leaving /mnt mounted
>[tmt@tmt2 cthon04]$ ls -lsa /mnt/tmt2.test
>total 0
>0 drwxrwxrwx 2 tmt  tmt  17 Mar 14 11:23 
>0 drwxrwxrwx 2 tmt  tmt  17 Mar 14 11:23 
>0 drwxrwxrwx 2 tmt  tmt  17 Mar 14 11:23 .
>0 drwxrwxrwx 3 root root 32 Mar 14 11:23 ..
>[tmt@tmt2 cthon04]$ 


Those first two entries are entirely null - piping the output
to "od" shows no filename at all. The same result is seen if
listed from a login shell on the server.

The problem stems from connectathon test4, which is attemting
to create 10 files and chmod them repeatedly. If the test is
run with a file count less than 8, it works fine. If >= 8,
then upon removing the 6th file (file.5), the remaining files
vanish, and the blank entry appears. There are no complaints
in dmesg (see below fyi). The behavior is the same across all
client mount options and server export options, only going
away by reformatting the export to ext.

Before giving the problem report a full work-up, I'm wondering
if I'm missing the obvious, or if it's a known issue. A look
around with google (etc) didn't turn up anything. Thanks for
any info.

(please include me in the reply, I'm not on the list).

Tom.

(dmesg)
>...
><6>SGI XFS with no debug enabled
><5>XFS mounting filesystem sda3
><7>Ending clean XFS mount for filesystem: sda3
><6>Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
><4>NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
><4>NFSD: starting 90-second grace period
>root@LKG881990:/$ 
>root@LKG881990:/$ cat /proc/fs/xfs/stat
>extent_alloc 19 73 18 69
>abt 119 87 43 41
>blk_map 8 1 1 1 1 11 0
>bmbt 0 0 0 0
>dir 233 232 230 10
>trans 0 1726 0
>ig 2659 2426 0 233 0 227 1002
>log 67 2010 0 11 1
>push_ail 1729 0 0 0 0 0 0 0 0 0
>xstrat 0 0
>rw 0 0
>attr 0 0 0 0
>icluster 5 2 19
>vnodes 6 233 0 1382 227 227 227 0
>buf 1940 239 1709 0 0 0 0 231 64
>xpc 0 0 0
>debug 0
>root@LKG881990:/$ mkfs.xfs -v
>mkfs.xfs version 2.8.16
>root@LKG881990:/$ 

Re: Strange XFS issue on tiny-NAS ARM NFS server

[Eric Sandeen]

Talpey, Thomas wrote:
> This might be pilot error, but a *very* strange thing happens with
> an XFS filesystem on an NFS server I'm experimenting with. This is
> an NSLU2 ARM-based machine, running 2.6.20.1 and an XFS filesystem
> freshly built on a usb-attached 2.5" drive.
> 
> Running Connectathon 04 basic tests against the server, things are
> fine with an EXT-formatted filesystem. However, reformatting the
> export as a default XFS filesystem (mkfs.xfs -f /dev/sda3), the
> following occurs:

arm compiler has bugs that miscompile xfs... I think if you google arm + 
xfs and maybe search the list archives, you'll find a possible workaround.

-Eric

Re: Strange XFS issue on tiny-NAS ARM NFS server

[Talpey, Thomas]

Wow good memory, 3 years ago:
<http://lists.arm.linux.org.uk/pipermail/linux-arm-kernel/2004-March/020287.html>

I'm compiling with gcc4.1.1 for arm5t big endian, patched for multiple
arm ports (OpenEmbedded) and it still botches that code today.
Works fine with the arithmetic decomposition in the message.
And I never blame the compiler! ;-)

Thanks!

BTW, XFS gives this little machine a nice bump in NFS write
bandwidth. Goes from ~6MB/sec to ~7MB/s. CPU limited, mainly.

Tom.

At 04:32 PM 3/14/2007, Eric Sandeen wrote:
>Talpey, Thomas wrote:
>> This might be pilot error, but a *very* strange thing happens with
>> an XFS filesystem on an NFS server I'm experimenting with. This is
>> an NSLU2 ARM-based machine, running 2.6.20.1 and an XFS filesystem
>> freshly built on a usb-attached 2.5" drive.
>> 
>> Running Connectathon 04 basic tests against the server, things are
>> fine with an EXT-formatted filesystem. However, reformatting the
>> export as a default XFS filesystem (mkfs.xfs -f /dev/sda3), the
>> following occurs:
>
>arm compiler has bugs that miscompile xfs... I think if you google arm + 
>xfs and maybe search the list archives, you'll find a possible workaround.
>
>-Eric
>
>

Re: Strange XFS issue on tiny-NAS ARM NFS server

[Eric Sandeen]

Talpey, Thomas wrote:
> Wow good memory, 3 years ago:
> <http://lists.arm.linux.org.uk/pipermail/linux-arm-kernel/2004-March/020287.html>
> 
> I'm compiling with gcc4.1.1 for arm5t big endian, patched for multiple
> arm ports (OpenEmbedded) and it still botches that code today.
> Works fine with the arithmetic decomposition in the message.
> And I never blame the compiler! ;-)
> 
> Thanks!
> 
> BTW, XFS gives this little machine a nice bump in NFS write
> bandwidth. Goes from ~6MB/sec to ~7MB/s. CPU limited, mainly.

Glad it worked!

So is netapp selling those now?  ;-)

-Eric

Re: Strange XFS issue on tiny-NAS ARM NFS server

[Talpey, Thomas]

At 05:21 PM 3/14/2007, Eric Sandeen wrote:
>Glad it worked!
>
>So is netapp selling those now?  ;-)

Nope, but they sure are instructive platforms for expermenting
with Linux. Their memory bandwidth is so low (and they only
have 32MB of it) that little things make a big difference. 

Tom.

Re: Strange XFS issue on tiny-NAS ARM NFS server

[Talpey, Thomas]

Evidently that was not the only compiling issue (surprise):

><6>attempt to access beyond end of device
><6>sda3: rw=2, want=2574098408, limit=154272195
><6>attempt to access beyond end of device
><6>sda3: rw=2, want=2574098408, limit=154272195
><6>attempt to access beyond end of device
><6>sda3: rw=2, want=2574098408, limit=154272195
><6>attempt to access beyond end of device
><6>sda3: rw=0, want=2574098408, limit=154272195
><1>I/O error in filesystem ("sda3") meta-data dev sda3 block 0x996d9fe0       ("xfs_trans_read_buf") error 5 buf count 4096

Oh, well.

Tom.


At 05:19 PM 3/14/2007, Talpey, Thomas wrote:
>Wow good memory, 3 years ago:
><http://lists.arm.linux.org.uk/pipermail/linux-arm-kernel/2004-March/0>20287.html>
>
>I'm compiling with gcc4.1.1 for arm5t big endian, patched for multiple
>arm ports (OpenEmbedded) and it still botches that code today.
>Works fine with the arithmetic decomposition in the message.
>And I never blame the compiler! ;-)
>
>Thanks!
>
>BTW, XFS gives this little machine a nice bump in NFS write
>bandwidth. Goes from ~6MB/sec to ~7MB/s. CPU limited, mainly.
>
>Tom.
>
>At 04:32 PM 3/14/2007, Eric Sandeen wrote:
>>Talpey, Thomas wrote:
>>> This might be pilot error, but a *very* strange thing happens with
>>> an XFS filesystem on an NFS server I'm experimenting with. This is
>>> an NSLU2 ARM-based machine, running 2.6.20.1 and an XFS filesystem
>>> freshly built on a usb-attached 2.5" drive.
>>> 
>>> Running Connectathon 04 basic tests against the server, things are
>>> fine with an EXT-formatted filesystem. However, reformatting the
>>> export as a default XFS filesystem (mkfs.xfs -f /dev/sda3), the
>>> following occurs:
>>
>>arm compiler has bugs that miscompile xfs... I think if you google arm + 
>>xfs and maybe search the list archives, you'll find a possible workaround.
>>
>>-Eric
>>
>>

Re: Strange XFS issue on tiny-NAS ARM NFS server

[David Chinner]

On Thu, Mar 15, 2007 at 07:34:37AM -0400, Talpey, Thomas wrote:
> Evidently that was not the only compiling issue (surprise):
> 
> ><6>attempt to access beyond end of device
> ><6>sda3: rw=2, want=2574098408, limit=154272195
> ><6>attempt to access beyond end of device
> ><6>sda3: rw=2, want=2574098408, limit=154272195
> ><6>attempt to access beyond end of device
> ><6>sda3: rw=2, want=2574098408, limit=154272195
> ><6>attempt to access beyond end of device
> ><6>sda3: rw=0, want=2574098408, limit=154272195
> ><1>I/O error in filesystem ("sda3") meta-data dev sda3 block 0x996d9fe0       ("xfs_trans_read_buf") error 5 buf count 4096

That's a long way past the end of the partition.

Tom, did you run xfs_repair on that filesystem after
running with a busted compiler? Who knows how
it broke stuff on disk.....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: Strange XFS issue on tiny-NAS ARM NFS server

[Talpey, Thomas]

At 06:59 PM 3/15/2007, David Chinner wrote:
>On Thu, Mar 15, 2007 at 07:34:37AM -0400, Talpey, Thomas wrote:
>> Evidently that was not the only compiling issue (surprise):
>> 
>> ><6>attempt to access beyond end of device
>> ><6>sda3: rw=2, want=2574098408, limit=154272195...
>That's a long way past the end of the partition.
>
>Tom, did you run xfs_repair on that filesystem after
>running with a busted compiler? Who knows how
>it broke stuff on disk.....

No, this was a freshly rebuilt ~80GB xfs. I was running a test on
it over NFS which wrote and read 500MB files repeatedly. And, it
ran for quite some time before failing. I haven't had a chance to
track it down at all.

Tom.