* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-02 0:45 ` XFS filesystem corruption on the arm(el) architecture Dave Chinner
@ 2008-10-02 0:56 ` Eric Sandeen
2008-10-02 1:17 ` Eric Sandeen
2008-10-02 1:42 ` Eric Sandeen
2 siblings, 0 replies; 21+ messages in thread
From: Eric Sandeen @ 2008-10-02 0:56 UTC (permalink / raw)
To: Tobias Frost, linux-kernel, debian-arm, xfs
Dave Chinner wrote:
> Adding xfs@oss.sgi.com to the cc list so all the XFS folk see this.
>
> On Wed, Oct 01, 2008 at 10:38:22PM +0200, Tobias Frost wrote:
>> (Note: Please CC me, as I am NOT on the lkml!!)
>>
>> Some time ago, I discovered some problems with xfs. Unfortunatly, I had
>> no time diving into it. However, some weeks ago some other people
>> running debian on ARM machines confirmed the problem on their machines
>> starting at [1], so I think it is appropitate to at least report it.
>> It has also been seen on 2.6.27-rc4 [2].
>>
>> summary: the xfs partition corrupts almost immediatly after creation. I
>> had the impression, that the first unlink (rm) causes the corruption,
>> but this might be just an impression.
>>
>> During the tests I made, I conserved a image of the corrupted filesystem
>> which I can make available on request (it's 26 Mbyte, gzipped).
>>
>> Please let me know how I can assist you in finding the problem.
>>
>>
>> [1] http://lists.debian.org/debian-arm/2008/08/msg00155.html
>> [2] http://lists.debian.org/debian-arm/2008/08/msg00184.html
>>
>> Best regards,
>> Tobias Frost
>> http://blog.coldtobi.de
>>
>> PS: Thank you for your great work!
At one point there were other patches floating around to "fix" arm which
were not correct. Are these problems seen on a pristine 2.6.27-rc4
kernel, or with other special arm patches applied?
-Eric
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-02 0:45 ` XFS filesystem corruption on the arm(el) architecture Dave Chinner
2008-10-02 0:56 ` Eric Sandeen
@ 2008-10-02 1:17 ` Eric Sandeen
2008-10-16 21:25 ` Martin Michlmayr
2008-10-02 1:42 ` Eric Sandeen
2 siblings, 1 reply; 21+ messages in thread
From: Eric Sandeen @ 2008-10-02 1:17 UTC (permalink / raw)
To: Tobias Frost, linux-kernel, debian-arm, xfs
Dave Chinner wrote:
> Adding xfs@oss.sgi.com to the cc list so all the XFS folk see this.
>
> On Wed, Oct 01, 2008 at 10:38:22PM +0200, Tobias Frost wrote:
>> (Note: Please CC me, as I am NOT on the lkml!!)
>>
>> Some time ago, I discovered some problems with xfs. Unfortunatly, I had
>> no time diving into it. However, some weeks ago some other people
>> running debian on ARM machines confirmed the problem on their machines
>> starting at [1], so I think it is appropitate to at least report it.
>> It has also been seen on 2.6.27-rc4 [2].
>>
>> summary: the xfs partition corrupts almost immediatly after creation. I
>> had the impression, that the first unlink (rm) causes the corruption,
>> but this might be just an impression.
>>
>> During the tests I made, I conserved a image of the corrupted filesystem
>> which I can make available on request (it's 26 Mbyte, gzipped).
>>
>> Please let me know how I can assist you in finding the problem.
>>
>>
>> [1] http://lists.debian.org/debian-arm/2008/08/msg00155.html
>> [2] http://lists.debian.org/debian-arm/2008/08/msg00184.html
>>
>> Best regards,
>> Tobias Frost
>> http://blog.coldtobi.de
You could try undoing this:
/* ARM old ABI has some weird alignment/padding */
#if defined(__arm__) && !defined(__ARM_EABI__)
#define __arch_pack __attribute__((packed))
#else
#define __arch_pack
#endif
and just define __arch_pack to nothing unconditionally, to see if that's
what broke...
Or if someone can provide an xfs.ko, or point "pahole" at it yourself,
and see if xfs_dir2_sf_hdr, xfs_dir2_sf_entry, and xfs_dir2_sf look more
like
http://sandeen.net/xfs-diskformat/xfs-oldabi-arm-pahole-structs
or
http://sandeen.net/xfs-diskformat/xfs-oldabi-fixed-arm-pahole-structs
?
otherwise I will give this a whirl in the emulator again...
Thanks,
-Eric
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-02 1:17 ` Eric Sandeen
@ 2008-10-16 21:25 ` Martin Michlmayr
2008-10-16 22:13 ` Eric Sandeen
0 siblings, 1 reply; 21+ messages in thread
From: Martin Michlmayr @ 2008-10-16 21:25 UTC (permalink / raw)
To: Eric Sandeen; +Cc: Tobias Frost, linux-kernel, debian-arm, xfs
Hi Eric,
I tried to reproduce this problem on my ARM machine and it's really
easy to trigger. See the transcript below.
I tried with 2.6.26.6 (without the ARM old ABI fix) and 2.6.27 (with
the fix), and with xfsprogs 2.9.8-1.
Note that I'm actually using the ARM EABI, and not the old ABI.
I'm not sure what Tobias used.
xfs.ko compiled with -g can be found at http://www.cyrius.com/tmp/xfs.ko.bz2
(3.1 MB)
Here's the transcript. It's really easy to trigger. Just copy some
files to the XFS partition (works) and then run 'ls' (oops):
debian:~# modprobe xfs
debian:~# mkfs.xfs -f /dev/sda6
meta-data=/dev/sda6 isize=256 agcount=4, agsize=17778431 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=71113722, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096
log =internal log bsize=4096 blocks=32768, version=2
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0
debian:~# dmesg | tail -n 2
[42949548.970000] SGI XFS with ACLs, security attributes, realtime, large block numbers, no debug enabled
[42949548.980000] SGI XFS Quota Management subsystem
debian:~# mount /dev/sda6 /mnt
debian:~# dmesg | tail -n 2
[42949596.470000] XFS mounting filesystem sda6
[42949596.610000] Ending clean XFS mount for filesystem: sda6
debian:~# cp /usr/bin/* /mnt/
debian:~# dmesg | tail -n 2
[42949596.470000] XFS mounting filesystem sda6
[42949596.610000] Ending clean XFS mount for filesystem: sda6
debian:~# ls /mnt
ls: reading directory /mnt: Structure needs cleaning
debian:~# dmesg | tail -n 16
[42949596.610000] Ending clean XFS mount for filesystem: sda6
[42949619.790000] 00000000: 58 46 53 42 00 00 10 00 00 00 00 00 04 3d 1b fa XFSB.........=..
[42949619.800000] Filesystem "sda6": XFS internal error xfs_da_do_buf(2) at line 2107 of file fs/xfs/xfs_da_btree.c. Caller 0xbf148b44
[42949619.820000] [<c002a370>] (dump_stack+0x0/0x14) from [<bf154968>] (xfs_error_report+0x4c/0x5c [xfs])
[42949619.820000] [<bf15491c>] (xfs_error_report+0x0/0x5c [xfs]) from [<bf1549d4>] (xfs_corruption_error+0x5c/0x68 [xfs])
[42949619.830000] r4:c7914400
[42949619.840000] [<bf154978>] (xfs_corruption_error+0x0/0x68 [xfs]) from [<bf1489b8>] (xfs_da_do_buf+0x554/0x654 [xfs])
[42949619.850000] r6:bf148b44 r5:00000000 r4:c7073418
[42949619.850000] [<bf148464>] (xfs_da_do_buf+0x0/0x654 [xfs]) from [<bf148b44>] (xfs_da_read_buf+0x34/0x3c [xfs])
[42949619.860000] [<bf148b10>] (xfs_da_read_buf+0x0/0x3c [xfs]) from [<bf14edec>] (xfs_dir2_leaf_getdents+0x480/0x8b4 [xfs])
[42949619.880000] [<bf14e96c>] (xfs_dir2_leaf_getdents+0x0/0x8b4 [xfs]) from [<bf14b07c>] (xfs_readdir+0xcc/0xe0 [xfs])
[42949619.890000] [<bf14afb0>] (xfs_readdir+0x0/0xe0 [xfs]) from [<bf18140c>] (xfs_file_readdir+0x144/0x194 [xfs])
[42949619.900000] [<bf1812c8>] (xfs_file_readdir+0x0/0x194 [xfs]) from [<c009ffb0>] (vfs_readdir+0x84/0xb8)
[42949619.910000] [<c009ff2c>] (vfs_readdir+0x0/0xb8) from [<c00a0050>] (sys_getdents64+0x6c/0xc0)
[42949619.920000] [<c009ffe4>] (sys_getdents64+0x0/0xc0) from [<c0025bc0>] (ret_fast_syscall+0x0/0x3c)
[42949619.930000] r7:000000d9 r6:0002a01c r5:0002a030 r4:00000000
debian:~#
--
Martin Michlmayr
http://www.cyrius.com/
^ permalink raw reply [flat|nested] 21+ messages in thread* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-16 21:25 ` Martin Michlmayr
@ 2008-10-16 22:13 ` Eric Sandeen
2008-10-17 7:01 ` Martin Michlmayr
0 siblings, 1 reply; 21+ messages in thread
From: Eric Sandeen @ 2008-10-16 22:13 UTC (permalink / raw)
To: Martin Michlmayr; +Cc: Tobias Frost, linux-kernel, debian-arm, xfs
Martin Michlmayr wrote:
> Hi Eric,
>
> I tried to reproduce this problem on my ARM machine and it's really
> easy to trigger. See the transcript below.
>
> I tried with 2.6.26.6 (without the ARM old ABI fix) and 2.6.27 (with
> the fix), and with xfsprogs 2.9.8-1.
>
> Note that I'm actually using the ARM EABI, and not the old ABI.
> I'm not sure what Tobias used.
>
> xfs.ko compiled with -g can be found at http://www.cyrius.com/tmp/xfs.ko.bz2
> (3.1 MB)
Thanks; a quick look at the disk structure sizes & offsets shows no
differences (as I'd hope/expect for ARM EABI).
> Here's the transcript. It's really easy to trigger. Just copy some
> files to the XFS partition (works) and then run 'ls' (oops):
So is this a regression? did it used to work? If so, when? :)
(just for the record; it didn't oops, it shut down the filesystem and
gave you a backtrace to the error...)
It's trying to get a buffer for a directory leaf block from disk, and
it's finding that the magic number is bad.
What's a little odd is that the buffer it dumped out looks like the
beginning of a perfectly valid superblock for your filesystem (magic,
block size, and block count all match). If you printk the "bno"
variable right around line 2106 in xfs_da_btree.c, can you see what you get?
creating an xfs_metadump of the filesystem for examination on a non-arm
box might also be interesting.
Thanks,
-Eric
> debian:~# modprobe xfs
> debian:~# mkfs.xfs -f /dev/sda6
> meta-data=/dev/sda6 isize=256 agcount=4, agsize=17778431 blks
> = sectsz=512 attr=2
> data = bsize=4096 blocks=71113722, imaxpct=25
> = sunit=0 swidth=0 blks
> naming =version 2 bsize=4096
> log =internal log bsize=4096 blocks=32768, version=2
> = sectsz=512 sunit=0 blks, lazy-count=0
> realtime =none extsz=4096 blocks=0, rtextents=0
> debian:~# dmesg | tail -n 2
> [42949548.970000] SGI XFS with ACLs, security attributes, realtime, large block numbers, no debug enabled
> [42949548.980000] SGI XFS Quota Management subsystem
> debian:~# mount /dev/sda6 /mnt
> debian:~# dmesg | tail -n 2
> [42949596.470000] XFS mounting filesystem sda6
> [42949596.610000] Ending clean XFS mount for filesystem: sda6
> debian:~# cp /usr/bin/* /mnt/
> debian:~# dmesg | tail -n 2
> [42949596.470000] XFS mounting filesystem sda6
> [42949596.610000] Ending clean XFS mount for filesystem: sda6
> debian:~# ls /mnt
> ls: reading directory /mnt: Structure needs cleaning
> debian:~# dmesg | tail -n 16
> [42949596.610000] Ending clean XFS mount for filesystem: sda6
> [42949619.790000] 00000000: 58 46 53 42 00 00 10 00 00 00 00 00 04 3d 1b fa XFSB.........=..
> [42949619.800000] Filesystem "sda6": XFS internal error xfs_da_do_buf(2) at line 2107 of file fs/xfs/xfs_da_btree.c. Caller 0xbf148b44
> [42949619.820000] [<c002a370>] (dump_stack+0x0/0x14) from [<bf154968>] (xfs_error_report+0x4c/0x5c [xfs])
> [42949619.820000] [<bf15491c>] (xfs_error_report+0x0/0x5c [xfs]) from [<bf1549d4>] (xfs_corruption_error+0x5c/0x68 [xfs])
> [42949619.830000] r4:c7914400
> [42949619.840000] [<bf154978>] (xfs_corruption_error+0x0/0x68 [xfs]) from [<bf1489b8>] (xfs_da_do_buf+0x554/0x654 [xfs])
> [42949619.850000] r6:bf148b44 r5:00000000 r4:c7073418
> [42949619.850000] [<bf148464>] (xfs_da_do_buf+0x0/0x654 [xfs]) from [<bf148b44>] (xfs_da_read_buf+0x34/0x3c [xfs])
> [42949619.860000] [<bf148b10>] (xfs_da_read_buf+0x0/0x3c [xfs]) from [<bf14edec>] (xfs_dir2_leaf_getdents+0x480/0x8b4 [xfs])
> [42949619.880000] [<bf14e96c>] (xfs_dir2_leaf_getdents+0x0/0x8b4 [xfs]) from [<bf14b07c>] (xfs_readdir+0xcc/0xe0 [xfs])
> [42949619.890000] [<bf14afb0>] (xfs_readdir+0x0/0xe0 [xfs]) from [<bf18140c>] (xfs_file_readdir+0x144/0x194 [xfs])
> [42949619.900000] [<bf1812c8>] (xfs_file_readdir+0x0/0x194 [xfs]) from [<c009ffb0>] (vfs_readdir+0x84/0xb8)
> [42949619.910000] [<c009ff2c>] (vfs_readdir+0x0/0xb8) from [<c00a0050>] (sys_getdents64+0x6c/0xc0)
> [42949619.920000] [<c009ffe4>] (sys_getdents64+0x0/0xc0) from [<c0025bc0>] (ret_fast_syscall+0x0/0x3c)
> [42949619.930000] r7:000000d9 r6:0002a01c r5:0002a030 r4:00000000
> debian:~#
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-16 22:13 ` Eric Sandeen
@ 2008-10-17 7:01 ` Martin Michlmayr
2008-10-17 9:46 ` Gaudenz Steinlin
2008-10-17 14:15 ` Eric Sandeen
0 siblings, 2 replies; 21+ messages in thread
From: Martin Michlmayr @ 2008-10-17 7:01 UTC (permalink / raw)
To: Eric Sandeen; +Cc: Tobias Frost, linux-kernel, debian-arm, xfs
* Eric Sandeen <sandeen@sandeen.net> [2008-10-16 17:13]:
> So is this a regression? did it used to work? If so, when? :)
The original report was with 2.6.18 but that was with the old ABI:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=423562
I just installed a 2.6.22 kernel with EABI and I can also trigger
the bug. So it's not a (recent) regression.
> What's a little odd is that the buffer it dumped out looks like the
> beginning of a perfectly valid superblock for your filesystem
> (magic, block size, and block count all match). If you printk the
> "bno" variable right around line 2106 in xfs_da_btree.c, can you see
> what you get?
bno is 0.
> creating an xfs_metadump of the filesystem for examination on a
> non-arm box might also be interesting.
http://www.cyrius.com/tmp/dump5
(11 MB)
--
Martin Michlmayr
http://www.cyrius.com/
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-17 7:01 ` Martin Michlmayr
@ 2008-10-17 9:46 ` Gaudenz Steinlin
2008-10-17 13:49 ` Eric Sandeen
` (2 more replies)
2008-10-17 14:15 ` Eric Sandeen
1 sibling, 3 replies; 21+ messages in thread
From: Gaudenz Steinlin @ 2008-10-17 9:46 UTC (permalink / raw)
To: Martin Michlmayr
Cc: Eric Sandeen, Tobias Frost, linux-kernel, debian-arm, xfs
On Fri, Oct 17, 2008 at 09:01:09AM +0200, Martin Michlmayr wrote:
> * Eric Sandeen <sandeen@sandeen.net> [2008-10-16 17:13]:
> > So is this a regression? did it used to work? If so, when? :)
>
> The original report was with 2.6.18 but that was with the old ABI:
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=423562
> I just installed a 2.6.22 kernel with EABI and I can also trigger
> the bug. So it's not a (recent) regression.
As far as I can remember (I only used old ABI arm) this is not a
regression. XFS never worked on arm for me.
If you need tests on old ABI just tell me.
--
Ever tried. Ever failed. No matter.
Try again. Fail again. Fail better.
~ Samuel Beckett ~
^ permalink raw reply [flat|nested] 21+ messages in thread* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-17 9:46 ` Gaudenz Steinlin
@ 2008-10-17 13:49 ` Eric Sandeen
2008-10-18 13:11 ` Tobias Frost
2008-10-18 19:48 ` Kirill A. Shutemov
2 siblings, 0 replies; 21+ messages in thread
From: Eric Sandeen @ 2008-10-17 13:49 UTC (permalink / raw)
To: Martin Michlmayr, Eric Sandeen, Tobias Frost, linux-kernel,
debian-arm, xfs
Gaudenz Steinlin wrote:
> On Fri, Oct 17, 2008 at 09:01:09AM +0200, Martin Michlmayr wrote:
>> * Eric Sandeen <sandeen@sandeen.net> [2008-10-16 17:13]:
>>> So is this a regression? did it used to work? If so, when? :)
>> The original report was with 2.6.18 but that was with the old ABI:
>> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=423562
>> I just installed a 2.6.22 kernel with EABI and I can also trigger
>> the bug. So it's not a (recent) regression.
>
> As far as I can remember (I only used old ABI arm) this is not a
> regression. XFS never worked on arm for me.
Ok; I think there are probably a few problems, so trying to keep them
straight. I at least had xfs working properly in a qemu arm emulator a
few weeks ago... :)
> If you need tests on old ABI just tell me.
>
ok, thanks!
-Eric
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-17 9:46 ` Gaudenz Steinlin
2008-10-17 13:49 ` Eric Sandeen
@ 2008-10-18 13:11 ` Tobias Frost
2008-10-18 19:48 ` Kirill A. Shutemov
2 siblings, 0 replies; 21+ messages in thread
From: Tobias Frost @ 2008-10-18 13:11 UTC (permalink / raw)
To: Gaudenz Steinlin
Cc: Martin Michlmayr, Eric Sandeen, linux-kernel, debian-arm, xfs
I tried xfs both on the old and new ABI.
(my first try was Jan 2008, old ABI)
ASAIR at kernel version 2.6.18
On Fri, 2008-10-17 at 11:46 +0200, Gaudenz Steinlin wrote:
> On Fri, Oct 17, 2008 at 09:01:09AM +0200, Martin Michlmayr wrote:
> > * Eric Sandeen <sandeen@sandeen.net> [2008-10-16 17:13]:
> > > So is this a regression? did it used to work? If so, when? :)
> >
> > The original report was with 2.6.18 but that was with the old ABI:
> > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=423562
> > I just installed a 2.6.22 kernel with EABI and I can also trigger
> > the bug. So it's not a (recent) regression.
>
> As far as I can remember (I only used old ABI arm) this is not a
> regression. XFS never worked on arm for me.
>
> If you need tests on old ABI just tell me.
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-17 9:46 ` Gaudenz Steinlin
2008-10-17 13:49 ` Eric Sandeen
2008-10-18 13:11 ` Tobias Frost
@ 2008-10-18 19:48 ` Kirill A. Shutemov
2008-10-18 20:09 ` Christoph Hellwig
2 siblings, 1 reply; 21+ messages in thread
From: Kirill A. Shutemov @ 2008-10-18 19:48 UTC (permalink / raw)
To: Martin Michlmayr, Eric Sandeen, Tobias Frost, linux-kernel,
debian-arm, xfs
[-- Attachment #1: Type: text/plain, Size: 930 bytes --]
On Fri, Oct 17, 2008 at 11:46:15AM +0200, Gaudenz Steinlin wrote:
> On Fri, Oct 17, 2008 at 09:01:09AM +0200, Martin Michlmayr wrote:
> > * Eric Sandeen <sandeen@sandeen.net> [2008-10-16 17:13]:
> > > So is this a regression? did it used to work? If so, when? :)
> >
> > The original report was with 2.6.18 but that was with the old ABI:
> > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=423562
> > I just installed a 2.6.22 kernel with EABI and I can also trigger
> > the bug. So it's not a (recent) regression.
>
> As far as I can remember (I only used old ABI arm) this is not a
> regression. XFS never worked on arm for me.
You are wrong.
Our team have experience with XFS on ARM OABI on NAS. It worked fine on
2.6.12, but got broken on 2.6.17.
http://oss.sgi.com/bugzilla/show_bug.cgi?id=712
--
Regards, Kirill A. Shutemov
+ Belarus, Minsk
+ ALT Linux Team, http://www.altlinux.com/
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-18 19:48 ` Kirill A. Shutemov
@ 2008-10-18 20:09 ` Christoph Hellwig
2008-10-18 20:17 ` Kirill A. Shutemov
0 siblings, 1 reply; 21+ messages in thread
From: Christoph Hellwig @ 2008-10-18 20:09 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Martin Michlmayr, Eric Sandeen, Tobias Frost, linux-kernel,
debian-arm, xfs
On Sat, Oct 18, 2008 at 10:48:53PM +0300, Kirill A. Shutemov wrote:
> Our team have experience with XFS on ARM OABI on NAS. It worked fine on
> 2.6.12, but got broken on 2.6.17.
So any chance to bisect it down to at least a kernel release where it
stopped working?
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-18 20:09 ` Christoph Hellwig
@ 2008-10-18 20:17 ` Kirill A. Shutemov
0 siblings, 0 replies; 21+ messages in thread
From: Kirill A. Shutemov @ 2008-10-18 20:17 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Martin Michlmayr, Eric Sandeen, Tobias Frost, linux-kernel,
debian-arm, xfs
[-- Attachment #1: Type: text/plain, Size: 475 bytes --]
On Sat, Oct 18, 2008 at 04:09:39PM -0400, Christoph Hellwig wrote:
> On Sat, Oct 18, 2008 at 10:48:53PM +0300, Kirill A. Shutemov wrote:
> > Our team have experience with XFS on ARM OABI on NAS. It worked fine on
> > 2.6.12, but got broken on 2.6.17.
>
> So any chance to bisect it down to at least a kernel release where it
> stopped working?
I'll try to do it.
--
Regards, Kirill A. Shutemov
+ Belarus, Minsk
+ ALT Linux Team, http://www.altlinux.com/
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-17 7:01 ` Martin Michlmayr
2008-10-17 9:46 ` Gaudenz Steinlin
@ 2008-10-17 14:15 ` Eric Sandeen
2008-10-18 8:57 ` Martin Michlmayr
1 sibling, 1 reply; 21+ messages in thread
From: Eric Sandeen @ 2008-10-17 14:15 UTC (permalink / raw)
To: Martin Michlmayr; +Cc: Tobias Frost, linux-kernel, debian-arm, xfs
Martin Michlmayr wrote:
> * Eric Sandeen <sandeen@sandeen.net> [2008-10-16 17:13]:
>> So is this a regression? did it used to work? If so, when? :)
>
> The original report was with 2.6.18 but that was with the old ABI:
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=423562
> I just installed a 2.6.22 kernel with EABI and I can also trigger
> the bug. So it's not a (recent) regression.
>
>> What's a little odd is that the buffer it dumped out looks like the
>> beginning of a perfectly valid superblock for your filesystem
>> (magic, block size, and block count all match). If you printk the
>> "bno" variable right around line 2106 in xfs_da_btree.c, can you see
>> what you get?
>
> bno is 0.
Ok, that's a little odd. (correlates with the "bad" magic that was
seen, because block 0 is the superblock, but doesn't make sense because
we were trying to read a directory leaf block, in theory)
If you unmount & remount, does the ls work then?
>> creating an xfs_metadump of the filesystem for examination on a
>> non-arm box might also be interesting.
>
> http://www.cyrius.com/tmp/dump5
> (11 MB)
Thanks.
xfs_repair on x86 shows no errors; however it won't mount normally (bad
log clientid) - but mount -o norecovery,ro and subsequent ls works fine
(at first I thought filenames were badly scrambled but then remembered
that xfs_metadump does this by default ;))
The remaining problem that I know of on some arm architectures is a vmap
cache aliasing problem that usually shows up as log corruption; that may
explain the bad clientid thing but not sure why we're reading block 0 above.
Do you know what cachepolicy you're booted with? If it's writeallocate,
you might try cachepolicy=writeback, otherwise try cachepolicy=uncached
(which will be horribly slow) and see if the problem goes away or not;
it'd be a clue.
-Eric
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-17 14:15 ` Eric Sandeen
@ 2008-10-18 8:57 ` Martin Michlmayr
2008-10-18 14:48 ` Eric Sandeen
2008-10-19 1:48 ` Dave Chinner
0 siblings, 2 replies; 21+ messages in thread
From: Martin Michlmayr @ 2008-10-18 8:57 UTC (permalink / raw)
To: Eric Sandeen; +Cc: Tobias Frost, linux-kernel, debian-arm, xfs
* Eric Sandeen <sandeen@sandeen.net> [2008-10-17 09:15]:
> If you unmount & remount, does the ls work then?
I cannot even mount it:
debian:~# mkfs.xfs -f /dev/sda5
meta-data=/dev/sda5 isize=256 agcount=4, agsize=94380 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=377519, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0
debian:~# mount /dev/sda5 /mnt
[42949596.920000] XFS mounting filesystem sda5
debian:~# cp /usr/bin/* /mnt/
debian:~# umount /mnt
debian:~# mount -t xfs /dev/sda5 /mnt
[42949612.290000] XFS mounting filesystem sda5
[42949612.460000] Starting XFS recovery on filesystem: sda5 (logdev: internal)
[42949612.480000] XFS: xlog_recover_process_data: bad flag
[42949612.500000] XFS: log mount/recovery failed: error 5
[42949612.500000] XFS: log mount failed
mount: /dev/sda5: can't read superblock
debian:~#
> Do you know what cachepolicy you're booted with? If it's writeallocate,
> you might try cachepolicy=writeback, otherwise try cachepolicy=uncached
> (which will be horribly slow) and see if the problem goes away or not;
> it'd be a clue.
I just tried with cachepolicy=writeback and cachepolicy=uncached but I
get the same problem.
--
Martin Michlmayr
http://www.cyrius.com/
^ permalink raw reply [flat|nested] 21+ messages in thread* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-18 8:57 ` Martin Michlmayr
@ 2008-10-18 14:48 ` Eric Sandeen
2008-10-19 1:48 ` Dave Chinner
1 sibling, 0 replies; 21+ messages in thread
From: Eric Sandeen @ 2008-10-18 14:48 UTC (permalink / raw)
To: Martin Michlmayr; +Cc: Tobias Frost, linux-kernel, debian-arm, xfs
Martin Michlmayr wrote:
> * Eric Sandeen <sandeen@sandeen.net> [2008-10-17 09:15]:
>> Do you know what cachepolicy you're booted with? If it's writeallocate,
>> you might try cachepolicy=writeback, otherwise try cachepolicy=uncached
>> (which will be horribly slow) and see if the problem goes away or not;
>> it'd be a clue.
>
> I just tried with cachepolicy=writeback and cachepolicy=uncached but I
> get the same problem.
Oh, wow. This sounds like a new problem then; not a cache problem, and
not an alignment problem... hrm. I'll try to think of something else
to try.
Thanks,
-Eric
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-18 8:57 ` Martin Michlmayr
2008-10-18 14:48 ` Eric Sandeen
@ 2008-10-19 1:48 ` Dave Chinner
2008-10-19 3:06 ` Eric Sandeen
2008-10-19 5:12 ` Martin Michlmayr
1 sibling, 2 replies; 21+ messages in thread
From: Dave Chinner @ 2008-10-19 1:48 UTC (permalink / raw)
To: Martin Michlmayr
Cc: Eric Sandeen, Tobias Frost, linux-kernel, debian-arm, xfs
On Sat, Oct 18, 2008 at 10:57:56AM +0200, Martin Michlmayr wrote:
> * Eric Sandeen <sandeen@sandeen.net> [2008-10-17 09:15]:
> > If you unmount & remount, does the ls work then?
>
> I cannot even mount it:
>
> debian:~# mkfs.xfs -f /dev/sda5
> meta-data=/dev/sda5 isize=256 agcount=4, agsize=94380 blks
> = sectsz=512 attr=2
> data = bsize=4096 blocks=377519, imaxpct=25
> = sunit=0 swidth=0 blks
> naming =version 2 bsize=4096
> log =internal log bsize=4096 blocks=2560, version=2
> = sectsz=512 sunit=0 blks, lazy-count=0
> realtime =none extsz=4096 blocks=0, rtextents=0
> debian:~# mount /dev/sda5 /mnt
> [42949596.920000] XFS mounting filesystem sda5
> debian:~# cp /usr/bin/* /mnt/
> debian:~# umount /mnt
> debian:~# mount -t xfs /dev/sda5 /mnt
> [42949612.290000] XFS mounting filesystem sda5
> [42949612.460000] Starting XFS recovery on filesystem: sda5 (logdev: internal)
> [42949612.480000] XFS: xlog_recover_process_data: bad flag
> [42949612.500000] XFS: log mount/recovery failed: error 5
> [42949612.500000] XFS: log mount failed
> mount: /dev/sda5: can't read superblock
> debian:~#
That's the log replay indicating that there's a bad transaction
header in the log. Very strange - it shoul dbe a clean log. What
does xfs_logprint -t /dev/sda5 tell you about the transactions
in the log?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-19 1:48 ` Dave Chinner
@ 2008-10-19 3:06 ` Eric Sandeen
2008-10-19 9:07 ` Christoph Hellwig
2008-10-19 5:12 ` Martin Michlmayr
1 sibling, 1 reply; 21+ messages in thread
From: Eric Sandeen @ 2008-10-19 3:06 UTC (permalink / raw)
To: Martin Michlmayr, Eric Sandeen, Tobias Frost, linux-kernel,
debian-arm, xfs
Dave Chinner wrote:
> On Sat, Oct 18, 2008 at 10:57:56AM +0200, Martin Michlmayr wrote:
>> * Eric Sandeen <sandeen@sandeen.net> [2008-10-17 09:15]:
>>> If you unmount & remount, does the ls work then?
>> I cannot even mount it:
>>
>> debian:~# mkfs.xfs -f /dev/sda5
>> meta-data=/dev/sda5 isize=256 agcount=4, agsize=94380 blks
>> = sectsz=512 attr=2
>> data = bsize=4096 blocks=377519, imaxpct=25
>> = sunit=0 swidth=0 blks
>> naming =version 2 bsize=4096
>> log =internal log bsize=4096 blocks=2560, version=2
>> = sectsz=512 sunit=0 blks, lazy-count=0
>> realtime =none extsz=4096 blocks=0, rtextents=0
>> debian:~# mount /dev/sda5 /mnt
>> [42949596.920000] XFS mounting filesystem sda5
>> debian:~# cp /usr/bin/* /mnt/
>> debian:~# umount /mnt
>> debian:~# mount -t xfs /dev/sda5 /mnt
>> [42949612.290000] XFS mounting filesystem sda5
>> [42949612.460000] Starting XFS recovery on filesystem: sda5 (logdev: internal)
>> [42949612.480000] XFS: xlog_recover_process_data: bad flag
>> [42949612.500000] XFS: log mount/recovery failed: error 5
>> [42949612.500000] XFS: log mount failed
>> mount: /dev/sda5: can't read superblock
>> debian:~#
>
> That's the log replay indicating that there's a bad transaction
> header in the log. Very strange - it shoul dbe a clean log. What
> does xfs_logprint -t /dev/sda5 tell you about the transactions
> in the log?
This is probably the vmap cache aliasing problem that we paid a bit of
attention to a few months ago, no?
-Eric
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-19 3:06 ` Eric Sandeen
@ 2008-10-19 9:07 ` Christoph Hellwig
2008-10-19 16:22 ` Lennert Buytenhek
0 siblings, 1 reply; 21+ messages in thread
From: Christoph Hellwig @ 2008-10-19 9:07 UTC (permalink / raw)
To: Eric Sandeen
Cc: Martin Michlmayr, Tobias Frost, linux-kernel, debian-arm, xfs,
kernel, Hiroki KUMAGAI
On Sat, Oct 18, 2008 at 10:06:45PM -0500, Eric Sandeen wrote:
> This is probably the vmap cache aliasing problem that we paid a bit of
> attention to a few months ago, no?
Shouldn't the cachepolicy switch take care of that? But yes, there
are various problems with I/O on vmap regions with virtually indexed
caches, see:
http://www.spinics.net/lists/linux-arch/msg04301.html
http://tree.celinuxforum.org/CelfPubWiki/JapanTechnicalJamboree21?action=AttachFile&do=get&target=celf_discussions.pdf
Actually, looking again at the second document it shows exactly the
symptoms you're seeing.
>
> -Eric
>
>
---end quoted text---
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-19 9:07 ` Christoph Hellwig
@ 2008-10-19 16:22 ` Lennert Buytenhek
0 siblings, 0 replies; 21+ messages in thread
From: Lennert Buytenhek @ 2008-10-19 16:22 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Eric Sandeen, Martin Michlmayr, Tobias Frost, linux-kernel,
debian-arm, xfs, Hiroki KUMAGAI
On Sun, Oct 19, 2008 at 05:07:02AM -0400, Christoph Hellwig wrote:
> > This is probably the vmap cache aliasing problem that we paid a bit of
> > attention to a few months ago, no?
>
> Shouldn't the cachepolicy switch take care of that?
Setting cachepolicy=uncached should make aliasing issues disappear.
If you're still seeing issues with cachepolicy=uncached, it's likely
some other issue.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-19 1:48 ` Dave Chinner
2008-10-19 3:06 ` Eric Sandeen
@ 2008-10-19 5:12 ` Martin Michlmayr
1 sibling, 0 replies; 21+ messages in thread
From: Martin Michlmayr @ 2008-10-19 5:12 UTC (permalink / raw)
To: Eric Sandeen, Tobias Frost, linux-kernel, debian-arm, xfs
* Dave Chinner <david@fromorbit.com> [2008-10-19 12:48]:
> That's the log replay indicating that there's a bad transaction
> header in the log. Very strange - it shoul dbe a clean log. What
> does xfs_logprint -t /dev/sda5 tell you about the transactions
> in the log?
mdebian:~# mkfs.xfs -f /dev/sda5
meta-data=/dev/sda5 isize=256 agcount=4, agsize=94380
blks
= sectsz=512 attr=2
data = bsize=4096 blocks=377519,
imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks,
lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0
debian:~# mount -t xfs /dev/sda5 /mnt
debian:~# cp /usr/bin/* /mnt/
debian:~# umount /mnt
debian:~# mount -t xfs /dev/sda5 /mnt
mount: /dev/sda5: can't read superblock
debian:~# xfs_logprint -t /dev/sda5
xfs_logprint:
data device: 0x805
log device: 0x805 daddr: 1510112 length: 20480
XFS: Log inconsistent (didn't find previous header)
XFS: empty log check failed
xfs_logprint: failed to find head and tail, error: 5
debian:~#
--
Martin Michlmayr
http://www.cyrius.com/
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: XFS filesystem corruption on the arm(el) architecture
2008-10-02 0:45 ` XFS filesystem corruption on the arm(el) architecture Dave Chinner
2008-10-02 0:56 ` Eric Sandeen
2008-10-02 1:17 ` Eric Sandeen
@ 2008-10-02 1:42 ` Eric Sandeen
2 siblings, 0 replies; 21+ messages in thread
From: Eric Sandeen @ 2008-10-02 1:42 UTC (permalink / raw)
To: Tobias Frost, linux-kernel, debian-arm, xfs
Dave Chinner wrote:
> Adding xfs@oss.sgi.com to the cc list so all the XFS folk see this.
>
> On Wed, Oct 01, 2008 at 10:38:22PM +0200, Tobias Frost wrote:
>> (Note: Please CC me, as I am NOT on the lkml!!)
>>
>> Some time ago, I discovered some problems with xfs. Unfortunatly, I had
>> no time diving into it. However, some weeks ago some other people
>> running debian on ARM machines confirmed the problem on their machines
>> starting at [1], so I think it is appropitate to at least report it.
>> It has also been seen on 2.6.27-rc4 [2].
>>
>> summary: the xfs partition corrupts almost immediatly after creation. I
>> had the impression, that the first unlink (rm) causes the corruption,
>> but this might be just an impression.
>>
>> During the tests I made, I conserved a image of the corrupted filesystem
>> which I can make available on request (it's 26 Mbyte, gzipped).
>>
>> Please let me know how I can assist you in finding the problem.
>>
>>
>> [1] http://lists.debian.org/debian-arm/2008/08/msg00155.html
>> [2] http://lists.debian.org/debian-arm/2008/08/msg00184.html
>>
>> Best regards,
>> Tobias Frost
>> http://blog.coldtobi.de
>>
>> PS: Thank you for your great work!
Ok, actually: first - sorry for the scattershot replies. I thought
userspace was updated earlier, but:
xfsprogs-2.10.1 (5 September 2008)...
- Add packed on-disk shortform directory for ARM's old ABI, thanks to
Eric Sandeen.
and the original kernel change:
[XFS] Pack some shortform dir2 structures for the ARM old ABI
architecture.
...
Note that userspace needs a similar treatment, and any filesystems which
were running with the previous rogue "fix" will now see corruption
...
So perhaps as a first easy test, can you please re-test with
xfsprogs-2.10.1 or newer.
Thanks,
-Eric
^ permalink raw reply [flat|nested] 21+ messages in thread