State of XFS on ARM

State of XFS on ARM

[Daniel Goller]

Dear Sir or Madam,

I Would like to find out about the current state of XFS on ARM.
I have been using XFS for a while successfully on my laptop, and was
using XFS on a ARM device (little endian) since it would hold the same
data i had on the laptop.
It seems i have not been able to unmount it and then mounting it
cleanly once, always required xfs_repair -L   /dev/sdc3
I Could understand power issues or lockups causing this, but on clean
umount and followed mount to see it fail is surprising.
When mounting fails on the headless arm machine i move the drive to a
x86_64 and run xfs_repair there when mounting there fails too (so log
can't be replayed, making -L necessary).
All of this leads me to ask:  "Is XFS as well maintained on ARM as it
is on x86/x86_64?"

Thank you in advance for any info you can provide,

Daniel Goller

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: State of XFS on ARM

[Eric Sandeen]

Daniel Goller wrote:
> Dear Sir or Madam,
> 
> I Would like to find out about the current state of XFS on ARM.
> I have been using XFS for a while successfully on my laptop, and was
> using XFS on a ARM device (little endian) since it would hold the same
> data i had on the laptop.
> It seems i have not been able to unmount it and then mounting it
> cleanly once, always required xfs_repair -L   /dev/sdc3
> I Could understand power issues or lockups causing this, but on clean
> umount and followed mount to see it fail is surprising.

well, actually neither power issues nor lockups should cause it either ;)

> When mounting fails on the headless arm machine i move the drive to a
> x86_64 and run xfs_repair there when mounting there fails too (so log
> can't be replayed, making -L necessary).
> All of this leads me to ask:  "Is XFS as well maintained on ARM as it
> is on x86/x86_64?"

Short answer no, but effort is made.  The last known issue, as far
as I know, is a cache aliasing problem.

This patch is a big-hammer approach, better things have been proposed
but not yet upstream as far as I know.

With "what doesn't work" helpfully commented out  ;) 

Index: linux-2.6.22.18/fs/xfs/linux-2.6/xfs_buf.c
===================================================================
--- linux-2.6.22.18.orig/fs/xfs/linux-2.6/xfs_buf.c
+++ linux-2.6.22.18/fs/xfs/linux-2.6/xfs_buf.c
@@ -1185,6 +1185,8 @@ _xfs_buf_ioapply(
 		bio->bi_end_io = xfs_buf_bio_end_io;
 		bio->bi_private = bp;
 
+		//flush_dcache_page(bp->b_pages[0]);
+		flush_cache_all();
 		bio_add_page(bio, bp->b_pages[0], PAGE_CACHE_SIZE, 0);
 		size = 0;
 
@@ -1211,6 +1213,8 @@ next_chunk:
 		if (nbytes > size)
 			nbytes = size;
 
+		//flush_dcache_page(bp->b_pages[map_i]);
+		flush_cache_all();
 		rbytes = bio_add_page(bio, bp->b_pages[map_i], nbytes, offset);
 		if (rbytes < nbytes)
 			break;



> Thank you in advance for any info you can provide,
> 
> Daniel Goller
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: State of XFS on ARM

[Christoph Hellwig]

On Mon, Feb 01, 2010 at 09:24:33PM -0600, Daniel Goller wrote:
> It seems i have not been able to unmount it and then mounting it
> cleanly once, always required xfs_repair -L   /dev/sdc3
> I Could understand power issues or lockups causing this, but on clean
> umount and followed mount to see it fail is surprising.
> When mounting fails on the headless arm machine i move the drive to a
> x86_64 and run xfs_repair there when mounting there fails too (so log
> can't be replayed, making -L necessary).
> All of this leads me to ask:  "Is XFS as well maintained on ARM as it
> is on x86/x86_64?"

XFS itself is platform neutral.  But it seems like you have an ARM
platform with virtually indexed caches, which currently can't support
the I/O XFS does.  James has been working on fixing this for a while,
but so far the architecture support patches for it unfortunately still
haven't made it to mainline despite many people running into this issue.

The git tree with the current versions of the patches to fix this is
here:

	http://git.kernel.org/?p=linux/kernel/git/jejb/xfs-vipt/.git;a=summary

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: State of XFS on ARM

[Richard Sharpe]

On Tue, Feb 2, 2010 at 3:23 AM, Christoph Hellwig <hch@infradead.org> wrote:
> On Mon, Feb 01, 2010 at 09:24:33PM -0600, Daniel Goller wrote:
>> It seems i have not been able to unmount it and then mounting it
>> cleanly once, always required xfs_repair -L   /dev/sdc3
>> I Could understand power issues or lockups causing this, but on clean
>> umount and followed mount to see it fail is surprising.
>> When mounting fails on the headless arm machine i move the drive to a
>> x86_64 and run xfs_repair there when mounting there fails too (so log
>> can't be replayed, making -L necessary).
>> All of this leads me to ask:  "Is XFS as well maintained on ARM as it
>> is on x86/x86_64?"
>
> XFS itself is platform neutral.  But it seems like you have an ARM
> platform with virtually indexed caches, which currently can't support
> the I/O XFS does.  James has been working on fixing this for a while,
> but so far the architecture support patches for it unfortunately still
> haven't made it to mainline despite many people running into this issue.

I would be interested in a precise explanation for this problem. We
ran into an issue here with XFS on the Marvell 78200 (ARM) which has a
VIVT cache. We eventually solved the problem by using
flush_icache_range in a strategic place (after also discovering the
need for flush_dcache_page, but that has to do with doing IO between
the cores on the 78200), but we really did not understand why that
fixed the problem. However, it stopped the corruption we were
experiencing when using a file system exerciser.

(We have subsequently stopped using XFS, but not because of any
corruption issues.)

-- 
Regards,
Richard Sharpe

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: State of XFS on ARM

[James Bottomley]

On Tue, 2010-02-02 at 07:42 -0800, Richard Sharpe wrote:
> On Tue, Feb 2, 2010 at 3:23 AM, Christoph Hellwig <hch@infradead.org> wrote:
> > On Mon, Feb 01, 2010 at 09:24:33PM -0600, Daniel Goller wrote:
> >> It seems i have not been able to unmount it and then mounting it
> >> cleanly once, always required xfs_repair -L   /dev/sdc3
> >> I Could understand power issues or lockups causing this, but on clean
> >> umount and followed mount to see it fail is surprising.
> >> When mounting fails on the headless arm machine i move the drive to a
> >> x86_64 and run xfs_repair there when mounting there fails too (so log
> >> can't be replayed, making -L necessary).
> >> All of this leads me to ask:  "Is XFS as well maintained on ARM as it
> >> is on x86/x86_64?"
> >
> > XFS itself is platform neutral.  But it seems like you have an ARM
> > platform with virtually indexed caches, which currently can't support
> > the I/O XFS does.  James has been working on fixing this for a while,
> > but so far the architecture support patches for it unfortunately still
> > haven't made it to mainline despite many people running into this issue.
> 
> I would be interested in a precise explanation for this problem. We
> ran into an issue here with XFS on the Marvell 78200 (ARM) which has a
> VIVT cache. We eventually solved the problem by using
> flush_icache_range in a strategic place (after also discovering the
> need for flush_dcache_page, but that has to do with doing IO between
> the cores on the 78200), but we really did not understand why that
> fixed the problem. However, it stopped the corruption we were
> experiencing when using a file system exerciser.

XFS uses large buffers for log operations.  Because large physically
contiguous pieces of memory are hard to come by, it makes its logs from
virtually contiguous pages.  The problem is that doing this sets up
aliasing within the kernel: each page in the range now has two virtual
addresses: one for the standard kernel offset map and the other for the
xfs created virtual area.  When I/O is done on a kernel page, the kernel
always flushes the offset map to ensure that the data is in main memory.
However, when the page has another alias in the xfs created virtual
area, the kernel I/O subsystem has no idea about this.  Since xfs is
reading and writing to this other alias, that's what needs to be flushed
to make sure the data is safely in main memory before I/O takes place.

James


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: State of XFS on ARM

[Daniel Goller]

Thanks for the many replys, it also made me realize how basic and
general the question was, without giving much details about the
hardware.

I see my "can't mount a xfs partition after successful unmounting
without needing to clear the log" in 2.6.32, 2.6.33-rc2 and now
2.6.33-rc6.

The arm device:

Processor       : Feroceon 88FR131 rev 1 (v5l)
BogoMIPS        : 1192.75
Features        : swp half thumb fastmult edsp
CPU implementer : 0x56
CPU architecture: 5TE
CPU variant     : 0x2
CPU part        : 0x131
CPU revision    : 1

Hardware        : Marvell SheevaPlug Reference Board
Revision        : 0000
Serial          : 0000000000000000


Typically i just can't mount without a xfs_repair -L  which seems to
be all i need to get it back and lose no data in the process.

In 2.6.33-rc6 i had it for the first time do this: (which basically
stopped the boot process cold as it would not go past this point)
keeping this partition from being mounted at boot then allowed me to
record the following when manually mounting it)


* Mounting local filesystems...
Internal error: Oops: 5 [#1]
last sysfs file: /sys/kernel/uevent_seqnum
Modules linked in:
CPU: 0    Not tainted  (2.6.33-rc6 #1)
PC is at xlog_recover_process_data+0x1c4/0x304
LR is at xlog_recover_unmount_trans+0x10/0x1c
pc : []    lr : []    psr: 20000013
sp : df091c78  ip : 0000000a  fp : 00000001
r10: df0794d0  r9 : df091d14  r8 : e0ffea00
r7 : df091d04  r6 : e0ffe034  r5 : 00000010  r4 : 02e00a34
r3 : 00000001  r2 : c0563190  r1 : c05631b8  r0 : 00000000
Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Control: 0005317f  Table: 1f314000  DAC: 00000015
Process mount (pid: 857, stack limit = 0xdf090270)
Stack: (0xdf091c78 to 0xdf092000)
1c60:                                                       00000007 df291000
1c80: 00000005 df3329c0 00000000 00000001 00000005 df3329c0 df291000 00000005
1ca0: 00000001 00000000 df332cc0 0000000f 00000000 c0206eb0 00000001 df332cc0
1cc0: df091d44 ffffffff c0563174 c003d13c df091d44 c058dddc c0563174 00000002
1ce0: 00000002 00000000 00000000 00000000 df332cc0 c003d66c 00000028 df332c00
1d00: c04f0121 00000000 00000000 00000000 00000000 df0ade00 00000000 00000000
1d20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
1d40: 00000000 e0ffe000 00000000 df3329c0 00000000 00000028 00000000 00000006
1d60: 00000000 a3d7b000 00000000 c02074d8 00000006 00000000 00000001 c0427758
1d80: 60000013 df3329c0 00000000 00000006 00000000 00408100 00000000 c0207534
1da0: 00000006 00000000 00000000 df3329c0 00000000 00000006 00000000 00408100
1dc0: 00000000 c0208be4 00000006 00000000 00000006 00000000 00000028 00000000
1de0: df0fd800 000001c0 00000000 c0202b58 00020000 df0fd800 c0220a64 df0fd800
1e00: c0220a64 073904d0 00000000 073904cc 00000000 c020ac34 00020000 000002d0
1e20: 00000001 00000000 51eb851f c02154fc 00000000 00000058 df296000 c01f1b1c
1e40: 0000000a 00000058 df189120 c01f1b1c 0000000a df0fda9c df296000 c01f1b1c
1e60: 00000000 df0fd800 df218a00 00000000 df28a000 00000000 00000000 df28a000
1e80: 00000000 c0220a64 df218a00 00000000 df6b8340 00000003 df6b839c 00000000
1ea0: df218a00 c00991fc 33616473 df096f00 df096da0 000000d0 df096da0 df096f40
1ec0: 0000000a c007cdd8 df296d00 df296d00 c056dfc4 df096da0 df28a000 00000000
1ee0: df28a000 c021ed60 c02208e8 df296d00 00000000 c00981fc df296d00 c00ab304
1f00: c056dfc4 df096d60 df28a000 df096da0 00000000 c00982d0 00000000 00000030
1f20: df091f48 df096da0 df096d60 c00addc4 00000a18 00200800 df28a000 df090000
1f40: 00000000 0001e560 dfbebe00 df705d80 c007186c df241000 0001e560 00200800
1f60: 00000000 c002b044 df090000 00000000 0001e560 c00ae3f4 df28a000 c00476c8
1f80: 00000000 df28a000 df096da0 df096d60 00000000 0001ea18 00200800 0001e570
1fa0: 00000015 c002aec0 0001ea18 00200800 0001e560 0001e570 0001e988 00200800
1fc0: 0001ea18 00200800 0001e570 00000015 0001c444 00000000 00000000 0001e560
1fe0: 00200800 befff5cc 0000b0f4 40115384 60000010 0001e560 00000000 00000000
[] (xlog_recover_process_data+0x1c4/0x304) from []
(xlog_do_recovery_pass+0x1b0/0x794)
[] (xlog_do_recovery_pass+0x1b0/0x794) from [] (xlog_do_log_recovery+0x44/0x88)
[] (xlog_do_log_recovery+0x44/0x88) from [] (xlog_do_recover+0x18/0x124)
[] (xlog_do_recover+0x18/0x124) from [] (xlog_recover+0x7c/0xc4)
[] (xlog_recover+0x7c/0xc4) from [] (xfs_log_mount+0x154/0x18c)
[] (xfs_log_mount+0x154/0x18c) from [] (xfs_mountfs+0x34c/0x670)
[] (xfs_mountfs+0x34c/0x670) from [] (xfs_fs_fill_super+0x17c/0x2c0)
[] (xfs_fs_fill_super+0x17c/0x2c0) from [] (get_sb_bdev+0x138/0x170)
[] (get_sb_bdev+0x138/0x170) from [] (xfs_fs_get_sb+0x1c/0x28)
[] (xfs_fs_get_sb+0x1c/0x28) from [] (vfs_kern_mount+0x50/0xe0)
[] (vfs_kern_mount+0x50/0xe0) from [] (do_kern_mount+0x34/0xdc)
[] (do_kern_mount+0x34/0xdc) from [] (do_mount+0x154/0x700)
[] (do_mount+0x154/0x700) from [] (sys_mount+0x84/0xc4)
[] (sys_mount+0x84/0xc4) from [] (ret_fast_syscall+0x0/0x28)
Code: 93a03000 82033001 e3530000 0a000048 (e5d43008)
---[ end trace b80cd4d01963a09f ]---
/etc/init.d/localmount: line 15:   857 Segmentation fault      mount
-at "$types"

it seems to operate fine since running a xfs_repair -L  on it from the
x86_64 laptop (It is a disk in a usb enclosure, which makes the moving
of the drive to clear the log rather easy)

> The git tree with the current versions of the patches to fix this is
> here:
>
>        http://git.kernel.org/?p=linux/kernel/git/jejb/xfs-vipt/.git;a=summary
>
>

I will probably not get a chance to try pulling this from git before
the weekend, if the Sheevaplug however the type of arm machine you
were referring to (i honestly never looked at its cache), i would be
interested in trying the fix and (hopefully) stick to xfs for this
partition.

Thanks,

Daniel Goller

Let me know if there is further information you would like to look at
for this issue.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: State of XFS on ARM

[Daniel Goller]

Any particular stress test you can suggest, to use once i try git for
xfs on this Marvell box?
To try to help bringing up issues faster than through normal use of this box?

Thanks in advance for any suggestions,

Daniel

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: State of XFS on ARM

[Christoph Hellwig]

On Wed, Feb 03, 2010 at 02:34:43PM -0600, Daniel Goller wrote:
> Any particular stress test you can suggest, to use once i try git for
> xfs on this Marvell box?
> To try to help bringing up issues faster than through normal use of this box?

The primary use of the large buffers where this matters is the log.
So you just need to cleanly unmount and mount the filesystem again,
and the issues your previously reported should be gone.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: State of XFS on ARM

[Eric Sandeen]

Daniel Goller wrote:
> Any particular stress test you can suggest, to use once i try git for
> xfs on this Marvell box?
> To try to help bringing up issues faster than through normal use of this box?

Run it through the xfstests suite:

http://git.kernel.org/?p=fs/xfs/xfstests-dev.git;a=summary

(I have a sheevaplug too, thanks to some nice folks @Marvell, and
I really need to get regular qa going on it in return... have been
too busy so far) :(

There's a readme in the tree, but basically just create a local.config like:

#        - setenv TEST_DEV "device containing TEST PARTITION"
#        - setenv TEST_DIR "mount point of TEST PARTITION"
#        - setenv SCRATCH_DEV "device containing SCRATCH PARTITION"
#        - setenv SCRATCH_MNT "mount point for SCRATCH PARTITION"

TEST_DEV=/dev/sdb1
TEST_DIR=/mnt/test
SCRATCH_DEV=/dev/sdb3
SCRATCH_MNT=/mnt/scratch

mkfs.xfs $TEST_DEV, and then run

#./check -g auto

-Eric

> Thanks in advance for any suggestions,
> 
> Daniel
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs