XFS getting EAGAIN from block device

XFS getting EAGAIN from block device

[Marko Huttunen]

[-- Attachment #1.1: Type: text/plain, Size: 4324 bytes --]

Hello,

After upgrading Ubuntu from 10.10 to 11.04 I started getting error messages
in my syslog when accessing one of my xfs partitions. It is not the root
partition.

The helpful team on IRC (#xfs on irc.freenode.net) says that what I want to
do is figure out why I'm getting -EAGAIN from the block device.

The error message I get when accessing the filesystem (through nautilus, or
in a terminal) is:

<snip>
Jun  7 18:08:14 picard kernel: [ 2602.542458] I/O error in filesystem
("dm-4") meta-data dev dm-4 block 0x41c6dc18       ("xfs_trans_read_buf")
error 11 buf count 4096
Jun  7 18:08:14 picard kernel: [ 2602.548673] I/O error in filesystem
("dm-4") meta-data dev dm-4 block 0x41c23898       ("xfs_trans_read_buf")
error 11 buf count 4096
Jun  7 18:08:14 picard kernel: [ 2602.553135] I/O error in filesystem
("dm-4") meta-data dev dm-4 block 0x41c133a8       ("xfs_trans_read_buf")
error 11 buf count 4096
</snip>
... and so on.

The helpful team on IRC supplied me with a patch to understand the error
message alittle better.

Filename: hch_patch
<snip>
--- a/fs/xfs/linux-2.6/xfs_buf.c
+++ b/fs/xfs/linux-2.6/xfs_buf.c
@@ -1264,6 +1264,9 @@ xfs_buf_bio_end_io(
unsigned int            blocksize = bp->b_target->bt_bsize;
struct bio_vec          *bvec = bio->bi_io_vec + bio->bi_vcnt - 1;

+       if (error && error != -EIO && error != -EOPNOTSUPP)
+               printk("XFS: got error %d in %s\n", error, __func__);
+
xfs_buf_ioerror(bp, -error);

if (!error && xfs_buf_is_vmapped(bp) && (bp->b_flags & XBF_READ))
</snip>

After fetching a new kernel source, unmounting my xfs partition, applying
the patch (running: "patch -p1 < hch_patch" in /usr/src/linux-2.6.38) and
making a new module (running: "make -C /lib/modules/`uname -r`/build
M=`pwd`/fs/xfs" in /usr/src/linux-2.6.38), removing the old module (running:
"rmmod xfs.ko") and then insmoding the new (running: "insmod xfs.ko") I get
the following error message:

<snip>
Jun  7 20:25:21 picard kernel: [10821.778982] XFS: got error -11 in
xfs_buf_bio_end_io
Jun  7 20:25:21 picard kernel: [10821.780263] I/O error in filesystem
("dm-4") meta-data dev dm-4 block 0x41c6dc18       ("xfs_trans_read_buf")
error 11 buf count 4096XFS: got error -11 in xfs_buf_bio_end_io
Jun  7 20:25:21 picard kernel: [10821.781403] I/O error in filesystem
("dm-4") meta-data dev dm-4 block 0x41c23898       ("xfs_trans_read_buf")
error 11 buf count 4096XFS: got error -11 in xfs_buf_bio_end_io
Jun  7 20:25:21 picard kernel: [10821.796254] I/O error in filesystem
("dm-4") meta-data dev dm-4 block 0x41c133a8       ("xfs_trans_read_buf")
error 11 buf count 4096XFS: got error -11 in xfs_buf_bio_end_io
</snip>
... and so on.

After a while when for example extracting a zip archive I get the following
error messages:

<snip>
Jun  7 20:28:38 picard kernel: [11018.263647] XFS: got error -11 in
xfs_buf_bio_end_io
Jun  7 20:28:38 picard kernel: [11018.263842] XFS: got error -11 in
xfs_buf_bio_end_io
Jun  7 20:28:41 picard kernel: [11021.399311] XFS: got error -11 in
xfs_buf_bio_end_io
Jun  7 20:28:41 picard kernel: [11021.435716] I/O error in filesystem
("dm-4") meta-data dev dm-4 block 0x3a53bbf0       ("xfs_trans_read_buf")
error 11 buf count 4096
Jun  7 20:28:41 picard kernel: [11021.435724] xfs_force_shutdown(dm-4,0x1)
called from line 3894 of file /usr/src/linux-2.6.38/fs/xfs/xfs_bmap.c.
 Return address = 0xffffffffa0fa34a2
Jun  7 20:28:41 picard kernel: [11021.580677] Filesystem dm-4: I/O Error
Detected.  Shutting down filesystem: dm-4
Jun  7 20:28:42 picard kernel: [11021.580683] Please umount the filesystem,
and rectify the problem(s)
Jun  7 20:29:12 picard kernel: [11023.043494] Filesystem dm-4:
xfs_log_force: error 5 returned.
Jun  7 20:29:42 picard kernel: [11053.015272] Filesystem dm-4:
xfs_log_force: error 5 returned.
</snip>
... and so on

Accessing the device is slower than before the upgrade to 11.04, and it is
not without errors that I can access the device.

$ uname -r
Linux picard 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011
x86_64 x86_64 x86_64 GNU/Linux


HW specs:
The XFS partition is run on a HW Raid on 2 x 1TB WD Black Caviar.
Motherboard is an ASUS Sabertooth P67 with Intel Sandy Bridge i5-2500K CPU @
3.30GHz.


Do you guys know what might be wrong, how I or you or we can solve this?


Regards,

Marko

[-- Attachment #1.2: Type: text/html, Size: 5886 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: XFS getting EAGAIN from block device

[Mike Snitzer]

On Tue, Jun 07 2011 at  4:44pm -0400,
Marko Huttunen <marko@codito.se> wrote:

> Hello,
> 
> After upgrading Ubuntu from 10.10 to 11.04 I started getting error messages
> in my syslog when accessing one of my xfs partitions. It is not the root
> partition.
> 
> The helpful team on IRC (#xfs on irc.freenode.net) says that what I want to
> do is figure out why I'm getting -EAGAIN from the block device.
> 
> The error message I get when accessing the filesystem (through nautilus, or
> in a terminal) is:
> 
> <snip>
> Jun  7 18:08:14 picard kernel: [ 2602.542458] I/O error in filesystem
> ("dm-4") meta-data dev dm-4 block 0x41c6dc18       ("xfs_trans_read_buf")
> error 11 buf count 4096
...
> Accessing the device is slower than before the upgrade to 11.04, and it is
> not without errors that I can access the device.
> 
> $ uname -r
> Linux picard 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011
> x86_64 x86_64 x86_64 GNU/Linux
> 
> 
> HW specs:
> The XFS partition is run on a HW Raid on 2 x 1TB WD Black Caviar.
> Motherboard is an ASUS Sabertooth P67 with Intel Sandy Bridge i5-2500K CPU @
> 3.30GHz.
> 
> 
> Do you guys know what might be wrong, how I or you or we can solve this?

Not aware of any -EAGAIN generated by DM in the IO path.

Could be some change beneath DM.

Do you get the same result if you try a stock 2.6.38 kernel?

Mike

Re: XFS getting EAGAIN from block device

[Bryn M. Reeves]

On 06/07/2011 09:44 PM, Marko Huttunen wrote:
> HW specs:
> The XFS partition is run on a HW Raid on 2 x 1TB WD Black Caviar. Motherboard is 
> an ASUS Sabertooth P67 with Intel Sandy Bridge i5-2500K CPU @ 3.30GHz.

As Mike said, it sounds like something further down the stack. You don't mention
what sort of array this is (i.e. which hardware RAID driver it's using, whether
it presents as a SCSI device etc.).

Depending on the answer to that question your next steps might involve enabling
debug in that driver, turning up the generic SCSI logging level or making use of
blktrace to try to understand where the I/O is going (or rather, where it isn't
going ;).

Regards,
Bryn.

Re: XFS getting EAGAIN from block device

[Marko Huttunen]

[-- Attachment #1.1: Type: text/plain, Size: 908 bytes --]

Hey guys,

The error messages stopped showing up in my syslog, without me doing
something.


Regards,

Marko



On Wed, Jun 8, 2011 at 11:11 AM, Bryn M. Reeves <bmr@redhat.com> wrote:

> On 06/07/2011 09:44 PM, Marko Huttunen wrote:
> > HW specs:
> > The XFS partition is run on a HW Raid on 2 x 1TB WD Black Caviar.
> Motherboard is
> > an ASUS Sabertooth P67 with Intel Sandy Bridge i5-2500K CPU @ 3.30GHz.
>
> As Mike said, it sounds like something further down the stack. You don't
> mention
> what sort of array this is (i.e. which hardware RAID driver it's using,
> whether
> it presents as a SCSI device etc.).
>
> Depending on the answer to that question your next steps might involve
> enabling
> debug in that driver, turning up the generic SCSI logging level or making
> use of
> blktrace to try to understand where the I/O is going (or rather, where it
> isn't
> going ;).
>
> Regards,
> Bryn.
>

[-- Attachment #1.2: Type: text/html, Size: 1357 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]