blktap error in linux-3.1rc2

blktap error in linux-3.1rc2

[George Shuklin]

Good day.

Found strange behavior of new linux kernel:

[750121.807355] blkfront: xvda: barrier or flush: disabled
[750121.819697] end_request: I/O error, dev xvda, sector 31726040
[750121.819711] end_request: I/O error, dev xvda, sector 31726040
[750121.819719] Buffer I/O error on device xvda1, logical block 3965499
[750121.819725] lost page write due to I/O error on xvda1
[750121.819742] Aborting journal on device xvda1.
[750121.821142] EXT3-fs (xvda1): error: ext3_journal_start_sb: Detected 
aborted journal
[750121.821157] EXT3-fs (xvda1): error: remounting filesystem read-only
[750128.389147] eth0: no IPv6 routers present
[750151.061146] ------------[ cut here ]------------
[750151.061163] WARNING: at fs/ext3/inode.c:1620 
ext3_ordered_writepage+0x43/0x1c5 [ext3]()
[750151.061166] Modules linked in: xenfs nfsd nfs lockd fscache 
auth_rpcgss nfs_acl sunrpc isofs loop evdev snd_pcm snd_timer snd 
soundcore snd_page_alloc pcspkr ext3 jbd mbcache xen_blkfront xen_netfront
[750151.061191] Pid: 977, comm: flush-202:0 Not tainted 3.1.0-rc2+ #2
[750151.061194] Call Trace:
[750151.061204]  [<ffffffff810458c9>] ? warn_slowpath_common+0x78/0x8c
[750151.061210]  [<ffffffffa00393d8>] ? 
ext3_ordered_writepage+0x43/0x1c5 [ext3]
[750151.061217]  [<ffffffff810b7330>] ? __writepage+0xa/0x21
[750151.061221]  [<ffffffff810b71b8>] ? write_cache_pages+0x1f8/0x2e9
[750151.061225]  [<ffffffff810b7326>] ? set_page_dirty_lock+0x2b/0x2b
[750151.061229]  [<ffffffff810b72e3>] ? generic_writepages+0x3a/0x52
[750151.061238]  [<ffffffff81110ac5>] ? writeback_single_inode+0x124/0x2d3
[750151.061242]  [<ffffffff81110f45>] ? writeback_sb_inodes+0x16b/0x204
[750151.061246]  [<ffffffff8111104b>] ? __writeback_inodes_wb+0x6d/0xab
[750151.061250]  [<ffffffff811111af>] ? wb_writeback+0x126/0x214
[750151.061254]  [<ffffffff81111637>] ? wb_do_writeback+0x13e/0x19d
[750151.061257]  [<ffffffff8111171a>] ? bdi_writeback_thread+0x84/0x1e5
[750151.061261]  [<ffffffff81111696>] ? wb_do_writeback+0x19d/0x19d
[750151.061266]  [<ffffffff8105e5b1>] ? kthread+0x76/0x7e
[750151.061274]  [<ffffffff813328b4>] ? kernel_thread_helper+0x4/0x10
[750151.061279]  [<ffffffff81330973>] ? int_ret_from_sys_call+0x7/0x1b
[750151.061284]  [<ffffffff8132bafc>] ? retint_restore_args+0x5/0x6
[750151.061288]  [<ffffffff813328b0>] ? gs_change+0x13/0x13
[750151.061290] ---[ end trace 92a8910c5fe1d21f ]---


This is definitively not an block device error: linux-3.0.2 works fine 
on same VM on same block device and no external signs of block device 
damage in dom0.

I do think something was broken between linux-3.0 and linux-3.1 (and 
now, before rc2 become an stable 3.1 it shall be fixed).

Tested on XCP1.0 (xen3.3, blktap, backend is lvmoiscsi SR).

Re: blktap error in linux-3.1rc2

[Konrad Rzeszutek Wilk]

On Sun, Aug 21, 2011 at 01:36:35PM +0400, George Shuklin wrote:
> Good day.
> 
> Found strange behavior of new linux kernel:
> 
> [750121.807355] blkfront: xvda: barrier or flush: disabled
> [750121.819697] end_request: I/O error, dev xvda, sector 31726040
> [750121.819711] end_request: I/O error, dev xvda, sector 31726040

There is the error that is causing all of this. Somehow that
sector (31726040) and (3965499) return an error. Is there any notification
in the backend (dom0) about failures?

Can you run 'blktrace' in Dom0 on the disk in question? Is it always
the same sector? What is the size of your guest disk?

> [750121.819719] Buffer I/O error on device xvda1, logical block 3965499
> [750121.819725] lost page write due to I/O error on xvda1

So all of that above caused the nasty WARN below.

> [750121.819742] Aborting journal on device xvda1.
> [750121.821142] EXT3-fs (xvda1): error: ext3_journal_start_sb:
> Detected aborted journal
> [750121.821157] EXT3-fs (xvda1): error: remounting filesystem read-only
> [750128.389147] eth0: no IPv6 routers present
> [750151.061146] ------------[ cut here ]------------
> [750151.061163] WARNING: at fs/ext3/inode.c:1620
> ext3_ordered_writepage+0x43/0x1c5 [ext3]()
> [750151.061166] Modules linked in: xenfs nfsd nfs lockd fscache
> auth_rpcgss nfs_acl sunrpc isofs loop evdev snd_pcm snd_timer snd
> soundcore snd_page_alloc pcspkr ext3 jbd mbcache xen_blkfront
> xen_netfront
> [750151.061191] Pid: 977, comm: flush-202:0 Not tainted 3.1.0-rc2+ #2
> [750151.061194] Call Trace:
> [750151.061204]  [<ffffffff810458c9>] ? warn_slowpath_common+0x78/0x8c
> [750151.061210]  [<ffffffffa00393d8>] ?
> ext3_ordered_writepage+0x43/0x1c5 [ext3]
> [750151.061217]  [<ffffffff810b7330>] ? __writepage+0xa/0x21
> [750151.061221]  [<ffffffff810b71b8>] ? write_cache_pages+0x1f8/0x2e9
> [750151.061225]  [<ffffffff810b7326>] ? set_page_dirty_lock+0x2b/0x2b
> [750151.061229]  [<ffffffff810b72e3>] ? generic_writepages+0x3a/0x52
> [750151.061238]  [<ffffffff81110ac5>] ? writeback_single_inode+0x124/0x2d3
> [750151.061242]  [<ffffffff81110f45>] ? writeback_sb_inodes+0x16b/0x204
> [750151.061246]  [<ffffffff8111104b>] ? __writeback_inodes_wb+0x6d/0xab
> [750151.061250]  [<ffffffff811111af>] ? wb_writeback+0x126/0x214
> [750151.061254]  [<ffffffff81111637>] ? wb_do_writeback+0x13e/0x19d
> [750151.061257]  [<ffffffff8111171a>] ? bdi_writeback_thread+0x84/0x1e5
> [750151.061261]  [<ffffffff81111696>] ? wb_do_writeback+0x19d/0x19d
> [750151.061266]  [<ffffffff8105e5b1>] ? kthread+0x76/0x7e
> [750151.061274]  [<ffffffff813328b4>] ? kernel_thread_helper+0x4/0x10
> [750151.061279]  [<ffffffff81330973>] ? int_ret_from_sys_call+0x7/0x1b
> [750151.061284]  [<ffffffff8132bafc>] ? retint_restore_args+0x5/0x6
> [750151.061288]  [<ffffffff813328b0>] ? gs_change+0x13/0x13
> [750151.061290] ---[ end trace 92a8910c5fe1d21f ]---
> 
> 
> This is definitively not an block device error: linux-3.0.2 works
> fine on same VM on same block device and no external signs of block
> device damage in dom0.

You said blktap in the title - so the problem does not surface if you
blkback ('phy' ?)

> 
> I do think something was broken between linux-3.0 and linux-3.1 (and
> now, before rc2 become an stable 3.1 it shall be fixed).
> 
> Tested on XCP1.0 (xen3.3, blktap, backend is lvmoiscsi SR).
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel