Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c

Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c

[Christopher S. Aker]

Either right on boot, or shortly thereafter, the following occurs with 
either writeback or ordered filesystems.  Multiple reports, all 
different machines, different users...  Some people had success after I 
changed CONFIG_EXT3_DEFAULTS_TO_ORDERED off, however some still had 
problems...  That was my only config change from 3.0.4.  Them rebooting 
back into 3.0.4 fixed it.


------------[ cut here ]------------
WARNING: at fs/ext3/inode.c:1620 ext3_ordered_writepage+0x20a/0x230()
Modules linked in:
Pid: 2149, comm: flush-202:0 Not tainted 3.1.0-linode39 #1
Call Trace:
  [<c01346f8>] ? warn_slowpath_common+0x78/0xb0
  [<c025666a>] ? ext3_ordered_writepage+0x20a/0x230
  [<c025666a>] ? ext3_ordered_writepage+0x20a/0x230
  [<c013474b>] ? warn_slowpath_null+0x1b/0x20
  [<c025666a>] ? ext3_ordered_writepage+0x20a/0x230
  [<c01067e4>] ? check_events+0x8/0xc
  [<c018c3c8>] ? __writepage+0x8/0x30
  [<c018cbc6>] ? write_cache_pages+0x196/0x390
  [<c018c3c0>] ? set_page_dirty+0x60/0x60
  [<c018cdfe>] ? generic_writepages+0x3e/0x60
  [<c01d4ae5>] ? writeback_single_inode+0x115/0x2b0
  [<c0102b18>] ? xen_end_context_switch+0x8/0x10
  [<c01d52c1>] ? writeback_sb_inodes+0x171/0x210
  [<c01d58d5>] ? __writeback_inodes_wb+0x75/0xb0
  [<c01d5b31>] ? wb_writeback+0x1a1/0x1b0
  [<c0106007>] ? xen_force_evtchn_callback+0x17/0x30
  [<c01067e4>] ? check_events+0x8/0xc
  [<c01d5cef>] ? wb_do_writeback+0x1af/0x1c0
  [<c01d5d71>] ? bdi_writeback_thread+0x71/0x130
  [<c01d5d00>] ? wb_do_writeback+0x1c0/0x1c0
  [<c014e1a4>] ? kthread+0x74/0x80
  [<c014e130>] ? kthread_worker_fn+0x170/0x170
  [<c06c63b6>] ? kernel_thread_helper+0x6/0x10
---[ end trace 702f842be742d945 ]---


And a different user:
------------[ cut here ]------------
WARNING: at fs/ext3/inode.c:1695 ext3_writeback_writepage+0x12a/0x150()
Modules linked in:
Pid: 2146, comm: flush-202:0 Not tainted 3.1.0-linode39 #1
Call Trace:
  [<c01346f8>] ? warn_slowpath_common+0x78/0xb0
  [<c025643a>] ? ext3_writeback_writepage+0x12a/0x150
  [<c025643a>] ? ext3_writeback_writepage+0x12a/0x150
  [<c013474b>] ? warn_slowpath_null+0x1b/0x20
  [<c025643a>] ? ext3_writeback_writepage+0x12a/0x150
  [<c01067e4>] ? check_events+0x8/0xc
  [<c018c3c8>] ? __writepage+0x8/0x30
  [<c018cbc6>] ? write_cache_pages+0x196/0x390
  [<c018c3c0>] ? set_page_dirty+0x60/0x60
  [<c010dfd8>] ? sched_clock+0x8/0x10
  [<c0154452>] ? sched_clock_local+0xb2/0x190
  [<c018cdfe>] ? generic_writepages+0x3e/0x60
  [<c01d4ae5>] ? writeback_single_inode+0x115/0x2b0
  [<c0102b18>] ? xen_end_context_switch+0x8/0x10
  [<c01d52c1>] ? writeback_sb_inodes+0x171/0x210
  [<c01d58d5>] ? __writeback_inodes_wb+0x75/0xb0
  [<c01d5b31>] ? wb_writeback+0x1a1/0x1b0
  [<c0106007>] ? xen_force_evtchn_callback+0x17/0x30
  [<c01067e4>] ? check_events+0x8/0xc
  [<c01d5cef>] ? wb_do_writeback+0x1af/0x1c0
  [<c01d5d71>] ? bdi_writeback_thread+0x71/0x130
  [<c01d5d00>] ? wb_do_writeback+0x1c0/0x1c0
  [<c014e1a4>] ? kthread+0x74/0x80
  [<c014e130>] ? kthread_worker_fn+0x170/0x170
  [<c06c63b6>] ? kernel_thread_helper+0x6/0x10
---[ end trace 0a37bc36a180e73c ]---

Am I barking up the wrong list?

Thanks,
-Chris

Re: Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c

[Konrad Rzeszutek Wilk]

On Tue, Oct 25, 2011 at 03:48:00PM -0400, Christopher S. Aker wrote:
> Either right on boot, or shortly thereafter, the following occurs
> with either writeback or ordered filesystems.  Multiple reports, all
> different machines, different users...  Some people had success
> after I changed CONFIG_EXT3_DEFAULTS_TO_ORDERED off, however some
> still had problems...  That was my only config change from 3.0.4.
> Them rebooting back into 3.0.4 fixed it.

Oh, so this is 3.1 DomU kernel? And your Dom0 is 2.6.18?

What does the guest show for the disk? As in, is the barrier feature
enabled? 

.. snip
> ------------[ cut here ]------------
> WARNING: at fs/ext3/inode.c:1695 ext3_writeback_writepage+0x12a/0x150()
> Modules linked in:
> Pid: 2146, comm: flush-202:0 Not tainted 3.1.0-linode39 #1
> Call Trace:
>  [<c01346f8>] ? warn_slowpath_common+0x78/0xb0
>  [<c025643a>] ? ext3_writeback_writepage+0x12a/0x150
>  [<c025643a>] ? ext3_writeback_writepage+0x12a/0x150
>  [<c013474b>] ? warn_slowpath_null+0x1b/0x20
>  [<c025643a>] ? ext3_writeback_writepage+0x12a/0x150
>  [<c01067e4>] ? check_events+0x8/0xc
>  [<c018c3c8>] ? __writepage+0x8/0x30
>  [<c018cbc6>] ? write_cache_pages+0x196/0x390
>  [<c018c3c0>] ? set_page_dirty+0x60/0x60
>  [<c010dfd8>] ? sched_clock+0x8/0x10
>  [<c0154452>] ? sched_clock_local+0xb2/0x190
>  [<c018cdfe>] ? generic_writepages+0x3e/0x60
>  [<c01d4ae5>] ? writeback_single_inode+0x115/0x2b0
>  [<c0102b18>] ? xen_end_context_switch+0x8/0x10
>  [<c01d52c1>] ? writeback_sb_inodes+0x171/0x210
>  [<c01d58d5>] ? __writeback_inodes_wb+0x75/0xb0
>  [<c01d5b31>] ? wb_writeback+0x1a1/0x1b0
>  [<c0106007>] ? xen_force_evtchn_callback+0x17/0x30
>  [<c01067e4>] ? check_events+0x8/0xc
>  [<c01d5cef>] ? wb_do_writeback+0x1af/0x1c0
>  [<c01d5d71>] ? bdi_writeback_thread+0x71/0x130
>  [<c01d5d00>] ? wb_do_writeback+0x1c0/0x1c0
>  [<c014e1a4>] ? kthread+0x74/0x80
>  [<c014e130>] ? kthread_worker_fn+0x170/0x170
>  [<c06c63b6>] ? kernel_thread_helper+0x6/0x10
> ---[ end trace 0a37bc36a180e73c ]---
> 
> Am I barking up the wrong list?

Not sure.. It depends on what your dom0 version is and what
the backend is? blktap or blkback?

Re: Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c

[Christopher S. Aker]

On Oct 25, 2011, at 4:12 PM, Konrad Rzeszutek Wilk wrote:
> Oh, so this is 3.1 DomU kernel? And your Dom0 is 2.6.18?
> 
> What does the guest show for the disk? As in, is the barrier feature
> enabled? 

2.6.18 dom0 and a 3.1.0 domU produces this on domU boot:

EXT3-fs (xvda): using internal journal
Adding 262140k swap on /dev/xvdb.  Priority:-1 extents:1 across:262140k SS
blkfront: barrier: empty write xvda op failed
blkfront: xvda: barrier or flush: disabled
end_request: I/O error, dev xvda, sector 1053048
end_request: I/O error, dev xvda, sector 1053048
Buffer I/O error on device xvda, logical block 131631
lost page write due to I/O error on xvda
Aborting journal on device xvda.
EXT3-fs (xvda): error: ext3_journal_start_sb: Detected aborted journal
EXT3-fs (xvda): error: remounting filesystem read-only

On my 3.0 based dom0 with a 3.1.0 domU blkfront doesn't complain - and I *think* 3.1 domU may actually work as expected -- initial tests couldn't get it to trace...

> .. snip
>> 
>> Am I barking up the wrong list?
> 
> Not sure.. It depends on what your dom0 version is and what
> the backend is? blktap or blkback?

blkback.

We suspected a dom0/domU incompatibility after doing some research.  Hopefully this can get resolved within the guest since we have literally thousands of 2.6.18 dom0s deployed and rebooting everyone is not my idea of a good time :)

Thanks,
-Chris

Re: Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c

[Konrad Rzeszutek Wilk]

On Tue, Oct 25, 2011 at 04:40:09PM -0400, Christopher S. Aker wrote:
> On Oct 25, 2011, at 4:12 PM, Konrad Rzeszutek Wilk wrote:
> > Oh, so this is 3.1 DomU kernel? And your Dom0 is 2.6.18?
> > 
> > What does the guest show for the disk? As in, is the barrier feature
> > enabled? 
> 
> 2.6.18 dom0 and a 3.1.0 domU produces this on domU boot:

You see the same exact thing with 3.0.6?

> 
> EXT3-fs (xvda): using internal journal
> Adding 262140k swap on /dev/xvdb.  Priority:-1 extents:1 across:262140k SS
> blkfront: barrier: empty write xvda op failed
> blkfront: xvda: barrier or flush: disabled
> end_request: I/O error, dev xvda, sector 1053048
> end_request: I/O error, dev xvda, sector 1053048
> Buffer I/O error on device xvda, logical block 131631
> lost page write due to I/O error on xvda
> Aborting journal on device xvda.
> EXT3-fs (xvda): error: ext3_journal_start_sb: Detected aborted journal
> EXT3-fs (xvda): error: remounting filesystem read-only
> 
> On my 3.0 based dom0 with a 3.1.0 domU blkfront doesn't complain - and I *think* 3.1 domU may actually work as expected -- initial tests couldn't get it to trace...
> 
> > .. snip
> >> 
> >> Am I barking up the wrong list?
> > 
> > Not sure.. It depends on what your dom0 version is and what
> > the backend is? blktap or blkback?
> 
> blkback.
> 
> We suspected a dom0/domU incompatibility after doing some research.  Hopefully this can get resolved within the guest since we have literally thousands of 2.6.18 dom0s deployed and rebooting everyone is not my idea of a good time :)
> 
> Thanks,
> -Chris

Re: Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c

[Konrad Rzeszutek Wilk]

On Tue, Oct 25, 2011 at 04:40:09PM -0400, Christopher S. Aker wrote:
> On Oct 25, 2011, at 4:12 PM, Konrad Rzeszutek Wilk wrote:
> > Oh, so this is 3.1 DomU kernel? And your Dom0 is 2.6.18?
> > 
> > What does the guest show for the disk? As in, is the barrier feature
> > enabled? 
> 
> 2.6.18 dom0 and a 3.1.0 domU produces this on domU boot:
> 
> EXT3-fs (xvda): using internal journal
> Adding 262140k swap on /dev/xvdb.  Priority:-1 extents:1 across:262140k SS
> blkfront: barrier: empty write xvda op failed
> blkfront: xvda: barrier or flush: disabled
> end_request: I/O error, dev xvda, sector 1053048
> end_request: I/O error, dev xvda, sector 1053048
> Buffer I/O error on device xvda, logical block 131631
> lost page write due to I/O error on xvda
> Aborting journal on device xvda.
> EXT3-fs (xvda): error: ext3_journal_start_sb: Detected aborted journal
> EXT3-fs (xvda): error: remounting filesystem read-only
> 
> On my 3.0 based dom0 with a 3.1.0 domU blkfront doesn't complain - and I *think* 3.1 domU may actually work as expected -- initial tests couldn't get it to trace...

OK, that might be very well true as the 3.0 dom0 would export the 'feature-flush-cache'..

but you said that if you use 3.0 domU with 2.6.18 you do not get the
type of warnings as above?


Also, when the DomU guest starts it should report if it "thinks" it
supports barriers. Something like:

blkbfront: xvda: barrier

Or is this the only message you get from blkfront??

> 
> > .. snip
> >> 
> >> Am I barking up the wrong list?
> > 
> > Not sure.. It depends on what your dom0 version is and what
> > the backend is? blktap or blkback?
> 
> blkback.
> 
> We suspected a dom0/domU incompatibility after doing some research.  Hopefully this can get resolved within the guest since we have literally thousands of 2.6.18 dom0s deployed and rebooting everyone is not my idea of a good time :)

> 
> Thanks,
> -Chris
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c

[Christopher S. Aker]

On 10/25/11 5:15 PM, Konrad Rzeszutek Wilk wrote:
> You see the same exact thing with 3.0.6?

It works with 3.0.4 - do you need me to try 3.0.6 specifically?

> but you said that if you use 3.0 domU with 2.6.18 you do not get the
> type of warnings as above?

Correct.

> Also, when the DomU guest starts it should report if it "thinks" it
> supports barriers. Something like: blkbfront: xvda: barrier
>
> Or is this the only message you get from blkfront??

2.6.18 dom0 + 3.0.4 domU produces this and nothing else:

blkfront: xvda: barrier: enabled
blkfront: xvdb: barrier: enabled

2.6.18 dom0 = works with 3.0.4, breaks with 3.1.

Thanks,
-Chris

Re: Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c

[Konrad Rzeszutek Wilk]

On Wed, Oct 26, 2011 at 04:26:40PM -0400, Christopher S. Aker wrote:
> On 10/25/11 5:15 PM, Konrad Rzeszutek Wilk wrote:
> >You see the same exact thing with 3.0.6?
> 
> It works with 3.0.4 - do you need me to try 3.0.6 specifically?

No need.
> 
> >but you said that if you use 3.0 domU with 2.6.18 you do not get the
> >type of warnings as above?
> 
> Correct.
> 
> >Also, when the DomU guest starts it should report if it "thinks" it
> >supports barriers. Something like: blkbfront: xvda: barrier
> >
> >Or is this the only message you get from blkfront??
> 
> 2.6.18 dom0 + 3.0.4 domU produces this and nothing else:
> 
> blkfront: xvda: barrier: enabled
> blkfront: xvdb: barrier: enabled
> 
> 2.6.18 dom0 = works with 3.0.4, breaks with 3.1.

OK, so there is nothing that changed in 3.0 through 3.1 in xen-blkfront.c.
Well, there was a name change, but nothing else.

I am going to bet that if you compile 3.0.4 or 3.0.6 with that
fancy new config option that turns EXT3_ORDERED_something you will
see the same exact problem.

Re: Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c

[Christopher S. Aker]

On 10/26/11 4:36 PM, Konrad Rzeszutek Wilk wrote:
 > I am going to bet that if you compile 3.0.4 or 3.0.6 with that
 > fancy new config option that turns EXT3_ORDERED_something you will
 > see the same exact problem.

Switching that back to match my 3.0 config is the first thing I tried, 
with no joy.

I'll reset and try a few more ideas...

-Chris

Re: Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c

[Konrad Rzeszutek Wilk]

On Wed, Oct 26, 2011 at 05:23:40PM -0400, Christopher S. Aker wrote:
> On 10/26/11 4:36 PM, Konrad Rzeszutek Wilk wrote:
> > I am going to bet that if you compile 3.0.4 or 3.0.6 with that
> > fancy new config option that turns EXT3_ORDERED_something you will
> > see the same exact problem.
> 
> Switching that back to match my 3.0 config is the first thing I
> tried, with no joy.
> 
> I'll reset and try a few more ideas...

OK. I am kind of lost of what made all this happen - there was nothing
between 3.0 and 3.1 in the Xen blkfront code.

Perhaps there is something new and fancy in the Ext3 (or ext4) that is
causing all this mayhem?

Re: Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c

[Pasi Kärkkäinen]

On Thu, Oct 27, 2011 at 11:41:59AM -0400, Konrad Rzeszutek Wilk wrote:
> On Wed, Oct 26, 2011 at 05:23:40PM -0400, Christopher S. Aker wrote:
> > On 10/26/11 4:36 PM, Konrad Rzeszutek Wilk wrote:
> > > I am going to bet that if you compile 3.0.4 or 3.0.6 with that
> > > fancy new config option that turns EXT3_ORDERED_something you will
> > > see the same exact problem.
> > 
> > Switching that back to match my 3.0 config is the first thing I
> > tried, with no joy.
> > 
> > I'll reset and try a few more ideas...
> 
> OK. I am kind of lost of what made all this happen - there was nothing
> between 3.0 and 3.1 in the Xen blkfront code.
> 
> Perhaps there is something new and fancy in the Ext3 (or ext4) that is
> causing all this mayhem?
> 

Could it be?:
http://kernelnewbies.org/Linux_3.1#head-12eb55d046763d8b789ff44c50b96c41b203d65e

"Filesystem barriers enabled by default in Ext3"

-- Pasi

Re: Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c

[Konrad Rzeszutek Wilk]

On Thu, Oct 27, 2011 at 07:23:28PM +0300, Pasi Kärkkäinen wrote:
> On Thu, Oct 27, 2011 at 11:41:59AM -0400, Konrad Rzeszutek Wilk wrote:
> > On Wed, Oct 26, 2011 at 05:23:40PM -0400, Christopher S. Aker wrote:
> > > On 10/26/11 4:36 PM, Konrad Rzeszutek Wilk wrote:
> > > > I am going to bet that if you compile 3.0.4 or 3.0.6 with that
> > > > fancy new config option that turns EXT3_ORDERED_something you will
> > > > see the same exact problem.
> > > 
> > > Switching that back to match my 3.0 config is the first thing I
> > > tried, with no joy.
> > > 
> > > I'll reset and try a few more ideas...
> > 
> > OK. I am kind of lost of what made all this happen - there was nothing
> > between 3.0 and 3.1 in the Xen blkfront code.
> > 
> > Perhaps there is something new and fancy in the Ext3 (or ext4) that is
> > causing all this mayhem?
> > 
> 
> Could it be?:
> http://kernelnewbies.org/Linux_3.1#head-12eb55d046763d8b789ff44c50b96c41b203d65e
> 
> "Filesystem barriers enabled by default in Ext3"

Oh, <smacks his head>

I hadn't realized this was a patch - thought it was some CONFIG option to turn it on.

So Christopher if this:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=00eacd66cd8ab5fff9df49aa3f261ad43d495434

is reverted do you get the problems?

Re: Linux 3.1 domU + ext3 = WARNING: at fs/ext3/inode.c

[Konrad Rzeszutek Wilk]

On Wed, Oct 26, 2011 at 05:23:40PM -0400, Christopher S. Aker wrote:
> On 10/26/11 4:36 PM, Konrad Rzeszutek Wilk wrote:
> > I am going to bet that if you compile 3.0.4 or 3.0.6 with that
> > fancy new config option that turns EXT3_ORDERED_something you will
> > see the same exact problem.
> 
> Switching that back to match my 3.0 config is the first thing I
> tried, with no joy.
> 
> I'll reset and try a few more ideas...

I think I know what the problem is. I believe you need the patch below
in your hosts that are using the older dom0. AT least that solved the problem
for me when I tried to use a 2.6.18 based dom0 with a 3.0 kernel
with barriers. If you are using the linux-2.6.18.hg tree you might want
to just pull it and rebuild it - it has this patch already.

# HG changeset patch
From: Jan Beulich <jbeulich@novell.com>
# Date 1306409621 -3600
# Node ID 876a5aaac0264cf38cae6581e5714b93ec380aaa
# Parent  aedb712c05cf065e943e15d0f38597c2e80f7982
Subject: xen/blkback: don't fail empty barrier requests

The sector number on empty barrier requests may (will?) be
uninitialized (neither bio_init() nor rq_init() set the respective
fields), which allows for exceeding the actual (virtual) disk's size.

Inspired by Konrad's "When writting barriers set the sector number to
zero...", but instead of zapping the sector number (which is wrong for
non-empty ones) just ignore the sector number when the sector count is
zero.

While at it also add overflow checking to the math in vbd_translate().

Signed-off-by: Jan Beulich <jbeulich@novell.com>

diff -r aedb712c05cf -r 876a5aaac026 drivers/xen/blkback/vbd.c
--- a/drivers/xen/blkback/vbd.c	Thu May 26 08:09:04 2011 +0100
+++ b/drivers/xen/blkback/vbd.c	Thu May 26 12:33:41 2011 +0100
@@ -108,8 +108,14 @@ int vbd_translate(struct phys_req *req, 
 	if ((operation != READ) && vbd->readonly)
 		goto out;
 
-	if (unlikely((req->sector_number + req->nr_sects) > vbd_sz(vbd)))
-		goto out;
+	if (likely(req->nr_sects)) {
+		blkif_sector_t end = req->sector_number + req->nr_sects;
+
+		if (unlikely(end < req->sector_number))
+			goto out;
+		if (unlikely(end > vbd_sz(vbd)))
+			goto out;
+	}
 
 	req->dev  = vbd->pdevice;
 	req->bdev = vbd->bdev;
> 
> -Chris