Re: Bug#615998: linux-image-2.6.32-5-xen-amd64: Repeatable "kernel BUG at fs/jbd2/commit.c:534" from Postfix on ext4

[Jan Kara <jack@suse.cz>]

On Fri 24-06-11 11:03:52, Moffett, Kyle D wrote:
> On Jun 24, 2011, at 09:46, Jan Kara wrote:
> > On Thu 23-06-11 16:19:08, Moffett, Kyle D wrote:
> >> Besides which, line 534 in the Debian 2.6.32 kernel I am using is this
> >> one:
> >> 
> >>  J_ASSERT(commit_transaction->t_nr_buffers <=
> >>           commit_transaction->t_outstanding_credits);
> > 
> >  Hmm, OK, so we've used more metadata buffers than we told JBD2 to
> > reserve. I suppose you are not using data=journal mode and the filesystem
> > was created as ext4 (i.e. not converted from ext3), right? Are you using
> > quotas?
> 
> The filesystem *is* using data=journal mode.  If I switch to data=ordered
> or data=writeback, the problem goes away.
  Ah, OK. Then bug https://bugzilla.kernel.org/show_bug.cgi?id=34642 is
probably ext3 incarnation of the same problem and it seems it's still
present even in the current kernel - that ext3 assertion triggered even
with 2.6.39 kernel. Frankly data=journal mode is far less tested than the
other two modes especially with ext4, so I'm not sure how good idea is to
use it in production.

> The filesystems were created as ext4 using the e2fstools in Debian squeeze:
> 1.41.12, and the kernel package is 2.6.32-5-xen-amd64 (2.6.32-34squeeze1).
> 
> The exact commands I used to create the Postfix filesystems were:
>   lvcreate -L  5G -n postfix dbnew
>   lvcreate -L 32M -n smtp    dbnew
>   mke2fs -t ext4 -L db:postfix /dev/dbnew/postfix
>   mke2fs -t ext4 -L db:smtp    /dev/dbnew/smtp
>   tune2fs -i 0 -c 1 -e remount-ro -o acl,user_xattr,journal_data /dev/dbnew/postfix
>   tune2fs -i 0 -c 1 -e remount-ro -o acl,user_xattr,journal_data /dev/dbnew/smtp
> 
> Then my fstab has:
>   /dev/mapper/dbnew-postfix /var/spool/postfix ext4 noauto,noatime,nosuid,nodev 0 2
>   /dev/mapper/dbnew-smtp    /var/lib/postfix   ext4 noauto,noatime,nosuid,nodev 0 2
> 
> I don't even think I have the quota tools installed on this system; there
> are certainly none configured.
  OK, thanks.

> >> If somebody can tell me what information would help to debug this I'd be
> >> more than happy to throw a whole bunch of debug printks under that error
> >> condition and try to trigger the crash with that.
> >> 
> >> Alternatively I could remove that J_ASSERT() and instead add some debug
> >> further down around the "commit_transaction->t_outstanding_credits--;"
> >> to try to see exactly what IO it's handling when it runs out of credits.
> > 
> >  The trouble is that the problem is likely in some journal list shuffling
> > code because if just some operation wrongly estimated the number of needed
> > buffers, we'd fail the assertion in jbd2_journal_dirty_metadata():
> > J_ASSERT_JH(jh, handle->h_buffer_credits > 0);
> 
> Hmm, ok...  I'm also going to turn that failing J_ASSERT() into a WARN_ON()
> just to see how much further it gets.  I have an easy script to recreate this
> data volume even if it gets totally hosed anyways, so...
  OK, we'll see what happens.

> > The patch below might catch the problem closer to the place where it
> > happens...
> > 
> > Also possibly you can try current kernel whether the bug happens with it or
> > not.
> 
> I'm definitely going to try this patch, but I'll also see what I can do about
> trying a more recent kernel.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR