ext3: Freeing blocks not in datazone

ext3: Freeing blocks not in datazone

[Mark Lord]

One of my test systems here, running 2.6.25-rc8-git*,
just now crapped out with this in the tail end of dmesg:

[   20.927780] EXT3-fs error (device sda1): ext3_free_blocks: Freeing blocks not in datazone - block
[   20.928756] Aborting journal on device sda1.
[    7.336461] ext3_abort called.
[   20.930208] Remounting filesystem read-only
[    7.336925] EXT3-fs error (device sda1): ext3_journal_start_sb: Detected aborted journal
[    7.336930] Remounting filesystem read-only
[   20.930847] EXT3-fs error (device sda1) in ext3_reserve_inode_write: Journal has aborted
[   20.930960] EXT3-fs error (device sda1) in ext3_truncate: Journal has aborted
[   20.931037] EXT3-fs error (device sda1) in ext3_reserve_inode_write: Journal has aborted
[   20.931148] EXT3-fs error (device sda1) in ext3_orphan_del: Journal has aborted
[   20.931257] EXT3-fs error (device sda1) in ext3_reserve_inode_write: Journal has aborted
[   20.932057] __journal_remove_journal_head: freeing b_committed_data
[   20.932071] __journal_remove_journal_head: freeing b_committed_data
[   20.932165] __journal_remove_journal_head: freeing b_committed_data
[   20.932185] __journal_remove_journal_head: freeing b_committed_data
[   20.932197] __journal_remove_journal_head: freeing b_committed_data

I had just booted the system, and was simultaneously doing "rm -r" on an old kernel tree
while also doing mirrordir to refressh a different kernel tree from a master copy off NFS.

Not much else to say.  e2fsck seems to have recovered things,
and I've repeated the commands successfully after rebooting.

-ml

Re: ext3: Freeing blocks not in datazone

[Eric Sandeen]

Mark Lord wrote:
> One of my test systems here, running 2.6.25-rc8-git*,
> just now crapped out with this in the tail end of dmesg:
> 
> [   20.927780] EXT3-fs error (device sda1): ext3_free_blocks: Freeing blocks not in datazone - block
> [   20.928756] Aborting journal on device sda1.

I've got a similar report from the F9 installer, also on 2.6.25-rc8* ...
 trying to reproduce locally.

Thanks,
-Eric

Re: ext3: Freeing blocks not in datazone

[Mark Lord]

Eric Sandeen wrote:
> Mark Lord wrote:
>> One of my test systems here, running 2.6.25-rc8-git*,
>> just now crapped out with this in the tail end of dmesg:
>>
>> [   20.927780] EXT3-fs error (device sda1): ext3_free_blocks: Freeing blocks not in datazone - block
>> [   20.928756] Aborting journal on device sda1.
> 
> I've got a similar report from the F9 installer, also on 2.6.25-rc8* ...
>  trying to reproduce locally.
..

Well, if it's of any help, here it was on a 2.4GHz Intel QuadCore,
32-bit kernel/userspace, 4GB RAM, PAE-enabled.

Cheers

Re: ext3: Freeing blocks not in datazone

[Eric Sandeen]

Mark Lord wrote:
> Eric Sandeen wrote:
>> Mark Lord wrote:
>>> One of my test systems here, running 2.6.25-rc8-git*,
>>> just now crapped out with this in the tail end of dmesg:
>>>
>>> [   20.927780] EXT3-fs error (device sda1): ext3_free_blocks: Freeing blocks not in datazone - block
>>> [   20.928756] Aborting journal on device sda1.
>> I've got a similar report from the F9 installer, also on 2.6.25-rc8* ...
>>  trying to reproduce locally.
> ..
> 
> Well, if it's of any help, here it was on a 2.4GHz Intel QuadCore,
> 32-bit kernel/userspace, 4GB RAM, PAE-enabled.

Any chance you had done a resize2fs, online or offline, before this?
Just a guess based on the installer thing I'm looking at...

Thanks,
-Eric

Re: ext3: Freeing blocks not in datazone

[Mark Lord]

Eric Sandeen wrote:
> Mark Lord wrote:
>> Eric Sandeen wrote:
>>> Mark Lord wrote:
>>>> One of my test systems here, running 2.6.25-rc8-git*,
>>>> just now crapped out with this in the tail end of dmesg:
>>>>
>>>> [   20.927780] EXT3-fs error (device sda1): ext3_free_blocks: Freeing blocks not in datazone - block
>>>> [   20.928756] Aborting journal on device sda1.
>>> I've got a similar report from the F9 installer, also on 2.6.25-rc8* ...
>>>  trying to reproduce locally.
>> ..
>>
>> Well, if it's of any help, here it was on a 2.4GHz Intel QuadCore,
>> 32-bit kernel/userspace, 4GB RAM, PAE-enabled.
> 
> Any chance you had done a resize2fs, online or offline, before this?
> Just a guess based on the installer thing I'm looking at...
..

No, nothing really special there.
It's just a single drive / single filesystem (K)Ubuntu install originally,
with various test 2.6.25-rc* kernels being substituted in.

There are lots of other drives in the box for driver development,
but they have nothing to do with the ext3 root fs.

My guess would be that the 4 fast CPU cores managed to race on something.

Cheers

Re: ext3: Freeing blocks not in datazone

[Jan Kara]

> Mark Lord wrote:
> > Eric Sandeen wrote:
> >> Mark Lord wrote:
> >>> One of my test systems here, running 2.6.25-rc8-git*,
> >>> just now crapped out with this in the tail end of dmesg:
> >>>
> >>> [   20.927780] EXT3-fs error (device sda1): ext3_free_blocks: Freeing blocks not in datazone - block
> >>> [   20.928756] Aborting journal on device sda1.
> >> I've got a similar report from the F9 installer, also on 2.6.25-rc8* ...
> >>  trying to reproduce locally.
> > ..
> > 
> > Well, if it's of any help, here it was on a 2.4GHz Intel QuadCore,
> > 32-bit kernel/userspace, 4GB RAM, PAE-enabled.
> 
> Any chance you had done a resize2fs, online or offline, before this?
> Just a guess based on the installer thing I'm looking at...
  Hmm, Eric, how exactly did the corruption looked like? Are you running
SLUB allocator? I'm just wondering whether it doesn't have something in
common with the memory corruption as discussed in the thread starting at
http://lkml.org/lkml/2008/4/19/85

								Honza
-- 
Jan Kara <jack@suse.cz>
SuSE CR Labs

Re: ext3: Freeing blocks not in datazone

[Eric Sandeen]

Jan Kara wrote:
>> Mark Lord wrote:
>>> Eric Sandeen wrote:
>>>> Mark Lord wrote:
>>>>> One of my test systems here, running 2.6.25-rc8-git*,
>>>>> just now crapped out with this in the tail end of dmesg:
>>>>>
>>>>> [   20.927780] EXT3-fs error (device sda1): ext3_free_blocks: Freeing blocks not in datazone - block
>>>>> [   20.928756] Aborting journal on device sda1.
>>>> I've got a similar report from the F9 installer, also on 2.6.25-rc8* ...
>>>>  trying to reproduce locally.
>>> ..
>>>
>>> Well, if it's of any help, here it was on a 2.4GHz Intel QuadCore,
>>> 32-bit kernel/userspace, 4GB RAM, PAE-enabled.
>> Any chance you had done a resize2fs, online or offline, before this?
>> Just a guess based on the installer thing I'm looking at...
>   Hmm, Eric, how exactly did the corruption looked like? Are you running
> SLUB allocator? I'm just wondering whether it doesn't have something in
> common with the memory corruption as discussed in the thread starting at
> http://lkml.org/lkml/2008/4/19/85

It turns out that that problem (sorry, should have followed up) was due
to the installer not copying the last bit of a filesystem image onto the
device, and the fs was then trying to use whatever it found on the
un-copied portion of the disk as metadata.

So not an ext3 bug in that case.

-Eric