ext3 bug

ext3 bug

[Jean-Marc Valin]

[-- Attachment #1: Type: text/plain, Size: 807 bytes --]

Hi,

Looks like I ran into an ext3 bug (or at least the log says so). I got a
bunch of messages like:
ext3_free_blocks_sb: aborting transaction: Journal has aborted in
__ext3_journal_get_undo_access<2>EXT3-fs error (device sda2) in
ext3_free_blocks_sb: Journal has aborted
EXT3-fs error (device sda2): ext3_free_blocks: Freeing blocks in system
zones -Block = 228, count = 1

It happened while I was doing an "rm -rf" on a directory. The "rm" gave
a segfault and now I can't unmount the filesystem: unmount says "device
is busy", even though lsof reports nothing. The filesystem is on a USB
hard disk. The actual dump is in attachment. I'm running Debian unstable
with a custom 2.6.10 kernel on a 1.6 GHz Pentium-M.

	Jean-Marc

-- 
Jean-Marc Valin <Jean-Marc.Valin@USherbrooke.ca>
Université de Sherbrooke

[-- Attachment #2: ext3_bug --]
[-- Type: text/plain, Size: 5420 bytes --]

Feb 27 01:15:48 idefix kernel: ------------[ cut here ]------------
Feb 27 01:15:48 idefix kernel: PREEMPT 
Feb 27 01:15:48 idefix kernel: Modules linked in: msdos sd_mod udf isofs sr_mod usb_storage scsi_mod joydev usbhid appletalk ax25 ipx radeon ipt_state iptable_filter iptable_mangle iptable_nat ip_conntrack ip_tables ipv6 orinoco_cs orinoco hermes pcmcia lp binfmt_misc af_packet parport_pc parport uhci_hcd pci_hotplug intel_agp agpgart yenta_socket pcmcia_core tg3 snd_intel8x0 snd_ac97_codec ehci_hcd usbcore nls_iso8859_1 nls_cp437 vfat fat ppp_async ppp_generic slhc crc_ccitt snd_pcm_oss tsdev evdev snd_pcm snd_timer snd_page_alloc snd_mixer_oss snd soundcore psmouse thermal fan button ac battery cpufreq_ondemand cpufreq_powersave speedstep_centrino freq_table processor
Feb 27 01:15:48 idefix kernel: CPU:    0
Feb 27 01:15:48 idefix kernel: EIP:    0060:[<b01af540>]    Not tainted VLI
Feb 27 01:15:48 idefix kernel: EFLAGS: 00210286   (2.6.10) 
Feb 27 01:15:48 idefix kernel: EIP is at journal_forget+0x1d0/0x220
Feb 27 01:15:48 idefix kernel: eax: 0000005f   ebx: d1f1c000   ecx: b032c7cc   edx: b032c7cc
Feb 27 01:15:48 idefix kernel: esi: b8932d48   edi: bb2ad41c   ebp: dd668080   esp: d1f1dda0
Feb 27 01:15:48 idefix kernel: ds: 007b   es: 007b   ss: 0068
Feb 27 01:15:48 idefix kernel: Process rm (pid: 10370, threadinfo=d1f1c000 task=c97f49e0)
Feb 27 01:15:48 idefix kernel: Stack: b02f67e0 b02e1027 b02f445b 000004ca b02f4571 00000000 be0a5aac b8932d48 
Feb 27 01:15:48 idefix kernel:        dfc002b8 b019c940 dfc002b8 b8932d48 e73d7980 b275f400 b8932d48 00000006 
Feb 27 01:15:48 idefix kernel:        b0aeb448 dfc002b8 be0a5aac b019f028 dfc002b8 00000000 be0a5aac b8932d48 
Feb 27 01:15:48 idefix kernel: Call Trace:
Feb 27 01:15:48 idefix kernel:  [<b019c940>] ext3_forget+0xf0/0x100
Feb 27 01:15:48 idefix kernel:  [<b019f028>] ext3_clear_blocks+0x118/0x170
Feb 27 01:15:48 idefix kernel:  [<b019f118>] ext3_free_data+0x98/0x150
Feb 27 01:15:48 idefix kernel:  [<b019f2bc>] ext3_free_branches+0xec/0x270
Feb 27 01:15:48 idefix kernel:  [<b019f8ab>] ext3_truncate+0x46b/0x5d0
Feb 27 01:15:48 idefix kernel:  [<b01a08b8>] ext3_mark_iloc_dirty+0x28/0x40
Feb 27 01:15:48 idefix kernel:  [<b01ae12d>] journal_start+0xad/0xe0
Feb 27 01:15:48 idefix kernel:  [<b01a5234>] __ext3_journal_stop+0x24/0x50
Feb 27 01:15:48 idefix kernel:  [<b019c9a9>] start_transaction+0x29/0x70
Feb 27 01:15:48 idefix kernel:  [<b019cb28>] ext3_delete_inode+0xc8/0x100
Feb 27 01:15:48 idefix kernel:  [<b019ca60>] ext3_delete_inode+0x0/0x100
Feb 27 01:15:48 idefix kernel:  [<b01726f5>] generic_delete_inode+0xa5/0x170
Feb 27 01:15:48 idefix kernel:  [<b01729a3>] iput+0x63/0x90
Feb 27 01:15:48 idefix kernel:  [<b0167f27>] sys_unlink+0xd7/0x150
Feb 27 01:15:48 idefix kernel:  [<b016ad40>] sys_getdents64+0xa0/0xaa
Feb 27 01:15:48 idefix kernel:  [<b016aba0>] filldir64+0x0/0x100
Feb 27 01:15:48 idefix kernel:  [<b01030df>] syscall_call+0x7/0xb
Feb 27 01:15:48 idefix kernel: Code: 2f b0 b8 71 45 2f b0 89 44 24 10 b8 ca 04 00 00 89 44 24 0c b8 5b 44 2f b0 89 44 24 08 b8 27 10 2e b0 89 44 24 04 e8 c0 a6 f6 ff <0f> 0b ca 04 5b 44 2f b0 e9 4d ff ff ff c7 04 24 e0 67 2f b0 b8 
Feb 27 01:15:48 idefix kernel:  <6>note: rm[10370] exited with preempt_count 2
Feb 27 01:15:48 idefix kernel:  [<b02d8772>] schedule+0x532/0x540
Feb 27 01:15:48 idefix kernel:  [<b0146c53>] unmap_page_range+0x53/0x80
Feb 27 01:15:48 idefix kernel:  [<b0146e36>] unmap_vmas+0x1b6/0x1d0
Feb 27 01:15:48 idefix kernel:  [<b014b53d>] exit_mmap+0x7d/0x160
Feb 27 01:15:48 idefix kernel:  [<b0117617>] mmput+0x37/0xa0
Feb 27 01:15:48 idefix kernel:  [<b011c06f>] do_exit+0x16f/0x470
Feb 27 01:15:48 idefix kernel:  [<b01046a0>] do_invalid_op+0x0/0xd0
Feb 27 01:15:48 idefix kernel:  [<b01042cb>] die+0x18b/0x190
Feb 27 01:15:48 idefix kernel:  [<b0104752>] do_invalid_op+0xb2/0xd0
Feb 27 01:15:48 idefix kernel:  [<b01af540>] journal_forget+0x1d0/0x220
Feb 27 01:15:48 idefix kernel:  [<b01162d1>] __wake_up_common+0x41/0x70
Feb 27 01:15:48 idefix kernel:  [<b0119e9f>] release_console_sem+0xbf/0xd0
Feb 27 01:15:48 idefix kernel:  [<b0103b17>] error_code+0x2b/0x30
Feb 27 01:15:48 idefix kernel:  [<b01af540>] journal_forget+0x1d0/0x220
Feb 27 01:15:48 idefix kernel:  [<b019c940>] ext3_forget+0xf0/0x100
Feb 27 01:15:48 idefix kernel:  [<b019f028>] ext3_clear_blocks+0x118/0x170
Feb 27 01:15:48 idefix kernel:  [<b019f118>] ext3_free_data+0x98/0x150
Feb 27 01:15:48 idefix kernel:  [<b019f2bc>] ext3_free_branches+0xec/0x270
Feb 27 01:15:48 idefix kernel:  [<b019f8ab>] ext3_truncate+0x46b/0x5d0
Feb 27 01:15:48 idefix kernel:  [<b01a08b8>] ext3_mark_iloc_dirty+0x28/0x40
Feb 27 01:15:48 idefix kernel:  [<b01ae12d>] journal_start+0xad/0xe0
Feb 27 01:15:48 idefix kernel:  [<b01a5234>] __ext3_journal_stop+0x24/0x50
Feb 27 01:15:48 idefix kernel:  [<b019c9a9>] start_transaction+0x29/0x70
Feb 27 01:15:48 idefix kernel:  [<b019cb28>] ext3_delete_inode+0xc8/0x100
Feb 27 01:15:48 idefix kernel:  [<b019ca60>] ext3_delete_inode+0x0/0x100
Feb 27 01:15:48 idefix kernel:  [<b01726f5>] generic_delete_inode+0xa5/0x170
Feb 27 01:15:48 idefix kernel:  [<b01729a3>] iput+0x63/0x90
Feb 27 01:15:48 idefix kernel:  [<b0167f27>] sys_unlink+0xd7/0x150
Feb 27 01:15:48 idefix kernel:  [<b016ad40>] sys_getdents64+0xa0/0xaa
Feb 27 01:15:48 idefix kernel:  [<b016aba0>] filldir64+0x0/0x100
Feb 27 01:15:48 idefix kernel:  [<b01030df>] syscall_call+0x7/0xb

Re: ext3 bug

[Parag Warudkar]

On Sunday 27 February 2005 02:04 am, Jean-Marc Valin wrote:
> Hi,
>
> Looks like I ran into an ext3 bug (or at least the log says so). I got a
> bunch of messages like:
> ext3_free_blocks_sb: aborting transaction: Journal has aborted in
> __ext3_journal_get_undo_access<2>EXT3-fs error (device sda2) in
> ext3_free_blocks_sb: Journal has aborted
> EXT3-fs error (device sda2): ext3_free_blocks: Freeing blocks in system
> zones -Block = 228, count = 1
>
> It happened while I was doing an "rm -rf" on a directory. The "rm" gave
> a segfault and now I can't unmount the filesystem: unmount says "device
> is busy", even though lsof reports nothing. The filesystem is on a USB
> hard disk. The actual dump is in attachment. I'm running Debian unstable
> with a custom 2.6.10 kernel on a 1.6 GHz Pentium-M.
>
> 	Jean-Marc

Please try stock kernel. 2.6.11-rc3 onwards should be fine. - I saw a similar 
problem while running 2.6.10 kernel from Fedora Core 3. It doesn't happen 
with stock kernels.

Parag

Re: ext3 bug

[Dave Jones]

On Sun, Feb 27, 2005 at 02:06:30PM -0500, Parag Warudkar wrote:
 > On Sunday 27 February 2005 02:04 am, Jean-Marc Valin wrote:
 > > Hi,
 > >
 > > Looks like I ran into an ext3 bug (or at least the log says so). I got a
 > > bunch of messages like:
 > > ext3_free_blocks_sb: aborting transaction: Journal has aborted in
 > > __ext3_journal_get_undo_access<2>EXT3-fs error (device sda2) in
 > > ext3_free_blocks_sb: Journal has aborted
 > > EXT3-fs error (device sda2): ext3_free_blocks: Freeing blocks in system
 > > zones -Block = 228, count = 1
 > >
 > > It happened while I was doing an "rm -rf" on a directory. The "rm" gave
 > > a segfault and now I can't unmount the filesystem: unmount says "device
 > > is busy", even though lsof reports nothing. The filesystem is on a USB
 > > hard disk. The actual dump is in attachment. I'm running Debian unstable
 > > with a custom 2.6.10 kernel on a 1.6 GHz Pentium-M.
 > >
 > > 	Jean-Marc
 > 
 > Please try stock kernel. 2.6.11-rc3 onwards should be fine. - I saw a similar 
 > problem while running 2.6.10 kernel from Fedora Core 3. It doesn't happen 
 > with stock kernels.

Which is very odd considering the only ext3 patches in the Fedora
kernel are in 2.6.11rc.

		Dave

Re: ext3 bug

[Parag Warudkar]

On Sunday 27 February 2005 02:27 pm, Dave Jones wrote:
> Which is very odd considering the only ext3 patches in the Fedora
> kernel are in 2.6.11rc.
This seems to be more of an USB-Storage issue than ext3. 

Re: ext3 bug

[Jean-Marc Valin]

> Please try stock kernel. 2.6.11-rc3 onwards should be fine. - I saw a similar 
> problem while running 2.6.10 kernel from Fedora Core 3. It doesn't happen 
> with stock kernels.

I did use a stock 2.6.10 kernel (I said custom in the sense that it
wasn't a Debian kernel). After a reboot, I was able to run fsck on the
disk (many, many errors) and it went fine after.

	Jean-Marc

-- 
Jean-Marc Valin <Jean-Marc.Valin@USherbrooke.ca>
Université de Sherbrooke

Re: ext3 bug

[Parag Warudkar]

On Sunday 27 February 2005 05:58 pm, Jean-Marc Valin wrote:
> I did use a stock 2.6.10 kernel (I said custom in the sense that it
> wasn't a Debian kernel). After a reboot, I was able to run fsck on the
> disk (many, many errors) and it went fine after.

Hmm.. So that error is not FC3 specific, it is present in stock 2.6.10 as 
well.  Also - This is on a USB disk, right? If so, the error may re-surface. 
Try upgrading to latest kernel if possible. 

Parag

Re: ext3 bug

[Jean-Marc Valin]

> Hmm.. So that error is not FC3 specific, it is present in stock 2.6.10 as 
> well.  Also - This is on a USB disk, right? If so, the error may re-surface. 
> Try upgrading to latest kernel if possible. 

It's a USB disk (3.5" IDE + IDE to USB). What has been changed in
2.6.11-rcX?

	Jean-Marc

-- 
Jean-Marc Valin <Jean-Marc.Valin@USherbrooke.ca>
Université de Sherbrooke

Re: ext3 bug

[Parag Warudkar]

On Sunday 27 February 2005 09:04 pm, Jean-Marc Valin wrote:
> What has been changed in
> 2.6.11-rcX?

Don't know exactly what changed but when I faced this issue Greg KH suggested 
to reproduce on 2.6.11-rc3 and it didn't happen there.

(I am assuming that the issue you are seeing is the same as what I saw since 
the  error message looks similar and even I was using a USB disk when the 
error happened.)

Parag

Re: ext3 bug

[jmerkey]

I see this problem infrequently on systems that have low memory 
conditions and
with heavy swapping.    I have not seen it on 2.6.9 but I have seen it 
on 2.6.10. 

Jeff

Jean-Marc Valin wrote:

>>Please try stock kernel. 2.6.11-rc3 onwards should be fine. - I saw a similar 
>>problem while running 2.6.10 kernel from Fedora Core 3. It doesn't happen 
>>with stock kernels.
>>    
>>
>
>I did use a stock 2.6.10 kernel (I said custom in the sense that it
>wasn't a Debian kernel). After a reboot, I was able to run fsck on the
>disk (many, many errors) and it went fine after.
>
>	Jean-Marc
>
>  
>

Re: ext3 bug

[Jean-Marc Valin]

Le lundi 28 février 2005 à 08:31 -0700, jmerkey a écrit :
> I see this problem infrequently on systems that have low memory 
> conditions and
> with heavy swapping.    I have not seen it on 2.6.9 but I have seen it 
> on 2.6.10. 

My machine has 1 GB RAM and I wasn't using much of it at that time (2GB
free on the swap), so I doubt that's the problem in my case.

	Jean-Marc

-- 
Jean-Marc Valin <Jean-Marc.Valin@USherbrooke.ca>
Université de Sherbrooke

Re: ext3 bug

[jmerkey]

Jean-Marc Valin wrote:

>Le lundi 28 février 2005 à 08:31 -0700, jmerkey a écrit :
>  
>
>>I see this problem infrequently on systems that have low memory 
>>conditions and
>>with heavy swapping.    I have not seen it on 2.6.9 but I have seen it 
>>on 2.6.10. 
>>    
>>
>
>My machine has 1 GB RAM and I wasn't using much of it at that time (2GB
>free on the swap), so I doubt that's the problem in my case.
>
>	Jean-Marc
>
>  
>
Running the ext2 recover program seems to trigger some good bugs in 
2.6.10 with ext3 -- try it.  I was doing this
to test some disk tools and I managed to cause these errors with forcing 
ext2 recovery from an ext3 fs (which is
probably something to be expected.  The recover tools need to get 
syncrhonized -- have not tried with
mc yet.)    Doesn't happen every time though.

Jeff

Re: ext3 bug

[jmerkey]

jmerkey wrote:

> Jean-Marc Valin wrote:
>
>> Le lundi 28 février 2005 à 08:31 -0700, jmerkey a écrit :
>>  
>>
>>> I see this problem infrequently on systems that have low memory 
>>> conditions and
>>> with heavy swapping.    I have not seen it on 2.6.9 but I have seen 
>>> it on 2.6.10.   
>>
>>
>> My machine has 1 GB RAM and I wasn't using much of it at that time (2GB
>> free on the swap), so I doubt that's the problem in my case.
>>
>>     Jean-Marc
>>
>>  
>>
> Running the ext2 recover program seems to trigger some good bugs in 
> 2.6.10 with ext3 -- try it.  I was doing this
> to test some disk tools and I managed to cause these errors with 
> forcing ext2 recovery from an ext3 fs (which is
> probably something to be expected.  The recover tools need to get 
> syncrhonized -- have not tried with
> mc yet.)    Doesn't happen every time though.
>
> Jeff
>
>
>

lde also causes some problems as well with ext3.  Just caused one on 
2.6.10.  stale or poisoned
cache blocks perhaps?

Jeff

Re: ext3 bug

[Jeffrey E. Hundstad]

linux-2.6.10 has some bio problems that are fixed in the current 
linux-2.6.11 release candidates.  The bio problems wreaked havoc with 
XFS and there were people reporting EXT3 problems as well with this 
bug.  I'd recommend trying the latest release candidate and see if your 
problem vanishes.

-- 
jeffrey hundstad


jmerkey wrote:

> jmerkey wrote:
>
>> Jean-Marc Valin wrote:
>>
>>> Le lundi 28 février 2005 à 08:31 -0700, jmerkey a écrit :
>>>  
>>>
>>>> I see this problem infrequently on systems that have low memory 
>>>> conditions and
>>>> with heavy swapping.    I have not seen it on 2.6.9 but I have seen 
>>>> it on 2.6.10.   
>>>
>>>
>>>
>>> My machine has 1 GB RAM and I wasn't using much of it at that time (2GB
>>> free on the swap), so I doubt that's the problem in my case.
>>>
>>>     Jean-Marc
>>>
>>>  
>>>
>> Running the ext2 recover program seems to trigger some good bugs in 
>> 2.6.10 with ext3 -- try it.  I was doing this
>> to test some disk tools and I managed to cause these errors with 
>> forcing ext2 recovery from an ext3 fs (which is
>> probably something to be expected.  The recover tools need to get 
>> syncrhonized -- have not tried with
>> mc yet.)    Doesn't happen every time though.
>>
>> Jeff
>>
>>
>>
>
> lde also causes some problems as well with ext3.  Just caused one on 
> 2.6.10.  stale or poisoned
> cache blocks perhaps?
>
> Jeff
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: ext3 bug

[jmerkey]

Jeffrey E. Hundstad wrote:

> linux-2.6.10 has some bio problems that are fixed in the current 
> linux-2.6.11 release candidates.  The bio problems wreaked havoc with 
> XFS and there were people reporting EXT3 problems as well with this 
> bug.  I'd recommend trying the latest release candidate and see if 
> your problem vanishes.
>
OK

Jeff