vanilla 2.5.13 severe file system corruption experienced follozing e2fsck ...

vanilla 2.5.13 severe file system corruption experienced follozing e2fsck ...

[william stinson]

Hi

as vanilla linux 2.5.13 compiled beautifully for me
last night one I couldn't resist the temptation to
boot it up and give it a whirl on my workstation (a
monoprocessor AMD  ATHLON on VIA motherboard with
recent 20GB IDE disk and EXT2 file system, NVIDIA
video card). 

Boot went OK until a message something like "checking
filesystems - check forced -mounted  31 times without
verification - verifying now". Shortly afterwards I
got an OOPS message. 


EIP : 0010: [<c01d59cb> Not Tainted
....
<0> Kernel Panic: Aiee, killing interrupt handler! In
interrupt handler - not syncing

Not to worry I try to reboot my stable kernel - this
fails at the mount command (library's needed by mount
command are missing). Impossible to login (password
file must be corrupted too). 

With the rescue disk I run e2fsck and home partition
is  dead (bad superblocks) and nothing recoverable.
The root file system is also corrupted (bad
superblocks but not as badly as home). I have some
other partitions which I haven't checked yet - maybe
some of them survived. 

As I am not subscribed to the list please CC me in any
response. If I can recover the kernel compile I will
try to give some configuration options and try to
decode the full oops message. More details available
on request.

Best regards
William Stinson (wstinsonfr@yahoo.fr.nospam)
 
P.S. 

 The hard disk is using VIA bus master PCI IDE and the
distribution is a "vanilla" mandrake 8.1. I have a
REALTEK  RTL8029 Ethernet Adaptor. USB is with VIA
VT83C572/VT82C586 PCI to VIA Universal Host
controller. 

___________________________________________________________
Do You Yahoo!? -- Une adresse @yahoo.fr gratuite et en français !
Yahoo! Mail : http://fr.mail.yahoo.com

Re: vanilla 2.5.13 severe file system corruption experienced follozing e2fsck ...

[Anton Altaparmakov]

The IDE core has a nasty PIO bug which causes it to scribble randomly over 
your disk (even when read-only mounted or not mounted at all).

Wait for 2.5.14, that has the fix in it, or use the latest -dj kernel which 
has the fix, too.

Note even with that fix IDE (at least TCQ) is really easy to crash when you 
put the system under heavier I/O (at least on my via box)...

Anton

At 19:34 05/05/02, william stinson wrote:
>Hi
>
>as vanilla linux 2.5.13 compiled beautifully for me
>last night one I couldn't resist the temptation to
>boot it up and give it a whirl on my workstation (a
>monoprocessor AMD  ATHLON on VIA motherboard with
>recent 20GB IDE disk and EXT2 file system, NVIDIA
>video card).
>
>Boot went OK until a message something like "checking
>filesystems - check forced -mounted  31 times without
>verification - verifying now". Shortly afterwards I
>got an OOPS message.
>
>
>EIP : 0010: [<c01d59cb> Not Tainted
>....
><0> Kernel Panic: Aiee, killing interrupt handler! In
>interrupt handler - not syncing
>
>Not to worry I try to reboot my stable kernel - this
>fails at the mount command (library's needed by mount
>command are missing). Impossible to login (password
>file must be corrupted too).
>
>With the rescue disk I run e2fsck and home partition
>is  dead (bad superblocks) and nothing recoverable.
>The root file system is also corrupted (bad
>superblocks but not as badly as home). I have some
>other partitions which I haven't checked yet - maybe
>some of them survived.
>
>As I am not subscribed to the list please CC me in any
>response. If I can recover the kernel compile I will
>try to give some configuration options and try to
>decode the full oops message. More details available
>on request.
>
>Best regards
>William Stinson (wstinsonfr@yahoo.fr.nospam)
>
>P.S.
>
>  The hard disk is using VIA bus master PCI IDE and the
>distribution is a "vanilla" mandrake 8.1. I have a
>REALTEK  RTL8029 Ethernet Adaptor. USB is with VIA
>VT83C572/VT82C586 PCI to VIA Universal Host
>controller.
>
>___________________________________________________________
>Do You Yahoo!? -- Une adresse @yahoo.fr gratuite et en français !
>Yahoo! Mail : http://fr.mail.yahoo.com
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: vanilla 2.5.13 severe file system corruption experienced follozing e2fsck ...

[Jens Axboe]

On Sun, May 05 2002, Anton Altaparmakov wrote:
> Note even with that fix IDE (at least TCQ) is really easy to crash when you 
> put the system under heavier I/O (at least on my via box)...

If you have stumpled upon a tcq bug, I'd like to know more about it.

-- 
Jens Axboe

Re: vanilla 2.5.13 severe file system corruption experienced follozing e2fsck ...

[Anton Altaparmakov]

At 06:55 06/05/02, Jens Axboe wrote:
>On Sun, May 05 2002, Anton Altaparmakov wrote:
> > Note even with that fix IDE (at least TCQ) is really easy to crash when 
> you
> > put the system under heavier I/O (at least on my via box)...
>
>If you have stumpled upon a tcq bug, I'd like to know more about it.

Back trace (sorry didn't have ckermit running so didn't catch the whole 
output and was too lazy to write it all down): blk_queue_invalidate_tags, 
tcq_invalidate_queue, ide_dmaq_complete, ide_dmaq_intr, ata_irq_request, 
ide_dmaq_intr, handle_IRQ_event, do_IRQ, ideprobe_init.

At the moment I try to not use 2.5.x as much as possible and only boot into 
it to test ntfs or other changes I make, so when I do that next I will make 
sure I am capturing on the serial console and send you details if I 
experince the panic again...

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: vanilla 2.5.13 severe file system corruption experienced follozing e2fsck ...

[Jens Axboe]

On Mon, May 06 2002, Anton Altaparmakov wrote:
> At 06:55 06/05/02, Jens Axboe wrote:
> >On Sun, May 05 2002, Anton Altaparmakov wrote:
> >> Note even with that fix IDE (at least TCQ) is really easy to crash when 
> >you
> >> put the system under heavier I/O (at least on my via box)...
> >
> >If you have stumpled upon a tcq bug, I'd like to know more about it.
> 
> Back trace (sorry didn't have ckermit running so didn't catch the whole 
> output and was too lazy to write it all down): blk_queue_invalidate_tags, 
> tcq_invalidate_queue, ide_dmaq_complete, ide_dmaq_intr, ata_irq_request, 
> ide_dmaq_intr, handle_IRQ_event, do_IRQ, ideprobe_init.

Same problem as Sebastian I'm sure, in which case the backtrace holds no
info for me, the IDE messages printed _before_ the panic would be
helpful though :-)

> At the moment I try to not use 2.5.x as much as possible and only boot into 
> it to test ntfs or other changes I make, so when I do that next I will make 
> sure I am capturing on the serial console and send you details if I 
> experince the panic again...

Sure, I understand. Just hang on, I'll be releasing a tcq backport for
2.4.19-pre8 today that you can test instead.

-- 
Jens Axboe

tcq problem details Re: vanilla 2.5.13 severe file system corruption experienced follozing e2fsck ...

[Anton Altaparmakov]

Jens,

I didn't get a panic in the limited testing I did just now on 2.5.14 for 
ntfs however I do get soemthing odd. Even when the box is fully idle 
proc/ide/blah/tcq shows this:

TCQ currently on:       yes
Max queue depth:        32
Max achieved depth:     14
Max depth since last:   1
Current depth:          0
Active tags:            [ 1, 3, 4, 6, 9, 11, 12, 14, 17, 19, 20, 22, 25, 
27, 28, 29, 30, 31, ]
Queue:                  released [ 1390 ] - started [ 3986 ]
pending request and queue count mismatch (counted: 18)
DMA status:             not running

Some times the number of active tags is higher, seems to vary...

/me ignorant: this looks wrong. Why are there active tags when no activity? 
If a am right and this is a problem then perhaps tags are "leaking" some how?

-- ide related msgs from boot --
ATA/ATAPI driver v7.0.0
ATA: system bus speed 33MHz
ATA: interface: VIA Technologies, Inc. Bus Master IDE, on PCI slot 00:07.1
ATA: chipset rev.: 6
ATA: non-legacy mode: IRQ probe delayed
VP_IDE: VIA vt82c686b (rev 40) IDE UDMA100 controller on pci00:07.1
     ide0: BM-DMA at 0xd000-0xd007, BIOS settings: hda:DMA, hdb:pio
     ide1: BM-DMA at 0xd008-0xd00f, BIOS settings: hdc:DMA, hdd:DMA
hda: IC35L040AVER07-0, ATA DISK drive
hdc: LITE-ON LTR-12102B, ATAPI CD/DVD-ROM drive
hdd: Maxtor 90288D2, ATA DISK drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
hda: tagged command queueing enabled, command queue depth 32
hda: 80418240 sectors w/1916KiB Cache, CHS=79780/16/63, UDMA(100)
hdd: 5627664 sectors w/256KiB Cache, CHS=5583/16/63, UDMA(33)
hdc: ATAPI 40X CD-ROM CD-R/RW drive, 8192kB Cache, (U)DMA
Uniform CD-ROM driver Revision: 3.12
Partition check:
  hda: [PTBL] [5005/255/63] hda1 hda2 < hda5 hda6 hda7 >
  hdd: [PTBL] [697/128/63] hdd1 hdd2 < hdd5 hdd6 hdd7 hdd8 hdd9 hdd10 >

Best regards,

         Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: tcq problem details Re: vanilla 2.5.13 severe file system corruption experienced follozing e2fsck ...

[Jens Axboe]

On Mon, May 06 2002, Anton Altaparmakov wrote:
> Jens,
> 
> I didn't get a panic in the limited testing I did just now on 2.5.14 for 
> ntfs however I do get soemthing odd. Even when the box is fully idle 
> proc/ide/blah/tcq shows this:
> 
> TCQ currently on:       yes
> Max queue depth:        32
> Max achieved depth:     14
> Max depth since last:   1
> Current depth:          0
> Active tags:            [ 1, 3, 4, 6, 9, 11, 12, 14, 17, 19, 20, 22, 25, 
> 27, 28, 29, 30, 31, ]
> Queue:                  released [ 1390 ] - started [ 3986 ]
> pending request and queue count mismatch (counted: 18)
> DMA status:             not running
> 
> Some times the number of active tags is higher, seems to vary...
> 
> /me ignorant: this looks wrong. Why are there active tags when no activity? 
> If a am right and this is a problem then perhaps tags are "leaking" some 
> how?

Agrh, that's a silly bug in blk_queue_init_tags(). Could you replace the
memset() of tags->tag_index in there with something ala:

	for (i = 0; i < depth; i++)
		tags->tag_index[i] = NULL;

and see if that solves it?

-- 
Jens Axboe

Re: tcq problem details Re: vanilla 2.5.13 severe file system corruption experienced follozing e2fsck ...

[Jens Axboe]

On Mon, May 06 2002, Jens Axboe wrote:
> Agrh, that's a silly bug in blk_queue_init_tags(). Could you replace the
> memset() of tags->tag_index in there with something ala:

Brown paper bag time, this should make it work. Linus, please apply.

# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher.
# This patch includes the following deltas:
#	           ChangeSet	1.549   -> 1.550  
#	drivers/block/ll_rw_blk.c	1.64    -> 1.65   
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 02/05/06	axboe@burns.home.kernel.dk	1.550
# Transposed the last two arguments to memset, causing a slab poisoned
# kernel not to use tagging correctly... Brown paper bag stuff.
# --------------------------------------------
#
diff -Nru a/drivers/block/ll_rw_blk.c b/drivers/block/ll_rw_blk.c
--- a/drivers/block/ll_rw_blk.c	Mon May  6 15:17:09 2002
+++ b/drivers/block/ll_rw_blk.c	Mon May  6 15:17:09 2002
@@ -358,8 +358,8 @@
 	if (!tags->tag_map)
 		goto fail_map;
 
-	memset(tags->tag_index, depth * sizeof(struct request *), 0);
-	memset(tags->tag_map, bits * sizeof(unsigned long), 0);
+	memset(tags->tag_index, 0, depth * sizeof(struct request *));
+	memset(tags->tag_map, 0, bits * sizeof(unsigned long));
 	INIT_LIST_HEAD(&tags->busy_list);
 	tags->busy = 0;
 	tags->max_depth = depth;

-- 
Jens Axboe

Re: vanilla 2.5.13 severe file system corruption experienced follozing e2fsck ...

[Sebastian Droege]

On Mon, 6 May 2002 10:50:33 +0200
Jens Axboe <axboe@suse.de> wrote:

> On Mon, May 06 2002, Anton Altaparmakov wrote:
> > At 06:55 06/05/02, Jens Axboe wrote:
> > >On Sun, May 05 2002, Anton Altaparmakov wrote:
> > >> Note even with that fix IDE (at least TCQ) is really easy to crash when 
> > >you
> > >> put the system under heavier I/O (at least on my via box)...
> > >
> > >If you have stumpled upon a tcq bug, I'd like to know more about it.
> > 
> > Back trace (sorry didn't have ckermit running so didn't catch the whole 
> > output and was too lazy to write it all down): blk_queue_invalidate_tags, 
> > tcq_invalidate_queue, ide_dmaq_complete, ide_dmaq_intr, ata_irq_request, 
> > ide_dmaq_intr, handle_IRQ_event, do_IRQ, ideprobe_init.
> 
> Same problem as Sebastian I'm sure, in which case the backtrace holds no
> info for me, the IDE messages printed _before_ the panic would be
> helpful though :-)
Ok here they are (or do you mean the ide initialisation?):

[normal stuff]

ide_tcq_intr_timeout: timeout waiting for service interrupt...
ide_tcq_intr_timeout: hwgroup not busy
hda: invalidating pending queue (10)
kernel BUG at ll_rw_blk.c:407!

[panic]

Re: vanilla 2.5.13 severe file system corruption experienced follozing e2fsck ...

[Jens Axboe]

On Mon, May 06 2002, Sebastian Droege wrote:
> On Mon, 6 May 2002 10:50:33 +0200
> Jens Axboe <axboe@suse.de> wrote:
> 
> > On Mon, May 06 2002, Anton Altaparmakov wrote:
> > > At 06:55 06/05/02, Jens Axboe wrote:
> > > >On Sun, May 05 2002, Anton Altaparmakov wrote:
> > > >> Note even with that fix IDE (at least TCQ) is really easy to crash when 
> > > >you
> > > >> put the system under heavier I/O (at least on my via box)...
> > > >
> > > >If you have stumpled upon a tcq bug, I'd like to know more about it.
> > > 
> > > Back trace (sorry didn't have ckermit running so didn't catch the whole 
> > > output and was too lazy to write it all down): blk_queue_invalidate_tags, 
> > > tcq_invalidate_queue, ide_dmaq_complete, ide_dmaq_intr, ata_irq_request, 
> > > ide_dmaq_intr, handle_IRQ_event, do_IRQ, ideprobe_init.
> > 
> > Same problem as Sebastian I'm sure, in which case the backtrace holds no
> > info for me, the IDE messages printed _before_ the panic would be
> > helpful though :-)
> Ok here they are (or do you mean the ide initialisation?):
> 
> [normal stuff]
> 
> ide_tcq_intr_timeout: timeout waiting for service interrupt...
> ide_tcq_intr_timeout: hwgroup not busy
> hda: invalidating pending queue (10)
> kernel BUG at ll_rw_blk.c:407!

Thanks, yes these were the messages I meant. Could you try 2.4.19-pre8
plus patches just posted?

-- 
Jens Axboe

Re: tcq problem details Re: vanilla 2.5.13 severe file system corruption experienced follozing e2fsck ...

[Anton Altaparmakov]

At 14:18 06/05/02, Jens Axboe wrote:
>On Mon, May 06 2002, Jens Axboe wrote:
> > Agrh, that's a silly bug in blk_queue_init_tags(). Could you replace the
> > memset() of tags->tag_index in there with something ala:
>
>Brown paper bag time, this should make it work. Linus, please apply.

It now works. (-; running 2.5.14 with ide, tcq, highmem (1G ram), NO 
preemption or smp compiled in, seems to be holding up. ran already one loop 
of my ntfs stress test without problems. now trying second time...

[aia21@drop aia21]$ sudo cat /proc/ide/ide0/hda/tcq
TCQ currently on:       yes
Max queue depth:        32
Max achieved depth:     32
Max depth since last:   4
Current depth:          0
Active tags:            [ ]
Queue:                  released [ 3852 ] - started [ 6539 ]
DMA status:             not running

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: vanilla 2.5.13 severe file system corruption experienced follozing e2fsck ...

[Sebastian Droege]

[-- Attachment #1: Type: text/plain, Size: 1896 bytes --]

On Mon, 6 May 2002 15:48:03 +0200
Jens Axboe <axboe@suse.de> wrote:

> On Mon, May 06 2002, Sebastian Droege wrote:
> > On Mon, 6 May 2002 10:50:33 +0200
> > Jens Axboe <axboe@suse.de> wrote:
> > 
> > > On Mon, May 06 2002, Anton Altaparmakov wrote:
> > > > At 06:55 06/05/02, Jens Axboe wrote:
> > > > >On Sun, May 05 2002, Anton Altaparmakov wrote:
> > > > >> Note even with that fix IDE (at least TCQ) is really easy to crash when 
> > > > >you
> > > > >> put the system under heavier I/O (at least on my via box)...
> > > > >
> > > > >If you have stumpled upon a tcq bug, I'd like to know more about it.
> > > > 
> > > > Back trace (sorry didn't have ckermit running so didn't catch the whole 
> > > > output and was too lazy to write it all down): blk_queue_invalidate_tags, 
> > > > tcq_invalidate_queue, ide_dmaq_complete, ide_dmaq_intr, ata_irq_request, 
> > > > ide_dmaq_intr, handle_IRQ_event, do_IRQ, ideprobe_init.
> > > 
> > > Same problem as Sebastian I'm sure, in which case the backtrace holds no
> > > info for me, the IDE messages printed _before_ the panic would be
> > > helpful though :-)
> > Ok here they are (or do you mean the ide initialisation?):
> > 
> > [normal stuff]
> > 
> > ide_tcq_intr_timeout: timeout waiting for service interrupt...
> > ide_tcq_intr_timeout: hwgroup not busy
> > hda: invalidating pending queue (10)
> > kernel BUG at ll_rw_blk.c:407!
> 
> Thanks, yes these were the messages I meant. Could you try 2.4.19-pre8
> plus patches just posted?
Ok... tested
it panics with a NULL pointer dereference at 00000004 after

hda: IBM-DTTA-351010, ATA DISK drive
hdb: WDC WD800BB-00BSA0, ATA DISK drive
hdc: CD-W512EB, ATAPI CD/DVD-ROM drive
hdd: CD-532E-B, ATAPI CD/DVD-ROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
[panic]

Sorry but I have no more time for testing and oops handcopying today
I'll do that tomorrow
Bye

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: vanilla 2.5.13 severe file system corruption experienced follozing e2fsck ...

[Jens Axboe]

On Mon, May 06 2002, Sebastian Droege wrote:
> On Mon, 6 May 2002 15:48:03 +0200
> Jens Axboe <axboe@suse.de> wrote:
> 
> > On Mon, May 06 2002, Sebastian Droege wrote:
> > > On Mon, 6 May 2002 10:50:33 +0200
> > > Jens Axboe <axboe@suse.de> wrote:
> > > 
> > > > On Mon, May 06 2002, Anton Altaparmakov wrote:
> > > > > At 06:55 06/05/02, Jens Axboe wrote:
> > > > > >On Sun, May 05 2002, Anton Altaparmakov wrote:
> > > > > >> Note even with that fix IDE (at least TCQ) is really easy to crash when 
> > > > > >you
> > > > > >> put the system under heavier I/O (at least on my via box)...
> > > > > >
> > > > > >If you have stumpled upon a tcq bug, I'd like to know more about it.
> > > > > 
> > > > > Back trace (sorry didn't have ckermit running so didn't catch the whole 
> > > > > output and was too lazy to write it all down): blk_queue_invalidate_tags, 
> > > > > tcq_invalidate_queue, ide_dmaq_complete, ide_dmaq_intr, ata_irq_request, 
> > > > > ide_dmaq_intr, handle_IRQ_event, do_IRQ, ideprobe_init.
> > > > 
> > > > Same problem as Sebastian I'm sure, in which case the backtrace holds no
> > > > info for me, the IDE messages printed _before_ the panic would be
> > > > helpful though :-)
> > > Ok here they are (or do you mean the ide initialisation?):
> > > 
> > > [normal stuff]
> > > 
> > > ide_tcq_intr_timeout: timeout waiting for service interrupt...
> > > ide_tcq_intr_timeout: hwgroup not busy
> > > hda: invalidating pending queue (10)
> > > kernel BUG at ll_rw_blk.c:407!
> > 
> > Thanks, yes these were the messages I meant. Could you try 2.4.19-pre8
> > plus patches just posted?
> Ok... tested
> it panics with a NULL pointer dereference at 00000004 after
> 
> hda: IBM-DTTA-351010, ATA DISK drive
> hdb: WDC WD800BB-00BSA0, ATA DISK drive
> hdc: CD-W512EB, ATAPI CD/DVD-ROM drive
> hdd: CD-532E-B, ATAPI CD/DVD-ROM drive
> ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
> ide1 at 0x170-0x177,0x376 on irq 15
> [panic]
> 
> Sorry but I have no more time for testing and oops handcopying today
> I'll do that tomorrow

Ok thanks, it's probably the auto_poll check though. Could you try one
more boot, just commenting out the call to ide_tcq_check_autopoll() in
ide_enable_queued()? It's in drivers/ide/ide-tcq.c.

-- 
Jens Axboe