* Re: ext3 crash with 2.4.22: Assertion failure in journal_forget_R10d91946()
2003-10-06 22:56 ` Marcelo Tosatti
@ 2003-10-07 9:11 ` Stephen C. Tweedie
2003-10-07 22:18 ` Andriy Rysin
2003-10-15 22:26 ` 2.4.20, 2.4.22, 2.4.6-test7: system locks up completely when writing to floppy (2.2.20 is ok) Andriy Rysin
2 siblings, 0 replies; 6+ messages in thread
From: Stephen C. Tweedie @ 2003-10-07 9:11 UTC (permalink / raw)
To: Marcelo Tosatti
Cc: Andriy Rysin, linux-kernel, Andrew Morton, Alan Cox,
Stephen Tweedie
Hi,
On Mon, 2003-10-06 at 23:56, Marcelo Tosatti wrote:
> On Thu, 2 Oct 2003, Andriy Rysin wrote:
> > I am having crashes on ext3 with 2.4.22 kernel. System was up for 8
> > days. I am not sure I can reproduce it real quick but we've seen it
> > occasionly on 2.4.20 for about several months and after we updated to
> > 2.4.22 it's here again.
> > the log looks like this:
> >
> > Sep 29 20:15:08 dunne-demo kernel: EXT3-fs error (device ide0(3,2)):
> > ext3_free_blocks: Freeing blocks not in datazone - bloc
> > k = 2907885836, count = 1
...
> > Oct 2 00:43:53 dunne-demo kernel: hda: dma_timer_expiry: dma status == 0x20
> > Oct 2 00:43:53 dunne-demo kernel: hda: timeout waiting for DMA
> > Oct 2 00:43:53 dunne-demo kernel: hda: timeout waiting for DMA
> > Oct 2 00:43:53 dunne-demo kernel: hda: (__ide_dma_test_irq) called
>
> You are getting DMA timeouts and such. Try turning off the DMA.
The ext3 errors are entirely consistent with disk corruption. They can
occur in many ways: for example, if the drive is dropping writes due to
the IDE errors, you could easily end up with a corrupt indirect block
that would lead to the above errors if you tried to delete the file
again.
The
journal_forget_R10d91946() at transaction.c:1259: "!jh->b_committed_data"
error occurs when we try to discard a buffer that has got "committed
data" versioning info attached. That info is only ever used for bitmap
blocks: it is there to ensure that we don't reallocate freed data until
the transaction that deleted the data has committed (if we crash before
the commit, we need to be able to roll back the delete, so we can't
overwrite the data on disk until after the commit.)
So we're deleting a bitmap block. No wonder jbd is getting worried.
:-) But this indicates that it's yet another consequence of the
underlying corruption of an indirect block.
Current 2.4 ext3 includes a patch from Andreas Dilger which detects when
we're attempting to free metadata, and which avoids the free so that we
don't end up reallocating critical metadata as file contents. I'll have
a look at using that detection to avoid the ext3_forget() on such
blocks.
It's definitely a case that you should never see except on a corrupt
filesystem, so I want to preserve the ability to panic() here for
testing: there's already an ext3 compile-time option you can set in
jbd.h to turn error-detection into assert-failures for errors which can
happen in real life on corrupt filesystems.
--Stephen
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ext3 crash with 2.4.22: Assertion failure in journal_forget_R10d91946()
2003-10-06 22:56 ` Marcelo Tosatti
2003-10-07 9:11 ` Stephen C. Tweedie
@ 2003-10-07 22:18 ` Andriy Rysin
2003-10-15 22:26 ` 2.4.20, 2.4.22, 2.4.6-test7: system locks up completely when writing to floppy (2.2.20 is ok) Andriy Rysin
2 siblings, 0 replies; 6+ messages in thread
From: Andriy Rysin @ 2003-10-07 22:18 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: linux-kernel, sct, Andrew Morton, Alan Cox
Marcelo Tosatti wrote:
>Andriy,
>
>On Thu, 2 Oct 2003, Andriy Rysin wrote:
>
>
>
>>I am having crashes on ext3 with 2.4.22 kernel. System was up for 8
>>days. I am not sure I can reproduce it real quick but we've seen it
>>occasionly on 2.4.20 for about several months and after we updated to
>>2.4.22 it's here again.
>>
>>please CC me if you answer or need more information.
>>
>>
>>the log looks like this:
>>
>>Sep 29 20:15:08 dunne-demo kernel: EXT3-fs error (device ide0(3,2)):
>>ext3_free_blocks: Freeing blocks not in datazone - bloc
>>k = 2907885836, count = 1
>>Sep 29 20:15:08 dunne-demo kernel: EXT3-fs error (device ide0(3,2)):
>>ext3_free_blocks: Freeing blocks not in datazone - bloc
>>k = 1660415916, count = 1
>>Sep 29 20:15:08 dunne-demo kernel: EXT3-fs error (device ide0(3,2)):
>>ext3_free_blocks: Freeing blocks not in datazone - bloc
>>k = 1438298218, count = 1
>>Sep 29 20:15:08 dunne-demo kernel: EXT3-fs error (device ide0(3,2)):
>>ext3_free_blocks: Freeing blocks not in datazone - bloc
>>k = 4209573569, count = 1
>>Sep 29 20:15:08 dunne-demo kernel: EXT3-fs error (device ide0(3,2)):
>>ext3_free_blocks: Freeing blocks not in datazone - bloc
>>k = 2918065562, count = 1
>>......
>>Sep 29 21:05:18 dunne-demo kernel: EXT3-fs error (device ide0(3,2)):
>>ext3_free_blocks: bit already cleared for block 5970190
>>......
>>Oct 2 00:43:53 dunne-demo kernel: hda: dma_timer_expiry: dma status == 0x20
>>Oct 2 00:43:53 dunne-demo kernel: hda: timeout waiting for DMA
>>Oct 2 00:43:53 dunne-demo kernel: hda: timeout waiting for DMA
>>Oct 2 00:43:53 dunne-demo kernel: hda: (__ide_dma_test_irq) called
>>
>>
>
>You are getting DMA timeouts and such. Try turning off the DMA.
>
>But anyway the ext3 fs errors shouldnt happen I guess. Andrew, Stephen?
>
>
If I turn DMA off the system won't be able handle the load we need. This
problem happens under heavy load on different machines. It seems like
IBM and WDC drives give DMA errors while Maxtor don't. But DMA problems
are not quite related to the filesystem problems. On several systems we
had the same ext3 problem while not observing any DMA errors
(particularly Maxtor case). I doubt all those drives are faulty.
I may add that nature of our application is writing media data files in
cycle manner. When filsystem gets close to full the script deletes
oldest files. Usual filesystem size is about 80G, file sizes range from
several kilos to 1.5G with average about several megs. The system can
simultaniously write up to 32 files and deletion happens in parallel.
Usual high load for the systems is about 4-5MB/s on writing reported by
sar -b (not much for reading ~ 300KB/s).
Also what interesting is that DMA errors happen mostly at pretty low
load on disk ~400KB/s.
A week ago I replaced ext3 with jfs on 3 systems and till now don't get
any DMA or filesystem errors. I even was loading those systems with
scripts constantly copying directory with ~4G of files and removing it
for about 4-5 hours (avg loag by sar was 13MB/s for reading and 15MB/s
for writing) and still did not get any problems.
So it seems like it's not the fault of the drives. I am still curious if
jfs somehow puts less load on the drive not causing any DMAs but I'd
like to spend couple of more weeks testing before claiming that.
Andriy
^ permalink raw reply [flat|nested] 6+ messages in thread
* 2.4.20, 2.4.22, 2.4.6-test7: system locks up completely when writing to floppy (2.2.20 is ok)
2003-10-06 22:56 ` Marcelo Tosatti
2003-10-07 9:11 ` Stephen C. Tweedie
2003-10-07 22:18 ` Andriy Rysin
@ 2003-10-15 22:26 ` Andriy Rysin
2003-10-17 18:28 ` 2.4.20, 2.4.22, 2.4.6-test7: system locks up completely when writing to floppy (2.2.20 is ok) - solution Andriy Rysin
2 siblings, 1 reply; 6+ messages in thread
From: Andriy Rysin @ 2003-10-15 22:26 UTC (permalink / raw)
Cc: linux-kernel
I've got two motherboards on which I have this problem, they're ASUS
P4S533-X and P4S533-MX. When I am trying to write something to a floppy
the system hangs completely. dd, mkfs or mount + cp are all the sme. The
floppy does several writing sounds and that's it, the light is on. No
oppses, no panic no log/console messages. I have this with 2.4.20,
2.4.22 both from RedHat and 2.4.6-test7 from
http://people.redhat.com/arjanv/2.5/RPMS.kernel/
I don't have this with DOS :) or linux 2.2.20 from tomsrtbt floppy and I
don't have this on any other motherboards.
I changed floppy, changed floppy cable, tried different BIOS settings,
changed video card, changed hardrives, booted in S mode. Reproducible 100%.
It seems pretty like this message
http://www.uwsg.iu.edu/hypermail/linux/kernel/0202.2/1557.html
dmesg from 2.4.22 is below.
Please CC me if you reply or need more info.
Andriy
Linux version 2.4.22-20.1.2024.2.36.nptl
(bhcompile@daffy.perf.redhat.com) (gcc version 3.2.3 20030422 (Red Hat
Linux 3.2.3-6)) #1 Wed Sep 3 10:53:22 EDT 2003
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000001fffc000 (usable)
BIOS-e820: 000000001fffc000 - 000000001ffff000 (ACPI data)
BIOS-e820: 000000001ffff000 - 0000000020000000 (ACPI NVS)
BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved)
0MB HIGHMEM available.
511MB LOWMEM available.
ACPI: have wakeup address 0xc0001000
On node 0 totalpages: 131068
zone(0): 4096 pages.
zone(1): 126972 pages.
zone(2): 0 pages.
ACPI: RSDP (v000 ASUS ) @ 0x000f5690
ACPI: RSDT (v001 ASUS P4S533-X 0x42302e31 MSFT 0x31313031) @ 0x1fffc000
ACPI: FADT (v001 ASUS P4S533-X 0x42302e31 MSFT 0x31313031) @ 0x1fffc0c0
ACPI: BOOT (v001 ASUS P4S533-X 0x42302e31 MSFT 0x31313031) @ 0x1fffc030
ACPI: MADT (v001 ASUS P4S533-X 0x42302e31 MSFT 0x31313031) @ 0x1fffc058
ACPI: DSDT (v001 ASUS P4S533-X 0x00001000 MSFT 0x0100000b) @ 0x00000000
Kernel command line: auto BOOT_IMAGE=2.4.22 ro
BOOT_FILE=/boot/vmlinuz-2.4.22-20.1.2024.2.36.nptl acpi=ht hdc=ide-scsi
root=LABEL=/ panic=60
ide_setup: hdc=ide-scsi
Initializing CPU#0
Detected 2200.180 MHz processor.
Console: colour VGA+ 80x25
Calibrating delay loop... 4390.91 BogoMIPS
Memory: 514312k/524272k available (1506k kernel code, 9572k reserved,
1110k data, 136k init, 0k highmem)
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode cache hash table entries: 32768 (order: 6, 262144 bytes)
Mount cache hash table entries: 512 (order: 0, 4096 bytes)
Buffer cache hash table entries: 32768 (order: 5, 131072 bytes)
Page-cache hash table entries: 131072 (order: 7, 524288 bytes)
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 128K
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU: After generic, caps: bfebfbff 00000000 00000000 00000000
CPU: Common caps: bfebfbff 00000000 00000000 00000000
CPU: Intel(R) Celeron(R) CPU 2.20GHz stepping 09
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... OK.
POSIX conformance testing by UNIFIX
mtrr: v1.40 (20010327) Richard Gooch (rgooch@atnf.csiro.au)
mtrr: detected mtrr type: Intel
ACPI: Subsystem revision 20030813
ACPI: Interpreter disabled.
PCI: PCI BIOS revision 2.10 entry at 0xf1060, last bus=1
PCI: Using configuration type 1
PCI: Probing PCI hardware
PCI: ACPI tables contain no PCI IRQ routing entries
PCI: Probing PCI hardware (bus 00)
PCI: Using IRQ router default [1039/0962] at 00:02.0
isapnp: Scanning for PnP cards...
isapnp: No Plug & Play device found
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
apm: BIOS version 1.2 Flags 0x03 (Driver version 1.16)
Starting kswapd
VFS: Disk quotas vdquot_6.5.1
Asus Laptop ACPI Extras version 0.24a
Couldn't get the DSDT table header
Error registering Asus Laptop ACPI Extras Driver
-0420: *** Error: Could not allocate an object descriptor
pty: 2048 Unix98 ptys configured
Serial driver version 5.05c (2001-07-08) with MANY_PORTS MULTIPORT
SHARE_IRQ SERIAL_PCI ISAPNP enabled
ttyS0 at 0x03f8 (irq = 4) is a 16550A
ttyS1 at 0x02f8 (irq = 3) is a 16550A
Real Time Clock Driver v1.10e
NET4: Frame Diverter 0.46
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
Uniform Multi-Platform E-IDE driver Revision: 7.00beta4-2.4
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
SIS5513: IDE controller at PCI slot 00:02.5
SIS5513: chipset revision 0
SIS5513: not 100% native mode: will probe irqs later
SIS5513: SiS 962/963 MuTIOL IDE UDMA133 controller
ide0: BM-DMA at 0xb400-0xb407, BIOS settings: hda:DMA, hdb:pio
ide1: BM-DMA at 0xb408-0xb40f, BIOS settings: hdc:DMA, hdd:pio
hda: WDC WD800JB-00ETA0, ATA DISK drive
blk: queue c040f3a0, I/O limit 4095Mb (mask 0xffffffff)
hdc: SONY CD-RW CRX225E, ATAPI CD/DVD-ROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
hda: attached ide-disk driver.
hda: host protected area => 1
hda: 156301488 sectors (80026 MB) w/8192KiB Cache, CHS=9729/255/63,
UDMA(100)
Partition check:
hda: hda1 hda2 hda3 hda4 < hda5 >
ide: late registration of driver.
md: md driver 0.90.0 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
Initializing Cryptographic API
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP, IGMP
IP: routing cache hash table of 4096 buckets, 32Kbytes
TCP: Hash tables configured (established 32768 bind 65536)
Linux IP multicast router 0.06 plus PIM-SM
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
RAMDISK: Compressed image found at block 0
Freeing initrd memory: 145k freed
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: 2.4.20, 2.4.22, 2.4.6-test7: system locks up completely when writing to floppy (2.2.20 is ok) - solution
2003-10-15 22:26 ` 2.4.20, 2.4.22, 2.4.6-test7: system locks up completely when writing to floppy (2.2.20 is ok) Andriy Rysin
@ 2003-10-17 18:28 ` Andriy Rysin
0 siblings, 0 replies; 6+ messages in thread
From: Andriy Rysin @ 2003-10-17 18:28 UTC (permalink / raw)
To: linux-kernel
Andriy Rysin wrote:
> I've got two motherboards on which I have this problem, they're ASUS
> P4S533-X and P4S533-MX. When I am trying to write something to a
> floppy the system hangs completely. dd, mkfs or mount + cp are all the
> sme. The floppy does several writing sounds and that's it, the light
> is on. No oppses, no panic no log/console messages. I have this with
> 2.4.20, 2.4.22 both from RedHat and 2.4.6-test7 from
> http://people.redhat.com/arjanv/2.5/RPMS.kernel/
> I don't have this with DOS :) or linux 2.2.20 from tomsrtbt floppy and
> I don't have this on any other motherboards.
> I changed floppy, changed floppy cable, tried different BIOS settings,
> changed video card, changed hardrives, booted in S mode. Reproducible
> 100%.
> It seems pretty like this message
> http://www.uwsg.iu.edu/hypermail/linux/kernel/0202.2/1557.html
> dmesg from 2.4.22 is below.
> Please CC me if you reply or need more info.
Ok, I've tested kernels from 2.4.0 to 2.4.22 and found that it's a
CONFIG_APM_CPU_IDLE=y which causes machine to hang. It seems like on
ASUS P4533 motherboards this option is unsafe so it must be turned off.
In RedHat kernels they turned on it after 2.4.18 that's why I saw the
hangs. Actually the option help says that some machine may hang and
probably it should not be on by default. Luckily I was able to fix that
adding "apm=idle_threshold=100" to the kernel parameters without
recompiling the kernel.
Maybe these motherboards/BIOS should be blacklisted?
Andriy
^ permalink raw reply [flat|nested] 6+ messages in thread