Re: XFS internal error xfs_da_do_buf(2) at line 2087 of file fs/xfs/xfs_da_btree.c. Caller 0xc01b00bd

Re: XFS internal error xfs_da_do_buf(2) at line 2087 of file fs/xfs/xfs_da_btree.c. Caller 0xc01b00bd

[David Chinner]

On Wed, Mar 14, 2007 at 12:34:29PM +0100, Marco Berizzi wrote:
> Hello everybody.
> Since 2.6.19.2 + commit 7fbbb01dca7704d52ace6f45a805c98a5b0362f9

What commit is that? gitweb search tells me it's an nmi watchdog
change. Doesn't seem likely to change XFS behaviour - can
you post a url to the commit?

> I'm experimenting these errors.
> 2.6.19.1 has been worked good for more
> than 30 days.

With the above commit?

> I have reverted back to 2.6.19.1 to see if
> this problem happens again.

without the above commit?

> find_or_create_page+0x37/0x8e
> _xfs_buf_lookup_pages+0x132/0x2ea
> _xfs_buf_initialize+0xc8/0xf6
> xfs_buf_get_flags+0xf8/0x11d
> xfs_buf_read_flags+0x1c/0x7f
> xfs_trans_read_buf+0x16a/0x34f
> xfs_itobp+0x7c/0x242
> xfs_iread+0x68/0x1d3
> xfs_iget_core+0xe7/0x687
> xfs_iget+0xd8/0x150
> xfs_dir_lookup_int+0x98/0x10e
> xfs_lookup+0x5a/0x90
> xfs_vn_lookup+0x52/0x93

Curious - never seen this before - possibly a corrupted inode
number in the directory has led to this.

> ba 4e 8b cd
> Mar 12 14:35:21 Pleiadi kernel: Filesystem "sda8": XFS internal error
> xfs_da_do_buf(2) at line 2087 of file fs/xfs/xfs_da_btree.c.  Caller
> 0xc01b00bd
> Mar 12 14:35:21 Pleiadi kernel:  [<c01affb3>] xfs_da_do_buf+0x70c/0x7b1
> Mar 12 14:35:21 Pleiadi kernel:  [<c01b00bd>] xfs_da_read_buf+0x30/0x35
> Mar 12 14:35:21 Pleiadi kernel:  [<c01b00bd>] xfs_da_read_buf+0x30/0x35

Hmm - these could simply be follow-on errors from the first
problem - the buffer would now probably be bad or corrupted,
and the directory buffer read code here is saying the buffer
is bad. All the errors appear to have thesame data in the buffer
(which is lacking the correct magic numbers) so i'd say they
are related to the above error.

Can you run xfs_repair on that filesystem and see if reports
(and fixes) any problems?

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS internal error xfs_da_do_buf(2) at line 2087 of file fs/xfs/xfs_da_btree.c. Caller 0xc01b00bd

[David Chinner]

On Fri, Mar 16, 2007 at 12:05:33PM +0100, Marco Berizzi wrote:
> David Chinner wrote:
> 
> > can
> > you post a url to the commit?
> 
> http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.19.y.git;a=commit;h=7fbbb01dca7704d52ace6f45a805c98a5b0362f9

Ok, so an ipsec change. And I see from the history below it
really has nothing to do with this problem. it seems the problem
has something to do with changes between 2.6.19.1 and 2.6.19.2.

There were no changes to XFS between 2.6.19.1 and 2.6.19.2,
so I'm thinking that your problems are related to something
other than XFS. Can you do a git bisect to determine what the
bad patch is?

> > Can you run xfs_repair on that filesystem and see if reports
> > (and fixes) any problems?
> 
> I don't need to run xfs_repair to fix the problem,

Except that the trigger might be on-disk corruption so we
need to rule that out first.

> I only unplug the power cable and reboot the system,
> xfs filesystem are correctly mounted.
> However tell me if I must run xfs_repair to check
> the filesystem.

Yes, you need to run xfs_repair.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS internal error xfs_da_do_buf(2) at line 2087 of file fs/xfs/xfs_da_btree.c. Caller 0xc01b00bd

[Marco Berizzi]

David Chinner wrote:

> Ok, so an ipsec change. And I see from the history below it
> really has nothing to do with this problem. it seems the problem
> has something to do with changes between 2.6.19.1 and 2.6.19.2.

indeed. Yesterday at 13:00 I have switched from 2.6.19.1 to 2.6.19.2
(without the ipsec fix) and at about 17:30 linux has crashed again.
I have recompiled 2.6.19.2 with all kernel debugging options enabled
and rebooted. Now I'm waiting for the crash...

> There were no changes to XFS between 2.6.19.1 and 2.6.19.2,
> so I'm thinking that your problems are related to something
> other than XFS. Can you do a git bisect to determine what the
> bad patch is?

Ok, monday morning I will try to do a git bisect. I just want tell you
that this machine is a firewall (3 nic) + ipsec gateway (openswan) +
http proxy (squid). I think that the problem is related to the networking.
I have other 2.6.19/2.6.20 linux boxes acting as proftpd/samba/sendmail
without any problem.

>> > Can you run xfs_repair on that filesystem and see if reports
>> > (and fixes) any problems?
>> 
>> I don't need to run xfs_repair to fix the problem,
> 
> Except that the trigger might be on-disk corruption so we
> need to rule that out first.

Ok I will run xfs_repair and post results.

>> I only unplug the power cable and reboot the system,
>> xfs filesystem are correctly mounted.
>> However tell me if I must run xfs_repair to check
>> the filesystem.
> 
> Yes, you need to run xfs_repair.

ok , thanks a lot for the feedback.

PS: I'm running slackware 11 with xfsprogs-2.8.10. Are they fine
or should I upgrade?

Re: XFS internal error xfs_da_do_buf(2) at line 2087 of file fs/xfs/xfs_da_btree.c. Caller 0xc01b00bd

[David Chinner]

On Mon, Mar 19, 2007 at 11:32:27AM +0100, Marco Berizzi wrote:
> Marco Berizzi wrote:
> > David Chinner wrote:
> >
> >> Ok, so an ipsec change. And I see from the history below it
> >> really has nothing to do with this problem. it seems the problem
> >> has something to do with changes between 2.6.19.1 and 2.6.19.2.
> >
> > indeed. Yesterday at 13:00 I have switched from 2.6.19.1 to 2.6.19.2
> > (without the ipsec fix) and at about 17:30 linux has crashed again.
> > I have recompiled 2.6.19.2 with all kernel debugging options enabled
> > and rebooted. Now I'm waiting for the crash...
> 
> Linux has not been crashed. However here is dmesg output
> with all debugging option enabled: (search for 'INFO:
> possible recursive locking detected'). Is that normal?

.....
> =============================================
> [ INFO: possible recursive locking detected ]
> 2.6.19.2 #1
> ---------------------------------------------
> rm/470 is trying to acquire lock:
>  (&(&ip->i_lock)->mr_lock){----}, at: [<c01cd64a>] xfs_ilock+0x5b/0xa1
> 
> but task is already holding lock:
>  (&(&ip->i_lock)->mr_lock){----}, at: [<c01cd64a>] xfs_ilock+0x5b/0xa1
> 
> other info that might help us debug this:
> 3 locks held by rm/470:
>  #0:  (&inode->i_mutex/1){--..}, at: [<c016e5a7>] do_unlinkat+0x70/0x115
>  #1:  (&inode->i_mutex){--..}, at: [<c030be35>] mutex_lock+0x1c/0x1f
>  #2:  (&(&ip->i_lock)->mr_lock){----}, at: [<c01cd64a>]
> xfs_ilock+0x5b/0xa1
> 
> stack backtrace:
>  [<c0103bc0>] dump_trace+0x215/0x21a
>  [<c0103c68>] show_trace_log_lvl+0x1a/0x30
>  [<c0103c90>] show_trace+0x12/0x14
>  [<c0103d8d>] dump_stack+0x19/0x1b
>  [<c01357e7>] print_deadlock_bug+0xc0/0xcf
>  [<c0135860>] check_deadlock+0x6a/0x79
>  [<c01372e1>] __lock_acquire+0x350/0x970
>  [<c0137fd1>] lock_acquire+0x75/0x97
>  [<c01331ab>] down_write+0x3a/0x54
>  [<c01cd64a>] xfs_ilock+0x5b/0xa1
>  [<c01eda0e>] xfs_lock_dir_and_entry+0x105/0x11b
>  [<c01edcc5>] xfs_remove+0x180/0x47f
>  [<c01f8a9e>] xfs_vn_unlink+0x22/0x4f
>  [<c016e533>] vfs_unlink+0x9e/0xa2
>  [<c016e5df>] do_unlinkat+0xa8/0x115
>  [<c016e68b>] sys_unlink+0x10/0x12
>  [<c0102cdb>] syscall_call+0x7/0xb
>  [<b7efaa7d>] 0xb7efaa7d
>  =======================

That's no problem - lockdep just doesn't know that we can nest i_lock
(we've got to get the annotations for this sorted out).

> Here is the relevant results:
> 
> Phase 2 - found root inode chunk
> Phase 3 - ...
>             agno = 0
>             ...
>             agno = 12
> LEAFN node level is 1 inode 1610612918 bno = 8388608

Hmmm - single bit error in the bno - that reminds of this:

http://oss.sgi.com/projects/xfs/faq.html#dir2

So I'd definitely make sure that is repaired....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS internal error xfs_da_do_buf(2) at line 2087 of file fs/xfs/xfs_da_btree.c. Caller 0xc01b00bd

[Marco Berizzi]

Hi David.
Three months ago I wrote the message below.
I had built various 2.6.20.x and 2.6.21.x
vanilla kernel with all the debug options
enabled and linux had never crashed.
On june 4, I have builded linux 2.6.21.3 without
any debugging options and after 2 days linux
has starting print these errors:

Jun  6 09:47:09 Pleiadi kernel:  =======================
Jun  6 09:47:09 Pleiadi kernel: 0x0: 28 f1 45 d4 22 53 35 11 09 80 37 5a
47 8a 22 ee
Jun  6 09:47:09 Pleiadi kernel: Filesystem "sda8": XFS internal error
xfs_da_do_buf(2) at line 2086 of file fs/xfs/xfs_da_btree.c.  Caller
0xc01b2301
Jun  6 09:47:09 Pleiadi kernel:  [<c01b21f7>] xfs_da_do_buf+0x70c/0x7b1
Jun  6 09:47:09 Pleiadi kernel:  [<c01b2301>] xfs_da_read_buf+0x30/0x35
Jun  6 09:47:09 Pleiadi kernel:  [<c01b2301>] xfs_da_read_buf+0x30/0x35
Jun  6 09:47:09 Pleiadi kernel:  [<c01eb13d>]
_xfs_buf_lookup_pages+0x1e8/0x2ea
Jun  6 09:47:09 Pleiadi kernel:  [<c01eade2>]
_xfs_buf_initialize+0xc8/0xf6
Jun  6 09:47:09 Pleiadi kernel:  [<c01dc307>]
xfs_trans_unreserve_and_mod_sb+0x241/0x264
Jun  6 09:47:09 Pleiadi kernel:  [<c01b2301>] xfs_da_read_buf+0x30/0x35
Jun  6 09:47:09 Pleiadi kernel:  [<c01b7af3>]
xfs_dir2_leaf_lookup_int+0x5c/0x2b6
Jun  6 09:47:09 Pleiadi kernel:  [<c01b7af3>]
xfs_dir2_leaf_lookup_int+0x5c/0x2b6
Jun  6 09:47:09 Pleiadi kernel:  [<c020ab2c>] __next_cpu+0x12/0x1f
Jun  6 09:47:09 Pleiadi kernel:  [<c01b79e0>]
xfs_dir2_leaf_lookup+0x2b/0xe2
Jun  6 09:47:09 Pleiadi kernel:  [<c01b36fb>] xfs_dir2_isleaf+0x1a/0x4f
Jun  6 09:47:09 Pleiadi kernel:  [<c01b2dbc>] xfs_dir_lookup+0xc1/0x11d
Jun  6 09:47:09 Pleiadi kernel:  [<c01de55e>]
xfs_dir_lookup_int+0x34/0x10e
Jun  6 09:47:09 Pleiadi kernel:  [<c01dc4f1>]
_xfs_trans_commit+0x1c7/0x3a2
Jun  6 09:47:09 Pleiadi kernel:  [<c01e3f16>] xfs_lookup+0x5a/0x90
Jun  6 09:47:09 Pleiadi kernel:  [<c01ef939>] xfs_vn_lookup+0x52/0x93
Jun  6 09:47:09 Pleiadi kernel:  [<c01600f4>] real_lookup+0xbb/0x116
Jun  6 09:47:09 Pleiadi kernel:  [<c016036a>] do_lookup+0x90/0xc2
Jun  6 09:47:09 Pleiadi kernel:  [<c01ef8e7>] xfs_vn_lookup+0x0/0x93
Jun  6 09:47:09 Pleiadi kernel:  [<c01604a8>]
__link_path_walk+0x10c/0xcf1
Jun  6 09:47:09 Pleiadi kernel:  [<c015ee4c>] pipe_read+0x23b/0x2bf
Jun  6 09:47:09 Pleiadi kernel:  [<c01610cb>] link_path_walk+0x3e/0xac
Jun  6 09:47:09 Pleiadi kernel:  [<c0159906>] vfs_read+0xee/0x141
Jun  6 09:47:09 Pleiadi kernel:  [<c0159bf4>] sys_read+0x41/0x6a
Jun  6 09:47:09 Pleiadi kernel:  [<c0161407>] do_path_lookup+0x11a/0x1ba
Jun  6 09:47:09 Pleiadi kernel:  [<c0162a00>] do_unlinkat+0x41/0x114
Jun  6 09:47:09 Pleiadi kernel:  [<c0159906>] vfs_read+0xee/0x141
Jun  6 09:47:09 Pleiadi kernel:  [<c0159bf4>] sys_read+0x41/0x6a
Jun  6 09:47:09 Pleiadi kernel:  [<c01028c8>] syscall_call+0x7/0xb
Jun  6 09:47:09 Pleiadi kernel:  =======================

PS: I haven't rebooted the system. It is printing
this message every few seconds on the console:
Filesystem "sda8": XFS internal error xfs_da_do_buf(2) at line 2086 of
file fs/xfs/xfs_da_btree.c.  Caller 0xc01b2301

Here is dmesg output:

Jun  4 20:53:05 Pleiadi kernel: sanitize start
Jun  4 20:53:05 Pleiadi kernel: sanitize end
Jun  4 20:53:05 Pleiadi kernel: copy_e820_map() start: 0000000000000000
size: 000000000009ac00 end: 000000000009ac00 type: 1
Jun  4 20:53:05 Pleiadi kernel: copy_e820_map() type is E820_RAM
Jun  4 20:53:05 Pleiadi kernel: copy_e820_map() start: 000000000009ac00
size: 0000000000005400 end: 00000000000a0000 type: 2
Jun  4 20:53:05 Pleiadi kernel: copy_e820_map() start: 00000000000ce000
size: 0000000000002000 end: 00000000000d0000 type: 2
Jun  4 20:53:05 Pleiadi kernel: copy_e820_map() start: 00000000000e0000
size: 0000000000020000 end: 0000000000100000 type: 2
Jun  4 20:53:05 Pleiadi kernel: copy_e820_map() start: 0000000000100000
size: 000000003fdf0000 end: 000000003fef0000 type: 1
Jun  4 20:53:05 Pleiadi kernel: copy_e820_map() type is E820_RAM
Jun  4 20:53:05 Pleiadi kernel: copy_e820_map() start: 000000003fef0000
size: 000000000000b000 end: 000000003fefb000 type: 3
Jun  4 20:53:05 Pleiadi kernel: copy_e820_map() start: 000000003fefb000
size: 0000000000005000 end: 000000003ff00000 type: 4
Jun  4 20:53:05 Pleiadi kernel: copy_e820_map() start: 000000003ff00000
size: 0000000000080000 end: 000000003ff80000 type: 1
Jun  4 20:53:05 Pleiadi kernel: copy_e820_map() type is E820_RAM
Jun  4 20:53:05 Pleiadi kernel: copy_e820_map() start: 000000003ff80000
size: 0000000000080000 end: 0000000040000000 type: 2
Jun  4 20:53:05 Pleiadi kernel: copy_e820_map() start: 00000000e0000000
size: 0000000010000000 end: 00000000f0000000 type: 2
Jun  4 20:53:05 Pleiadi kernel: copy_e820_map() start: 00000000fec00000
size: 0000000000100400 end: 00000000fed00400 type: 2
Jun  4 20:53:05 Pleiadi kernel: copy_e820_map() start: 00000000fee00000
size: 0000000000100000 end: 00000000fef00000 type: 2
Jun  4 20:53:05 Pleiadi kernel: copy_e820_map() start: 00000000ffb00000
size: 0000000000100000 end: 00000000ffc00000 type: 2
Jun  4 20:53:05 Pleiadi kernel: copy_e820_map() start: 00000000fff00000
size: 0000000000100000 end: 0000000100000000 type: 2
Jun  4 20:53:05 Pleiadi kernel:  BIOS-e820: 0000000000000000 -
000000000009ac00 (usable)
Jun  4 20:53:05 Pleiadi kernel:  BIOS-e820: 000000000009ac00 -
00000000000a0000 (reserved)
Jun  4 20:53:05 Pleiadi kernel:  BIOS-e820: 00000000000ce000 -
00000000000d0000 (reserved)
Jun  4 20:53:05 Pleiadi kernel:  BIOS-e820: 00000000000e0000 -
0000000000100000 (reserved)
Jun  4 20:53:05 Pleiadi kernel:  BIOS-e820: 0000000000100000 -
000000003fef0000 (usable)
Jun  4 20:53:05 Pleiadi kernel:  BIOS-e820: 000000003fef0000 -
000000003fefb000 (ACPI data)
Jun  4 20:53:05 Pleiadi kernel:  BIOS-e820: 000000003fefb000 -
000000003ff00000 (ACPI NVS)
Jun  4 20:53:05 Pleiadi kernel:  BIOS-e820: 000000003ff00000 -
000000003ff80000 (usable)
Jun  4 20:53:05 Pleiadi kernel:  BIOS-e820: 000000003ff80000 -
0000000040000000 (reserved)
Jun  4 20:53:05 Pleiadi kernel:  BIOS-e820: 00000000e0000000 -
00000000f0000000 (reserved)
Jun  4 20:53:05 Pleiadi kernel:  BIOS-e820: 00000000fec00000 -
00000000fed00400 (reserved)
Jun  4 20:53:05 Pleiadi kernel:  BIOS-e820: 00000000fee00000 -
00000000fef00000 (reserved)
Jun  4 20:53:05 Pleiadi kernel:  BIOS-e820: 00000000ffb00000 -
00000000ffc00000 (reserved)
Jun  4 20:53:05 Pleiadi kernel:  BIOS-e820: 00000000fff00000 -
0000000100000000 (reserved)
Jun  4 20:53:05 Pleiadi kernel: Zone PFN ranges:
Jun  4 20:53:05 Pleiadi kernel:   DMA             0 ->     4096
Jun  4 20:53:05 Pleiadi kernel:   Normal       4096 ->   229376
Jun  4 20:53:05 Pleiadi kernel:   HighMem    229376 ->   262016
Jun  4 20:53:05 Pleiadi kernel: early_node_map[1] active PFN ranges
Jun  4 20:53:05 Pleiadi kernel:     0:        0 ->   262016
Jun  4 20:53:05 Pleiadi kernel: ACPI: RSDP 000F6BA0, 0024 (r2 PTLTD )
Jun  4 20:53:05 Pleiadi kernel: ACPI: XSDT 3FEF5381, 004C (r1 PTLTD  ^I
XSDT    6040001  LTP        0)
Jun  4 20:53:05 Pleiadi kernel: ACPI: FACP 3FEF5441, 00F4 (r3 FSC
6040001         F4240)
Jun  4 20:53:05 Pleiadi kernel: ACPI: DSDT 3FEF5535, 597B (r1 FSC
D1649     6040001 MSFT  2000002)
Jun  4 20:53:05 Pleiadi kernel: ACPI: FACS 3FEFBFC0, 0040
Jun  4 20:53:05 Pleiadi kernel: ACPI: SPCR 3FEFAEB0, 0050 (r1 PTLTD
$UCRTBL$  6040001 PTL         1)
Jun  4 20:53:05 Pleiadi kernel: ACPI: MCFG 3FEFAF00, 0040 (r1 PTLTD
MCFG    6040001  LTP        0)
Jun  4 20:53:05 Pleiadi kernel: ACPI: APIC 3FEFAF40, 0098 (r1 PTLTD  ^I
APIC    6040001  LTP        0)
Jun  4 20:53:05 Pleiadi kernel: ACPI: BOOT 3FEFAFD8, 0028 (r1 PTLTD
$SBFTBL$  6040001  LTP        1)
Jun  4 20:53:05 Pleiadi kernel: Processor #0 15:4 APIC version 20
Jun  4 20:53:05 Pleiadi kernel: Processor #1 15:4 APIC version 20
Jun  4 20:53:05 Pleiadi kernel: IOAPIC[0]: apic_id 2, version 32,
address 0xfec00000, GSI 0-23
Jun  4 20:53:05 Pleiadi kernel: IOAPIC[1]: apic_id 3, version 32,
address 0xfec80000, GSI 24-47
Jun  4 20:53:05 Pleiadi kernel: IOAPIC[2]: apic_id 4, version 32,
address 0xfec80800, GSI 48-71
Jun  4 20:53:05 Pleiadi kernel: IOAPIC[3]: apic_id 5, version 32,
address 0xfec84000, GSI 72-95
Jun  4 20:53:05 Pleiadi kernel: IOAPIC[4]: apic_id 6, version 32,
address 0xfec84800, GSI 96-119
Jun  4 20:53:05 Pleiadi kernel: Enabling APIC mode:  Flat.  Using 5 I/O
APICs
Jun  4 20:53:05 Pleiadi kernel: Allocating PCI resources starting at
50000000 (gap: 40000000:a0000000)
Jun  4 20:53:05 Pleiadi kernel: Built 1 zonelists.  Total pages: 259969
Jun  4 20:53:05 Pleiadi kernel: PID hash table entries: 4096 (order: 12,
16384 bytes)
Jun  4 20:53:05 Pleiadi kernel: Detected 3200.428 MHz processor.
Jun  4 20:53:05 Pleiadi kernel: Console: colour VGA+ 80x25
Jun  4 20:53:05 Pleiadi kernel: Dentry cache hash table entries: 131072
(order: 7, 524288 bytes)
Jun  4 20:53:05 Pleiadi kernel: Inode-cache hash table entries: 65536
(order: 6, 262144 bytes)
Jun  4 20:53:05 Pleiadi kernel: virtual kernel memory layout:
Jun  4 20:53:05 Pleiadi kernel:     fixmap  : 0xfff9d000 - 0xfffff000
( 392 kB)
Jun  4 20:53:05 Pleiadi kernel:     pkmap   : 0xff800000 - 0xffc00000
(4096 kB)
Jun  4 20:53:05 Pleiadi kernel:     vmalloc : 0xf8800000 - 0xff7fe000
( 111 MB)
Jun  4 20:53:05 Pleiadi kernel:     lowmem  : 0xc0000000 - 0xf8000000
( 896 MB)
Jun  4 20:53:05 Pleiadi kernel:       .init : 0xc039f000 - 0xc03ce000
( 188 kB)
Jun  4 20:53:05 Pleiadi kernel:       .data : 0xc02fd400 - 0xc0398114
( 619 kB)
Jun  4 20:53:05 Pleiadi kernel:       .text : 0xc0100000 - 0xc02fd400
(2037 kB)
Jun  4 20:53:05 Pleiadi kernel: Checking if this processor honours the
WP bit even in supervisor mode... Ok.
Jun  4 20:53:05 Pleiadi kernel: Calibrating delay using timer specific
routine.. 6403.78 BogoMIPS (lpj=32018905)
Jun  4 20:53:05 Pleiadi kernel: Mount-cache hash table entries: 512
Jun  4 20:53:05 Pleiadi kernel: monitor/mwait feature present.
Jun  4 20:53:05 Pleiadi kernel: using mwait in idle threads.
Jun  4 20:53:05 Pleiadi kernel: CPU0: Intel(R) Xeon(TM) CPU 3.20GHz
stepping 0a
Jun  4 20:53:05 Pleiadi kernel: Booting processor 1/1 eip 2000
Jun  4 20:53:05 Pleiadi kernel: Calibrating delay using timer specific
routine.. 6400.45 BogoMIPS (lpj=32002267)
Jun  4 20:53:05 Pleiadi kernel: monitor/mwait feature present.
Jun  4 20:53:05 Pleiadi kernel: CPU1: Intel(R) Xeon(TM) CPU 3.20GHz
stepping 0a
Jun  4 20:53:05 Pleiadi kernel: ENABLING IO-APIC IRQs
Jun  4 20:53:05 Pleiadi kernel: migration_cost=142
Jun  4 20:53:05 Pleiadi kernel: Setting up standard PCI resources
Jun  4 20:53:05 Pleiadi kernel: ACPI: Device [PS2M] status [00000008]:
functional but not present; setting present
Jun  4 20:53:05 Pleiadi kernel: ACPI: Device [ECP] status [00000008]:
functional but not present; setting present
Jun  4 20:53:05 Pleiadi kernel: ACPI: Device [COM1] status [00000008]:
functional but not present; setting present
Jun  4 20:53:05 Pleiadi kernel: PCI quirk: region f000-f07f claimed by
ICH4 ACPI/GPIO/TCO
Jun  4 20:53:05 Pleiadi kernel: PCI quirk: region f180-f1bf claimed by
ICH4 GPIO
Jun  4 20:53:05 Pleiadi kernel: PCI: PXH quirk detected, disabling MSI
for SHPC device
Jun  4 20:53:05 Pleiadi kernel: PCI: PXH quirk detected, disabling MSI
for SHPC device
Jun  4 20:53:05 Pleiadi kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3
4 5 6 7 9 10 *11 12 14 15)
Jun  4 20:53:05 Pleiadi kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3
4 5 6 7 *9 10 11 12 14 15)
Jun  4 20:53:05 Pleiadi kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3
4 *5 6 7 9 10 11 12 14 15)
Jun  4 20:53:05 Pleiadi kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3
4 5 6 7 9 *10 11 12 14 15)
Jun  4 20:53:05 Pleiadi kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3
4 5 6 7 9 10 11 12 14 15) *0, disabled.
Jun  4 20:53:05 Pleiadi kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3
4 5 6 7 9 10 11 12 14 15) *0, disabled.
Jun  4 20:53:05 Pleiadi kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3
4 5 6 7 9 10 11 12 14 15) *0, disabled.
Jun  4 20:53:05 Pleiadi kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3
*4 5 6 7 9 10 11 12 14 15)
Jun  4 20:53:05 Pleiadi kernel: IP route cache hash table entries: 32768
(order: 5, 131072 bytes)
Jun  4 20:53:05 Pleiadi kernel: TCP established hash table entries:
131072 (order: 8, 1572864 bytes)
Jun  4 20:53:05 Pleiadi kernel: TCP bind hash table entries: 65536
(order: 7, 524288 bytes)
Jun  4 20:53:05 Pleiadi kernel: highmem bounce pool size: 64 pages
Jun  4 20:53:05 Pleiadi kernel: PNP: PS/2 controller doesn't have AUX
irq; using default 12
Jun  4 20:53:05 Pleiadi kernel: nf_conntrack version 0.5.0 (8188
buckets, 65504 max)
Jun  4 20:53:05 Pleiadi kernel: ip_tables: (C) 2000-2006 Netfilter Core
Team
Jun  4 20:53:05 Pleiadi kernel: Using IPI Shortcut mode
Jun  4 20:53:05 Pleiadi kernel: VFS: Mounted root (xfs filesystem)
readonly.
Jun  6 09:47:09 Pleiadi kernel: 0x0: 28 f1 45 d4 22 53 35 11 09 80 37 5a
47 8a 22 ee
Jun  6 09:47:09 Pleiadi kernel: Filesystem "sda8": XFS internal error
xfs_da_do_buf(2) at line 2086 of file fs/xfs/xfs_da_btree.c.  Caller
0xc01b2301
Jun  6 09:47:09 Pleiadi kernel:  [<c01b21f7>] xfs_da_do_buf+0x70c/0x7b1
Jun  6 09:47:09 Pleiadi kernel:  [<c01b2301>] xfs_da_read_buf+0x30/0x35
Jun  6 09:47:09 Pleiadi last message repeated 2 times
Jun  6 09:47:09 Pleiadi kernel:  [<c01b7af3>]
xfs_dir2_leaf_lookup_int+0x5c/0x2b6
Jun  6 09:47:09 Pleiadi kernel:  [<c01b7af3>]
xfs_dir2_leaf_lookup_int+0x5c/0x2b6
Jun  6 09:47:09 Pleiadi kernel:  [<c01b79e0>]
xfs_dir2_leaf_lookup+0x2b/0xe2
Jun  6 09:47:09 Pleiadi kernel:  [<c01b36fb>] xfs_dir2_isleaf+0x1a/0x4f
Jun  6 09:47:09 Pleiadi kernel:  [<c01b2dbc>] xfs_dir_lookup+0xc1/0x11d
Jun  6 09:47:09 Pleiadi kernel:  [<c0178a8b>]
__block_commit_write+0x7d/0xb0
Jun  6 09:47:09 Pleiadi kernel:  [<c013f337>]
generic_file_buffered_write+0x2d1/0x682
Jun  6 09:47:09 Pleiadi kernel:  [<c01de55e>]
xfs_dir_lookup_int+0x34/0x10e
Jun  6 09:47:09 Pleiadi kernel:  [<c01e3f16>] xfs_lookup+0x5a/0x90
Jun  6 09:47:09 Pleiadi kernel:  [<c01ef939>] xfs_vn_lookup+0x52/0x93
Jun  6 09:47:09 Pleiadi kernel:  [<c01600f4>] real_lookup+0xbb/0x116
Jun  6 09:47:09 Pleiadi kernel:  [<c016036a>] do_lookup+0x90/0xc2
Jun  6 09:47:09 Pleiadi kernel:  [<c01ef8e7>] xfs_vn_lookup+0x0/0x93
Jun  6 09:47:09 Pleiadi kernel:  [<c01604a8>]
__link_path_walk+0x10c/0xcf1
Jun  6 09:47:09 Pleiadi kernel:  [<c01610cb>] link_path_walk+0x3e/0xac
Jun  6 09:47:09 Pleiadi kernel:  [<c0158e0b>] get_unused_fd+0x2e/0xb6
Jun  6 09:47:09 Pleiadi kernel:  [<c0161407>] do_path_lookup+0x11a/0x1ba
Jun  6 09:47:09 Pleiadi kernel:  [<c016150f>]
__path_lookup_intent_open+0x50/0x90
Jun  6 09:47:09 Pleiadi kernel:  [<c016156f>] path_lookup_open+0x20/0x25
Jun  6 09:47:09 Pleiadi kernel:  [<c0161d4b>] open_namei+0x7a/0x550
Jun  6 09:47:09 Pleiadi kernel:  [<c014a03f>] do_wp_page+0x20e/0x3ec
Jun  6 09:47:09 Pleiadi kernel:  [<c0158c86>] do_filp_open+0x2e/0x5b
Jun  6 09:47:09 Pleiadi kernel:  [<c0158e0b>] get_unused_fd+0x2e/0xb6
Jun  6 09:47:09 Pleiadi kernel:  [<c0158f79>] do_sys_open+0x4e/0xdb
Jun  6 09:47:09 Pleiadi kernel:  [<c0159022>] sys_open+0x1c/0x20
Jun  6 09:47:09 Pleiadi kernel:  [<c01028c8>] syscall_call+0x7/0xb
Jun  6 09:47:09 Pleiadi kernel:  =======================
Jun  6 09:47:09 Pleiadi kernel: 0x0: 28 f1 45 d4 22 53 35 11 09 80 37 5a
47 8a 22 ee
Jun  6 09:47:09 Pleiadi kernel: Filesystem "sda8": XFS internal error
xfs_da_do_buf(2) at line 2086 of file fs/xfs/xfs_da_btree.c.  Caller
0xc01b2301
Jun  6 09:47:09 Pleiadi kernel:  [<c01b21f7>] xfs_da_do_buf+0x70c/0x7b1
Jun  6 09:47:09 Pleiadi kernel:  [<c01b2301>] xfs_da_read_buf+0x30/0x35
Jun  6 09:47:09 Pleiadi kernel:  [<c01b2301>] xfs_da_read_buf+0x30/0x35
Jun  6 09:47:09 Pleiadi kernel:  [<c01eb13d>]
_xfs_buf_lookup_pages+0x1e8/0x2ea
Jun  6 09:47:09 Pleiadi kernel:  [<c01eade2>]
_xfs_buf_initialize+0xc8/0xf6
Jun  6 09:47:09 Pleiadi kernel:  [<c01dc307>]
xfs_trans_unreserve_and_mod_sb+0x241/0x264
Jun  6 09:47:09 Pleiadi kernel:  [<c01b2301>] xfs_da_read_buf+0x30/0x35
Jun  6 09:47:09 Pleiadi kernel:  [<c01b7af3>]
xfs_dir2_leaf_lookup_int+0x5c/0x2b6
Jun  6 09:47:09 Pleiadi kernel:  [<c01b7af3>]
xfs_dir2_leaf_lookup_int+0x5c/0x2b6
Jun  6 09:47:09 Pleiadi kernel:  [<c020ab2c>] __next_cpu+0x12/0x1f
Jun  6 09:47:09 Pleiadi kernel:  [<c01b79e0>]
xfs_dir2_leaf_lookup+0x2b/0xe2
Jun  6 09:47:09 Pleiadi kernel:  [<c01b36fb>] xfs_dir2_isleaf+0x1a/0x4f
Jun  6 09:47:09 Pleiadi kernel:  [<c01b2dbc>] xfs_dir_lookup+0xc1/0x11d
Jun  6 09:47:09 Pleiadi kernel:  [<c01de55e>]
xfs_dir_lookup_int+0x34/0x10e
Jun  6 09:47:09 Pleiadi kernel:  [<c01dc4f1>]
_xfs_trans_commit+0x1c7/0x3a2
Jun  6 09:47:09 Pleiadi kernel:  [<c01e3f16>] xfs_lookup+0x5a/0x90
Jun  6 09:47:09 Pleiadi kernel:  [<c01ef939>] xfs_vn_lookup+0x52/0x93
Jun  6 09:47:09 Pleiadi kernel:  [<c01600f4>] real_lookup+0xbb/0x116
Jun  6 09:47:09 Pleiadi kernel:  [<c016036a>] do_lookup+0x90/0xc2
Jun  6 09:47:09 Pleiadi kernel:  [<c01ef8e7>] xfs_vn_lookup+0x0/0x93
Jun  6 09:47:09 Pleiadi kernel:  [<c01604a8>]
__link_path_walk+0x10c/0xcf1
Jun  6 09:47:09 Pleiadi kernel:  [<c015ee4c>] pipe_read+0x23b/0x2bf
Jun  6 09:47:09 Pleiadi kernel:  [<c01610cb>] link_path_walk+0x3e/0xac
Jun  6 09:47:09 Pleiadi kernel:  [<c0159906>] vfs_read+0xee/0x141
Jun  6 09:47:09 Pleiadi kernel:  [<c0159bf4>] sys_read+0x41/0x6a
Jun  6 09:47:09 Pleiadi kernel:  [<c0161407>] do_path_lookup+0x11a/0x1ba
Jun  6 09:47:09 Pleiadi kernel:  [<c0162a00>] do_unlinkat+0x41/0x114
Jun  6 09:47:09 Pleiadi kernel:  [<c0159906>] vfs_read+0xee/0x141
Jun  6 09:47:09 Pleiadi kernel:  [<c0159bf4>] sys_read+0x41/0x6a
Jun  6 09:47:09 Pleiadi kernel:  [<c01028c8>] syscall_call+0x7/0xb
Jun  6 09:47:09 Pleiadi kernel:  =======================
..

David Chinner wrote:

> On Mon, Mar 19, 2007 at 11:32:27AM +0100, Marco Berizzi wrote:
> > Marco Berizzi wrote:
> > > David Chinner wrote:
> > >
> > >> Ok, so an ipsec change. And I see from the history below it
> > >> really has nothing to do with this problem. it seems the problem
> > >> has something to do with changes between 2.6.19.1 and 2.6.19.2.
> > >
> > > indeed. Yesterday at 13:00 I have switched from 2.6.19.1 to
2.6.19.2
> > > (without the ipsec fix) and at about 17:30 linux has crashed
again.
> > > I have recompiled 2.6.19.2 with all kernel debugging options
enabled
> > > and rebooted. Now I'm waiting for the crash...
> >
> > Linux has not been crashed. However here is dmesg output
> > with all debugging option enabled: (search for 'INFO:
> > possible recursive locking detected'). Is that normal?
>
> .....
> > =============================================
> > [ INFO: possible recursive locking detected ]
> > 2.6.19.2 #1
> > ---------------------------------------------
> > rm/470 is trying to acquire lock:
> >  (&(&ip->i_lock)->mr_lock){----}, at: [<c01cd64a>]
xfs_ilock+0x5b/0xa1
> >
> > but task is already holding lock:
> >  (&(&ip->i_lock)->mr_lock){----}, at: [<c01cd64a>]
xfs_ilock+0x5b/0xa1
> >
> > other info that might help us debug this:
> > 3 locks held by rm/470:
> >  #0:  (&inode->i_mutex/1){--..}, at: [<c016e5a7>]
do_unlinkat+0x70/0x115
> >  #1:  (&inode->i_mutex){--..}, at: [<c030be35>] mutex_lock+0x1c/0x1f
> >  #2:  (&(&ip->i_lock)->mr_lock){----}, at: [<c01cd64a>]
> > xfs_ilock+0x5b/0xa1
> >
> > stack backtrace:
> >  [<c0103bc0>] dump_trace+0x215/0x21a
> >  [<c0103c68>] show_trace_log_lvl+0x1a/0x30
> >  [<c0103c90>] show_trace+0x12/0x14
> >  [<c0103d8d>] dump_stack+0x19/0x1b
> >  [<c01357e7>] print_deadlock_bug+0xc0/0xcf
> >  [<c0135860>] check_deadlock+0x6a/0x79
> >  [<c01372e1>] __lock_acquire+0x350/0x970
> >  [<c0137fd1>] lock_acquire+0x75/0x97
> >  [<c01331ab>] down_write+0x3a/0x54
> >  [<c01cd64a>] xfs_ilock+0x5b/0xa1
> >  [<c01eda0e>] xfs_lock_dir_and_entry+0x105/0x11b
> >  [<c01edcc5>] xfs_remove+0x180/0x47f
> >  [<c01f8a9e>] xfs_vn_unlink+0x22/0x4f
> >  [<c016e533>] vfs_unlink+0x9e/0xa2
> >  [<c016e5df>] do_unlinkat+0xa8/0x115
> >  [<c016e68b>] sys_unlink+0x10/0x12
> >  [<c0102cdb>] syscall_call+0x7/0xb
> >  [<b7efaa7d>] 0xb7efaa7d
> >  =======================
>
> That's no problem - lockdep just doesn't know that we can nest i_lock
> (we've got to get the annotations for this sorted out).
>
> > Here is the relevant results:
> >
> > Phase 2 - found root inode chunk
> > Phase 3 - ...
> >             agno = 0
> >             ...
> >             agno = 12
> > LEAFN node level is 1 inode 1610612918 bno = 8388608
>
> Hmmm - single bit error in the bno - that reminds of this:
>
> http://oss.sgi.com/projects/xfs/faq.html#dir2
>
> So I'd definitely make sure that is repaired....
>
> Cheers,
>
> Dave.
> -- 
> Dave Chinner
> Principal Engineer
> SGI Australian Software Group
>

Re: XFS internal error xfs_da_do_buf(2) at line 2087 of file fs/xfs/xfs_da_btree.c. Caller 0xc01b00bd

[David Chinner]

On Thu, Jun 07, 2007 at 09:44:51AM +0200, Marco Berizzi wrote:
> Hi David.
> Three months ago I wrote the message below.
> I had built various 2.6.20.x and 2.6.21.x
> vanilla kernel with all the debug options
> enabled and linux had never crashed.
> On june 4, I have builded linux 2.6.21.3 without
> any debugging options and after 2 days linux
> has starting print these errors:
> 
> Jun  6 09:47:09 Pleiadi kernel:  =======================
> Jun  6 09:47:09 Pleiadi kernel: 0x0: 28 f1 45 d4 22 53 35 11 09 80 37 5a
> 47 8a 22 ee
> Jun  6 09:47:09 Pleiadi kernel: Filesystem "sda8": XFS internal error
> xfs_da_do_buf(2) at line 2086 of file fs/xfs/xfs_da_btree.c.  Caller
> 0xc01b2301
> Jun  6 09:47:09 Pleiadi kernel:  [<c01b21f7>] xfs_da_do_buf+0x70c/0x7b1
> Jun  6 09:47:09 Pleiadi kernel:  [<c01b2301>] xfs_da_read_buf+0x30/0x35
> Jun  6 09:47:09 Pleiadi kernel:  [<c01b2301>] xfs_da_read_buf+0x30/0x35

These above stack trace is the sign of a corrupted directory.

Chopping out the rest of the top posting (please don't do that)
we get down to 3 months ago:

> > On Mon, Mar 19, 2007 at 11:32:27AM +0100, Marco Berizzi wrote:
> > > Marco Berizzi wrote:
> > > Here is the relevant results:
> > >
> > > Phase 2 - found root inode chunk
> > > Phase 3 - ...
> > >             agno = 0
> > >             ...
> > >             agno = 12
> > > LEAFN node level is 1 inode 1610612918 bno = 8388608
> >
> > Hmmm - single bit error in the bno - that reminds of this:
> >
> > http://oss.sgi.com/projects/xfs/faq.html#dir2
> >
> > So I'd definitely make sure that is repaired....

Where we saw signs of on disk directory corruption. Have you run
xfs_repair successfully on the filesystem since you reported
this?

If you did clean up the error, does xfs_repair report the same sort
of error again?

Have you run a 2.6.16-rcX or 2.6.17.[0-6] kernel since you last
reported this problem?

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS internal error xfs_da_do_buf(2) at line 2087 of file fs/xfs/xfs_da_btree.c. Caller 0xc01b00bd

[Marco Berizzi]

David Chinner wrote:

> > Jun  6 09:47:09 Pleiadi kernel:  =======================
> > Jun  6 09:47:09 Pleiadi kernel: 0x0: 28 f1 45 d4 22 53 35 11 09 80
37 5a
> > 47 8a 22 ee
> > Jun  6 09:47:09 Pleiadi kernel: Filesystem "sda8": XFS internal
error
> > xfs_da_do_buf(2) at line 2086 of file fs/xfs/xfs_da_btree.c.  Caller
> > 0xc01b2301
> > Jun  6 09:47:09 Pleiadi kernel:  [<c01b21f7>]
xfs_da_do_buf+0x70c/0x7b1
> > Jun  6 09:47:09 Pleiadi kernel:  [<c01b2301>]
xfs_da_read_buf+0x30/0x35
> > Jun  6 09:47:09 Pleiadi kernel:  [<c01b2301>]
xfs_da_read_buf+0x30/0x35
>
> These above stack trace is the sign of a corrupted directory.
>
> Chopping out the rest of the top posting (please don't do that)

apologies

> we get down to 3 months ago:
>
> > > On Mon, Mar 19, 2007 at 11:32:27AM +0100, Marco Berizzi wrote:
> > > > Marco Berizzi wrote:
> > > > Here is the relevant results:
> > > >
> > > > Phase 2 - found root inode chunk
> > > > Phase 3 - ...
> > > >             agno = 0
> > > >             ...
> > > >             agno = 12
> > > > LEAFN node level is 1 inode 1610612918 bno = 8388608
> > >
> > > Hmmm - single bit error in the bno - that reminds of this:
> > >
> > > http://oss.sgi.com/projects/xfs/faq.html#dir2
> > >
> > > So I'd definitely make sure that is repaired....
>
> Where we saw signs of on disk directory corruption. Have you run
> xfs_repair successfully on the filesystem since you reported
> this?

yes.

> If you did clean up the error, does xfs_repair report the same sort
> of error again?

I have run xfs_repair this morning.
Here is the report:

Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - clear lost+found (if it exists) ...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - ensuring existence of lost+found directory
        - traversing filesystem starting at / ...
        - traversal finished ...
        - traversing all unattached subtrees ...
        - traversals finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done

> Have you run a 2.6.16-rcX or 2.6.17.[0-6] kernel since you last
> reported this problem?

No. I have run only 2.6.19.x and 2.6.21.x

After the xfs_repair I have remounted the file system.
After few hours linux has crashed with this message:
BUG: at arch/i386/kernel/smp.c:546 smp_call_function()
I have also the monitor bitmap.

Re: XFS internal error xfs_da_do_buf(2) at line 2087 of file fs/xfs/xfs_da_btree.c. Caller 0xc01b00bd

[David Chinner]

On Fri, Jun 08, 2007 at 03:59:39PM +0200, Marco Berizzi wrote:
> David Chinner wrote:
> > Where we saw signs of on disk directory corruption. Have you run
> > xfs_repair successfully on the filesystem since you reported
> > this?
> 
> yes.
> 
> > If you did clean up the error, does xfs_repair report the same sort
> > of error again?
> 
> I have run xfs_repair this morning.
> Here is the report:

<reports no on disk errors>

> > Have you run a 2.6.16-rcX or 2.6.17.[0-6] kernel since you last
> > reported this problem?
> 
> No. I have run only 2.6.19.x and 2.6.21.x
> 
> After the xfs_repair I have remounted the file system.
> After few hours linux has crashed with this message:
> BUG: at arch/i386/kernel/smp.c:546 smp_call_function()
> I have also the monitor bitmap.

This is sounding like memory corruption is no corruption is being
found on disk by xfs_repair.  Have you run memtest86 on that box to
see if it's got bad memory?

Cheers

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS internal error xfs_da_do_buf(2) at line 2087 of file fs/xfs/xfs_da_btree.c. Caller 0xc01b00bd

[Marco Berizzi]

David Chinner wrote:

> On Fri, Jun 08, 2007 at 03:59:39PM +0200, Marco Berizzi wrote:
> > David Chinner wrote:
> > > Where we saw signs of on disk directory corruption. Have you run
> > > xfs_repair successfully on the filesystem since you reported
> > > this?
> >
> > yes.
> >
> > > If you did clean up the error, does xfs_repair report the same
sort
> > > of error again?
> >
> > I have run xfs_repair this morning.
> > Here is the report:
>
> <reports no on disk errors>
>
> > > Have you run a 2.6.16-rcX or 2.6.17.[0-6] kernel since you last
> > > reported this problem?
> >
> > No. I have run only 2.6.19.x and 2.6.21.x
> >
> > After the xfs_repair I have remounted the file system.
> > After few hours linux has crashed with this message:
> > BUG: at arch/i386/kernel/smp.c:546 smp_call_function()
> > I have also the monitor bitmap.
>
> This is sounding like memory corruption is no corruption is being
> found on disk by xfs_repair.  Have you run memtest86 on that box to
> see if it's got bad memory?

Yes. I have run memtest for one week:
no errors.
I have also changed the mother board,
scsi controller and ram. Only the cpu
and the 2 hot swap scsi disks were
not replaced. IMHO this isn't an
hardware problem, because the kernel
with debugging options enabled didn't
crash for a long time (>1 month). Just
for record, at this moment this box is
running 2.6.22-rc4 with no debug
options enabled. I will keep you
informed.
Thanks everybody for the support.

Re: XFS internal error xfs_da_do_buf(2) at line 2087 of file fs/xfs/xfs_da_btree.c. Caller 0xc01b00bd

[Marco Berizzi]

Marco Berizzi wrote:

> David Chinner wrote:
>
> > On Fri, Jun 08, 2007 at 03:59:39PM +0200, Marco Berizzi wrote:
> > > David Chinner wrote:
> > > > Where we saw signs of on disk directory corruption. Have you run
> > > > xfs_repair successfully on the filesystem since you reported
> > > > this?
> > >
> > > yes.
> > >
> > > > If you did clean up the error, does xfs_repair report the same
> sort
> > > > of error again?
> > >
> > > I have run xfs_repair this morning.
> > > Here is the report:
> >
> > <reports no on disk errors>
> >
> > > > Have you run a 2.6.16-rcX or 2.6.17.[0-6] kernel since you last
> > > > reported this problem?
> > >
> > > No. I have run only 2.6.19.x and 2.6.21.x
> > >
> > > After the xfs_repair I have remounted the file system.
> > > After few hours linux has crashed with this message:
> > > BUG: at arch/i386/kernel/smp.c:546 smp_call_function()
> > > I have also the monitor bitmap.
> >
> > This is sounding like memory corruption is no corruption is being
> > found on disk by xfs_repair.  Have you run memtest86 on that box to
> > see if it's got bad memory?
>
> Yes. I have run memtest for one week:
> no errors.
> I have also changed the mother board,
> scsi controller and ram. Only the cpu
> and the 2 hot swap scsi disks were
> not replaced. IMHO this isn't an
> hardware problem, because the kernel
> with debugging options enabled didn't
> crash for a long time (>1 month). Just
> for record, at this moment this box is
> running 2.6.22-rc4 with no debug
> options enabled. I will keep you
> informed.
> Thanks everybody for the support.

Hi David,
on another system which is doing the
same task (openswan + squid), this
morning I have found the following
errors (2.6.21.5 after 4 days uptime).
The tricky thing is that always the
squid file cache filesystem is
corrupted. The same box with 2.6.20.x
and 2.6.21.x with 'Debug slab memory
allocations' enabled, never show any
errors for 1 month.

# dmesg
Linux version 2.6.21.5 (root@Gemini) (gcc version 3.3.6) #1 Thu Jun 14
13:18:08 CEST 2007
BIOS-provided physical RAM map:
sanitize start
sanitize end
copy_e820_map() start: 0000000000000000 size: 000000000009f800 end:
000000000009f800 type: 1
copy_e820_map() type is E820_RAM
copy_e820_map() start: 000000000009f800 size: 0000000000000800 end:
00000000000a0000 type: 2
copy_e820_map() start: 00000000000f0000 size: 0000000000010000 end:
0000000000100000 type: 2
copy_e820_map() start: 0000000000100000 size: 0000000009f00000 end:
000000000a000000 type: 1
copy_e820_map() type is E820_RAM
copy_e820_map() start: 00000000ffff0000 size: 0000000000010000 end:
0000000100000000 type: 2
 BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
 BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000000a000000 (usable)
 BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved)
160MB LOWMEM available.
Entering add_active_range(0, 0, 40960) 0 entries of 256 used
Zone PFN ranges:
  DMA             0 -    4096
  Normal       4096 -   40960
early_node_map[1] active PFN ranges
    0:        0 -   40960
On node 0 totalpages: 40960
  DMA zone: 32 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 4064 pages, LIFO batch:0
  Normal zone: 288 pages used for memmap
  Normal zone: 36576 pages, LIFO batch:7
DMI 2.1 present.
Allocating PCI resources starting at 10000000 (gap: 0a000000:f5ff0000)
Built 1 zonelists.  Total pages: 40640
Kernel command line: auto BOOT_IMAGE=Linux ro root=301
Local APIC disabled by BIOS -- you can enable it with "lapic"
mapped APIC to ffffd000 (01141000)
Enabling fast FPU save and restore... done.
Initializing CPU#0
PID hash table entries: 1024 (order: 10, 4096 bytes)
Detected 267.302 MHz processor.
Console: colour VGA+ 80x25
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 159020k/163840k available (1945k kernel code, 4392k reserved,
609k data, 156k init, 0k highmem)
virtual kernel memory layout:
    fixmap  : 0xfffb7000 - 0xfffff000   ( 288 kB)
    vmalloc : 0xca800000 - 0xfffb5000   ( 855 MB)
    lowmem  : 0xc0000000 - 0xca000000   ( 160 MB)
      .init : 0xc0382000 - 0xc03a9000   ( 156 kB)
      .data : 0xc02e667c - 0xc037eb94   ( 609 kB)
      .text : 0xc0100000 - 0xc02e667c   (1945 kB)
Checking if this processor honours the WP bit even in supervisor mode...
Ok.
Calibrating delay using timer specific routine.. 535.23 BogoMIPS
(lpj=1070464)
Mount-cache hash table entries: 512
CPU: After generic identify, caps: 0183f9ff 00000000 00000000 00000000
00000000 00000000 00000000
CPU: L1 I cache: 16K, L1 D cache: 16K
CPU: After all inits, caps: 0183f9ff 00000000 00000000 00000040 00000000
00000000 00000000
CPU: Intel Celeron (Covington) stepping 00
Checking 'hlt' instruction... OK.
ACPI: Core revision 20070126
ACPI Exception (tbxface-0618): AE_NO_ACPI_TABLES, While loading
namespace from ACPI tables [20070126]
ACPI: Unable to load the System Description Tables
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xfda61, last bus=1
PCI: Using configuration type 1
Setting up standard PCI resources
ACPI: Interpreter disabled.
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI: disabled
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
* Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
* this clock source is slow. Consider trying other clock sources
PCI quirk: region 6100-613f claimed by PIIX4 ACPI
PCI quirk: region 5f00-5f0f claimed by PIIX4 SMB
Boot video device is 0000:01:00.0
PCI: Using IRQ router PIIX/ICH [8086/7110] at 0000:00:07.0
PCI: setting IRQ 11 as level-triggered
PCI: Found IRQ 11 for device 0000:00:07.2
PCI: Sharing IRQ 11 with 0000:00:0b.0
Time: tsc clocksource has been installed.
PCI: Bridge: 0000:00:01.0
  IO window: b000-bfff
  MEM window: efe00000-efefffff
  PREFETCH window: e5c00000-e7cfffff
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
TCP established hash table entries: 8192 (order: 4, 65536 bytes)
TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
TCP reno registered
SGI XFS with no debug enabled
io scheduler noop registered
io scheduler deadline registered (default)
Limiting direct PCI/PCI transfers.
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with
idebus=xx
PIIX4: IDE controller at PCI slot 0000:00:07.1
PIIX4: chipset revision 1
PIIX4: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:pio, hdd:pio
Probing IDE interface ide0...
hda: QUANTUM FIREBALL EX3.2A, ATA DISK drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
hda: max request size: 128KiB
hda: 6306048 sectors (3228 MB) w/418KiB Cache, CHS=6256/16/63, UDMA(33)
hda: cache flushes not supported
 hda: hda1 hda2 < hda5 hda6 hda7 hda8 hda9 >
PNP: No PS/2 controller found. Probing ports directly.
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
nf_conntrack version 0.5.0 (1280 buckets, 10240 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
NET: Registered protocol family 15
Using IPI Shortcut mode
Filesystem "hda1": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda1
Ending clean XFS mount for filesystem: hda1
VFS: Mounted root (xfs filesystem) readonly.
Freeing unused kernel memory: 156k freed
input: AT Translated Set 2 keyboard as /class/input/input0
Adding 209624k swap on /dev/hda9.  Priority:-1 extents:1 across:209624k
Filesystem "hda1": Disabling barriers, not supported by the underlying
device
Filesystem "hda1": Disabling barriers, not supported by the underlying
device
PCI: setting IRQ 9 as level-triggered
PCI: Found IRQ 9 for device 0000:00:09.0
3c59x: Donald Becker and others. www.scyld.com/network/vortex.html
0000:00:09.0: 3Com PCI 3c905 Boomerang 100baseTx at 0001de00.
PCI: setting IRQ 10 as level-triggered
PCI: Found IRQ 10 for device 0000:00:0a.0
0000:00:0a.0: 3Com PCI 3c905 Boomerang 100baseTx at 0001dc00.
PCI: Found IRQ 11 for device 0000:00:0b.0
PCI: Sharing IRQ 11 with 0000:00:07.2
0000:00:0b.0: 3Com PCI 3c905 Boomerang 100baseTx at 0001da00.
Filesystem "hda5": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda5
Ending clean XFS mount for filesystem: hda5
Filesystem "hda6": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda6
Ending clean XFS mount for filesystem: hda6
Filesystem "hda7": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda7
Ending clean XFS mount for filesystem: hda7
Filesystem "hda8": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda8
Ending clean XFS mount for filesystem: hda8
eth0:  setting full-duplex.
eth1:  setting full-duplex.
eth2:  setting full-duplex.
0x0: 59 fe cf 04 98 58 bc e2 42 3a 05 ee b2 12 b7 25
Filesystem "hda8": XFS internal error xfs_da_do_buf(2) at line 2086 of
file fs/xfs/xfs_da_btree.c.  Caller 0xc01a7aa8
 [<c01a75bb>] xfs_da_do_buf+0x37b/0x7c0
 [<c01a7aa8>] xfs_da_read_buf+0x48/0x60
 [<c01a7aa8>] xfs_da_read_buf+0x48/0x60
 [<c011649e>] profile_tick+0x3e/0x70
 [<c012bf4f>] tick_handle_periodic+0xf/0x60
 [<c01a7aa8>] xfs_da_read_buf+0x48/0x60
 [<c01ad28d>] xfs_dir2_leaf_lookup_int+0x16d/0x2b0
 [<c01ad28d>] xfs_dir2_leaf_lookup_int+0x16d/0x2b0
 [<c01ad07b>] xfs_dir2_leaf_lookup+0x2b/0xd0
 [<c01a8e30>] xfs_dir2_isleaf+0x20/0x70
 [<c01a8561>] xfs_dir_lookup+0xf1/0x110
 [<c0296712>] ip_route_output_flow+0x22/0x90
 [<c02a06a5>] inet_csk_route_req+0xa5/0x140
 [<c01d1f64>] xfs_dir_lookup_int+0x34/0x100
 [<c026e5db>] sk_alloc+0x2b/0xd0
 [<c01d758e>] xfs_lookup+0x4e/0x80
 [<c01e2ac2>] xfs_vn_lookup+0x52/0x90
 [<c0157337>] real_lookup+0xc7/0xf0
 [<c01575b0>] do_lookup+0x90/0xc0
 [<c0157b8b>] __link_path_walk+0x5ab/0xa70
 [<c026f2a7>] sk_stop_timer+0x17/0x20
 [<c0158095>] link_path_walk+0x45/0xd0
 [<c0276407>] process_backlog+0x77/0xf0
 [<c0150334>] get_unused_fd+0x54/0xa0
 [<c015836d>] do_path_lookup+0xdd/0x1a0
 [<c01584a0>] __path_lookup_intent_open+0x50/0x90
 [<c0158501>] path_lookup_open+0x21/0x30
 [<c0158ce8>] open_namei+0x68/0x580
 [<c0297cb2>] ip_rcv+0x212/0x460
 [<c0298090>] ip_rcv_finish+0x0/0x240
 [<c015017e>] do_filp_open+0x2e/0x50
 [<c0276407>] process_backlog+0x77/0xf0
 [<c0150334>] get_unused_fd+0x54/0xa0
 [<c0150442>] do_sys_open+0x42/0xd0
 [<c01504ec>] sys_open+0x1c/0x20
 [<c01028fc>] syscall_call+0x7/0xb
 [<c02e0000>] pfkey_xfrm_state2msg+0x4e0/0xb70
 =======================
0x0: 59 fe cf 04 98 58 bc e2 42 3a 05 ee b2 12 b7 25
Filesystem "hda8": XFS internal error xfs_da_do_buf(2) at line 2086 of
file fs/xfs/xfs_da_btree.c.  Caller 0xc01a7aa8
 [<c01a75bb>] xfs_da_do_buf+0x37b/0x7c0
 [<c01a7aa8>] xfs_da_read_buf+0x48/0x60
 [<c01a7aa8>] xfs_da_read_buf+0x48/0x60
 [<c01cfdea>] xfs_trans_unreserve_and_mod_sb+0x20a/0x210
 [<c01c36dc>] xlog_assign_tail_lsn+0xc/0x20
 [<c01a7aa8>] xfs_da_read_buf+0x48/0x60
 [<c01ad28d>] xfs_dir2_leaf_lookup_int+0x16d/0x2b0
 [<c01ad28d>] xfs_dir2_leaf_lookup_int+0x16d/0x2b0
 [<c01ad07b>] xfs_dir2_leaf_lookup+0x2b/0xd0
 [<c01a8e30>] xfs_dir2_isleaf+0x20/0x70
 [<c01a8561>] xfs_dir_lookup+0xf1/0x110
 [<c015608a>] pipe_read+0x20a/0x2c0
 [<c01d1f64>] xfs_dir_lookup_int+0x34/0x100
 [<c01580b9>] link_path_walk+0x69/0xd0
 [<c01d758e>] xfs_lookup+0x4e/0x80
 [<c01e2ac2>] xfs_vn_lookup+0x52/0x90
 [<c0158619>] __lookup_hash+0x89/0xb0
 [<c0159a21>] do_unlinkat+0x61/0x110
 [<c0150d48>] vfs_read+0xe8/0x110
 [<c0150fc7>] sys_read+0x47/0x80
 [<c01028fc>] syscall_call+0x7/0xb
 =======================
0x0: 59 fe cf 04 98 58 bc e2 42 3a 05 ee b2 12 b7 25
Filesystem "hda8": XFS internal error xfs_da_do_buf(2) at line 2086 of
file fs/xfs/xfs_da_btree.c.  Caller 0xc01a7aa8
 [<c01a75bb>] xfs_da_do_buf+0x37b/0x7c0
 [<c01a7aa8>] xfs_da_read_buf+0x48/0x60
 [<c01a7aa8>] xfs_da_read_buf+0x48/0x60
 [<ca835137>] issue_and_wait+0x27/0xb0 [3c59x]
 [<c01a7aa8>] xfs_da_read_buf+0x48/0x60
 [<c01ad28d>] xfs_dir2_leaf_lookup_int+0x16d/0x2b0
 [<c01ad28d>] xfs_dir2_leaf_lookup_int+0x16d/0x2b0
 [<c01ad07b>] xfs_dir2_leaf_lookup+0x2b/0xd0
 [<c01a8e30>] xfs_dir2_isleaf+0x20/0x70
 [<c01a8561>] xfs_dir_lookup+0xf1/0x110
 [<c0275dd5>] dev_queue_xmit+0x165/0x220
 [<c029ad18>] ip_output+0x158/0x270
 [<c01d1f64>] xfs_dir_lookup_int+0x34/0x100
 [<c01d758e>] xfs_lookup+0x4e/0x80
 [<c01e2ac2>] xfs_vn_lookup+0x52/0x90
 [<c0157337>] real_lookup+0xc7/0xf0
 [<c01575b0>] do_lookup+0x90/0xc0
 [<c0157b8b>] __link_path_walk+0x5ab/0xa70
 [<c0158095>] link_path_walk+0x45/0xd0
 [<c0150334>] get_unused_fd+0x54/0xa0
 [<c015836d>] do_path_lookup+0xdd/0x1a0
 [<c01344c7>] handle_IRQ_event+0x27/0x60
 [<c01584a0>] __path_lookup_intent_open+0x50/0x90
 [<c0158501>] path_lookup_open+0x21/0x30
 [<c0158ce8>] open_namei+0x68/0x580
 [<c0141f47>] do_wp_page+0x2a7/0x3a0
 [<c015017e>] do_filp_open+0x2e/0x50
 [<c0150334>] get_unused_fd+0x54/0xa0
 [<c0150442>] do_sys_open+0x42/0xd0
 [<c01504ec>] sys_open+0x1c/0x20
 [<c01028fc>] syscall_call+0x7/0xb
 =======================
0x0: 59 fe cf 04 98 58 bc e2 42 3a 05 ee b2 12 b7 25
Filesystem "hda8": XFS internal error xfs_da_do_buf(2) at line 2086 of
file fs/xfs/xfs_da_btree.c.  Caller 0xc01a7aa8
 [<c01a75bb>] xfs_da_do_buf+0x37b/0x7c0
 [<c01a7aa8>] xfs_da_read_buf+0x48/0x60
 [<c01a7aa8>] xfs_da_read_buf+0x48/0x60
 [<c01cfdea>] xfs_trans_unreserve_and_mod_sb+0x20a/0x210
 [<c01c36dc>] xlog_assign_tail_lsn+0xc/0x20
 [<c01a7aa8>] xfs_da_read_buf+0x48/0x60
 [<c01ad28d>] xfs_dir2_leaf_lookup_int+0x16d/0x2b0
 [<c01ad28d>] xfs_dir2_leaf_lookup_int+0x16d/0x2b0
 [<c01ad07b>] xfs_dir2_leaf_lookup+0x2b/0xd0
 [<c01a8e30>] xfs_dir2_isleaf+0x20/0x70
 [<c01a8561>] xfs_dir_lookup+0xf1/0x110
 [<c015608a>] pipe_read+0x20a/0x2c0
 [<c01d1f64>] xfs_dir_lookup_int+0x34/0x100
 [<c01580b9>] link_path_walk+0x69/0xd0
 [<c01d758e>] xfs_lookup+0x4e/0x80
 [<c01e2ac2>] xfs_vn_lookup+0x52/0x90
 [<c0158619>] __lookup_hash+0x89/0xb0
 [<c0159a21>] do_unlinkat+0x61/0x110
 [<c0150d48>] vfs_read+0xe8/0x110
 [<c0150fc7>] sys_read+0x47/0x80
 [<c01028fc>] syscall_call+0x7/0xb
 =======================
0x0: 59 fe cf 04 98 58 bc e2 42 3a 05 ee b2 12 b7 25
Filesystem "hda8": XFS internal error xfs_da_do_buf(2) at line 2086 of
file fs/xfs/xfs_da_btree.c.  Caller 0xc01a7aa8
 [<c01a75bb>] xfs_da_do_buf+0x37b/0x7c0
 [<c01a7aa8>] xfs_da_read_buf+0x48/0x60
 [<c01a7aa8>] xfs_da_read_buf+0x48/0x60
 [<c01a7aa8>] xfs_da_read_buf+0x48/0x60
 [<c01ac6cf>] xfs_dir2_leaf_getdents+0x35f/0xb40
 [<c01ac6cf>] xfs_dir2_leaf_getdents+0x35f/0xb40
 [<c013a740>] get_page_from_freelist+0x80/0xc0
 [<c01a8742>] xfs_dir_getdents+0xd2/0x120
 [<c01a8e80>] xfs_dir2_put_dirent64_direct+0x0/0x90
 [<c01a8e80>] xfs_dir2_put_dirent64_direct+0x0/0x90
 [<c01d9228>] xfs_readdir+0x48/0x70
 [<c01e0690>] xfs_file_readdir+0x100/0x220
 [<c015b9f0>] filldir+0x0/0x100
 [<c015402b>] sys_fstat64+0x2b/0x30
 [<c015b9f0>] filldir+0x0/0x100
 [<c015b891>] vfs_readdir+0x81/0xa0
 [<c015bb4e>] sys_getdents+0x5e/0xa0
 [<c01028fc>] syscall_call+0x7/0xb
 =======================
Filesystem "hda8": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda8
Ending clean XFS mount for filesystem: hda8

xfr_repair output:

Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - clear lost+found (if it exists) ...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - ensuring existence of lost+found directory
        - traversing filesystem starting at / ...
        - traversal finished ...
        - traversing all unattached subtrees ...
        - traversals finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done