XFS and 2.6.12-rc5

XFS and 2.6.12-rc5

[Goran Gajic]

xfs partition is exported via nfs to FreeBSD-5.4 machine. This is what I 
find every morning in my syslog:

  ------------[ cut here ]------------
  kernel BUG at fs/xfs/support/debug.c:106!
  invalid operand: 0000 [#1]
  SMP
  Modules linked in:
  CPU:    1
  EIP:    0060:[<c02c651d>]    Not tainted VLI
  EFLAGS: 00010246   (2.6.12-rc5)
  EIP is at cmn_err+0xad/0xd0
  eax: c0507cc4   ebx: c04b3c98   ecx: c04f8910 edx: 00000293
  esi: 00000206   edi: c0601300   ebp: 00000000 esp: dfcafbd4
  ds: 007b   es: 007b   ss: 0068
Process kswapd0 (pid: 168, threadinfo=dfcae000 task=dfec8020)
Stack: c04bf14d c04a1931 c0601300 00000293 dd6e5e50 00000206 00000000 dfcafd2c
        c0269758 00000000 c04b3c98 df5d4ba0 1abd9322 00000000 00000000 ffffffff
        003fffff 00000000 00000000 00000001 dfcafd0c dfcafd2c 00000000 00e2af60
Call Trace:
  [<c0269758>] xfs_bmap_search_extents+0x108/0x140
  [<c026accd>] xfs_bmapi+0x28d/0x1660
  [<c013cf5a>] find_trylock_page+0x4a/0x60
  [<c02ba25f>] xfs_probe_delalloc_page+0x1f/0xa0
  [<c014153e>] free_pages_bulk+0x1ee/0x230
  [<c0146134>] cache_flusharray+0x84/0xd0
  [<c0291bfb>] xfs_iextract+0xbb/0x1d0
  [<c02b3441>] xfs_inactive_free_eofblocks+0x101/0x300
  [<c02c5c91>] vn_wakeup+0x21/0xb0
  [<c02c6256>] vn_purge+0x156/0x170
  [<c014898e>] shrink_list+0x2de/0x450
  [<c02b3eb0>] xfs_inactive+0x120/0x5a0
  [<c0146134>] cache_flusharray+0x84/0xd0
  [<c02c637c>] vn_rele+0xac/0xf0
  [<c02c4715>] linvfs_clear_inode+0x15/0x30
  [<c01773c6>] clear_inode+0xd6/0xe0
  [<c01773ec>] dispose_list+0x1c/0xa0
  [<c01424bc>] __read_page_state+0x9c/0xd0
  [<c017781c>] prune_icache+0x18c/0x200
  [<c01778af>] shrink_icache_memory+0x1f/0x50
  [<c0148412>] shrink_slab+0x132/0x190
  [<c014989f>] balance_pgdat+0x27f/0x3e0
  [<c0149ac0>] kswapd+0xc0/0x100
  [<c0131c20>] autoremove_wake_function+0x0/0x60
  [<c0102e32>] ret_from_fork+0x6/0x14
  [<c0131c20>] autoremove_wake_function+0x0/0x60
  [<c0149a00>] kswapd+0x0/0x100
  [<c0101045>] kernel_thread_helper+0x5/0x10
Code: b8 00 13 60 c0 89 44 24 08 8b 04 ad e0 7c
50 c0 89 44 24 04 e8 75 5c e5 ff 8b 54 24 0c b8 c4 7c 50 c0 e8 c7 60 1b 00
85 ed 75 08 <0f> 0b 6a 00 49 19 4a c0 8b 5c 24 10 8b 74 24 14 8b 7c 24 18 
8b


Regards,
gg.

Re: XFS and 2.6.12-rc5

[Nathan Scott]

On Wed, Jun 01, 2005 at 01:10:47PM +0200, Goran Gajic wrote:
> xfs partition is exported via nfs to FreeBSD-5.4 machine. This is what I 
> find every morning in my syslog:
> 
>  ------------[ cut here ]------------
>  kernel BUG at fs/xfs/support/debug.c:106!
> ...
>  [<c0269758>] xfs_bmap_search_extents+0x108/0x140
>  [<c026accd>] xfs_bmapi+0x28d/0x1660

There should be some diagnostic text just above this panic message,
what does it say?  At a guess, I'd say you have a corrupt inode on
disk, and your nightly cron jobs are tripping this up each time.
The panic happens cos the kernel detects an inode with an extent
map which claiming to have an extent starting at the offset of the
primary superblock.  I've seen another case of this recently which
looked like a possible compiler bug, so could you send me both the
full diagnostic message and your compiler version number?

Also, the diagnostic will contain an inode number - for bonus points
run "xfs_db -r -c 'inode XXX' -c print /dev/foo" and send me that as
well.  Thanks!

cheers.

-- 
Nathan

Re: XFS and 2.6.12-rc5

[Goran Gajic]

On Fri, 3 Jun 2005, Nathan Scott wrote:

> On Wed, Jun 01, 2005 at 01:10:47PM +0200, Goran Gajic wrote:
>> xfs partition is exported via nfs to FreeBSD-5.4 machine. This is what I
>> find every morning in my syslog:
>>
>>  ------------[ cut here ]------------
>>  kernel BUG at fs/xfs/support/debug.c:106!
>> ...
>>  [<c0269758>] xfs_bmap_search_extents+0x108/0x140
>>  [<c026accd>] xfs_bmapi+0x28d/0x1660
>
> There should be some diagnostic text just above this panic message,
> what does it say?  At a guess, I'd say you have a corrupt inode on
> disk, and your nightly cron jobs are tripping this up each time.
> The panic happens cos the kernel detects an inode with an extent
> map which claiming to have an extent starting at the offset of the
> primary superblock.  I've seen another case of this recently which
> looked like a possible compiler bug, so could you send me both the
> full diagnostic message and your compiler version number?
>
> Also, the diagnostic will contain an inode number - for bonus points
> run "xfs_db -r -c 'inode XXX' -c print /dev/foo" and send me that as
> well.  Thanks!
>
> cheers.
>
> -- 
> Nathan
>

You are right about message here it is:

May 31 04:16:37 cbt kernel: Access to block zero: fs: <sdb1> inode: 
448631586 start_block : ffffffff00000000 start_off : 3fffff blkcnt : 
100000000 extent-state: dfcb1d0c

xfs_db -r -c 'inode 448631586' -c print /dev/sdb1
core.magic = 0x494e
core.mode = 0100644
core.version = 1
core.format = 2 (extents)
core.nlinkv1 = 1
core.uid = 0
core.gid = 0
core.flushiter = 2
core.atime.sec = Sun May 29 14:35:00 2005
core.atime.nsec = 521517264
core.mtime.sec = Sun May 29 14:40:00 2005
core.mtime.nsec = 676886712
core.ctime.sec = Sun May 29 14:40:00 2005
core.ctime.nsec = 676886712
core.size = 2120649
core.nblocks = 458
core.extsize = 0
core.nextents = 4
core.naextents = 0
core.forkoff = 0
core.aformat = 2 (extents)
core.dmevmask = 0
core.dmstate = 0
core.newrtbm = 0
core.prealloc = 0
core.realtime = 0
core.immutable = 0
core.append = 0
core.sync = 0
core.noatime = 0
core.nodump = 0
core.gen = 0
next_unlinked = null
u.bmx[0-3] = [startoff,startblock,blockcount,extentflag] 
0:[177,2557931,341,0] 1:[18014398509481983,4498651825045504,0,1] 
2:[15184073051865088,0,0,0] 3:[0,1422,1245184,0]

gcc -v
Reading specs from /usr/lib/gcc-lib/i586-suse-linux/3.3.3/specs
Configured with: ../configure --enable-threads=posix --prefix=/usr 
--with-local-prefix=/usr/local --infodir=/usr/share/info 
--mandir=/usr/share/man --enable-languages=c,c++,f77,objc,java,ada 
--disable-checking --libdir=/usr/lib --enable-libgcj 
--with-gxx-include-dir=/usr/include/g++ --with-slibdir=/lib 
--with-system-zlib --enable-shared --enable-__cxa_atexit i586-suse-linux
Thread model: posix
gcc version 3.3.3 (SuSE Linux)

Turning off cron.daily stops this message so I guess you are right.

Regards,
gg.

Re: XFS and 2.6.12-rc5

[Nathan Scott]

On Fri, Jun 03, 2005 at 10:56:24AM +0200, Goran Gajic wrote:
> >Also, the diagnostic will contain an inode number - for bonus points
> >run "xfs_db -r -c 'inode XXX' -c print /dev/foo" and send me that as
> >well.  Thanks!
> 
> You are right about message here it is:
> 
> xfs_db -r -c 'inode 448631586' -c print /dev/sdb1
> ...
> u.bmx[0-3] = [startoff,startblock,blockcount,extentflag] 
> 0:[177,2557931,341,0] 1:[18014398509481983,4498651825045504,0,1] 
> 2:[15184073051865088,0,0,0] 3:[0,1422,1245184,0]
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^

This is the guy - startblock being zero is fatal.  You'll need to
run xfs_repair to clear that up.  If you know how it got in that
state, I'd be keen to hear it, or if it persists after repair.

[Arun, those superblock checks you added awhile back should be
shutting down the filesystem, not panicing the system, I think?
It looks like theres a signedness issue in the diagnostics too.]

> Turning off cron.daily stops this message so I guess you are right.

Repairing that inode should make it go away entirely.

cheers.

-- 
Nathan