* 2.6.31.6: XFS DEBUG: Assertions cause kernel OOPS.
@ 2009-11-30 13:22 Justin Piszcz
2009-11-30 13:59 ` Alex Elder
2009-11-30 23:38 ` Dave Chinner
0 siblings, 2 replies; 6+ messages in thread
From: Justin Piszcz @ 2009-11-30 13:22 UTC (permalink / raw)
To: xfs
Hi,
I wanted to compile my kernel with DEBUG for XFS and also kernel frame pointers
to catch any issues.
However, DEBUG for XFS does more harm than good?
I tried to install bonnie++:
# apt-get install -y bonnie++
And then this happened:
[ 1578.768558] Assertion failed: *nmap >= 1, file: fs/xfs/xfs_bmap.c, line: 4846
[ 1578.768633] ------------[ cut here ]------------
[ 1578.768690] kernel BUG at fs/xfs/support/debug.c:109!
[ 1578.768742] invalid opcode: 0000 [#1] SMP
[ 1578.768839] last sysfs file: /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/host15/target15:0:0/15:0:0:0/queue_depth
[ 1578.768939] CPU 1
[ 1578.769016] Pid: 3273, comm: dpkg Not tainted 2.6.31.6 #6
[ 1578.769083] RIP: 0010:[<ffffffff81243ada>] [<ffffffff81243ada>] assfail+0x1a/0x20
[ 1578.769203] RSP: 0018:ffff880121d4bc08 EFLAGS: 00010296
[ 1578.769252] RAX: 0000000000000054 RBX: 0000000000000000 RCX: ffffffff81364fb0
[ 1578.769252] RDX: ffff880028038000 RSI: 0000000000000046 RDI: ffffffff816b4730
[ 1578.769252] RBP: ffff880121d4bc08 R08: 0000000000000000 R09: 0000000000000006
[ 1578.769252] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000800000000
[ 1578.769252] R13: ffff88012a295580 R14: 0000000000000002 R15: ffff88012a522000
[ 1578.769252] FS: 00007f2a7367d790(0000) GS:ffff880028038000(0000) knlGS:0000000000000000
[ 1578.769252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1578.769252] CR2: 000000000204e000 CR3: 0000000121d53000 CR4: 00000000000026e0
[ 1578.769252] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1578.769252] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1578.769252] Process dpkg (pid: 3273, threadinfo ffff880121d4a000, task ffff88012b98b990)
[ 1578.769252] Stack:
[ 1578.769252] ffff880121d4bdb8 ffffffff811eb916 0000000121d4bc38 000000000204e000
[ 1578.769252] <0> ffff8801244a85d0 0000000000000286 ffff880121d4bc68 ffffffff81260c51
[ 1578.769252] <0> ffff880121d4bc68 0000000000000000 000000080204e000 00000000007ffff8
[ 1578.769252] Call Trace:
[ 1578.769252] [<ffffffff811eb916>] xfs_bmapi+0x66/0x1940
[ 1578.769252] [<ffffffff81260c51>] ? __up_read+0x91/0xb0
[ 1578.769252] [<ffffffff8123a0a7>] ? xfs_buf_free+0xc7/0x110
[ 1578.769252] [<ffffffff8123a1e6>] ? xfs_buf_rele+0xf6/0x130
[ 1578.769252] [<ffffffff8122e70a>] ? xfs_trans_brelse+0x19a/0x2a0
[ 1578.769252] [<ffffffff811f7ac7>] ? xfs_da_brelse+0x77/0x120
[ 1578.769252] [<ffffffff810bf63d>] ? filldir+0x9d/0xe0
[ 1578.769252] [<ffffffff8120225a>] xfs_dir2_leaf_getdents+0x61a/0x780
[ 1578.769252] [<ffffffff810bf5a0>] ? filldir+0x0/0xe0
[ 1578.769252] [<ffffffff810bf5a0>] ? filldir+0x0/0xe0
[ 1578.769252] [<ffffffff811fc6ec>] xfs_readdir+0x10c/0x110
[ 1578.769252] [<ffffffff810bf5a0>] ? filldir+0x0/0xe0
[ 1578.769252] [<ffffffff8123b4f4>] xfs_file_readdir+0x34/0x50
[ 1578.769252] [<ffffffff810bf7f8>] vfs_readdir+0xa8/0xc0
[ 1578.769252] [<ffffffff810bf963>] sys_getdents+0x83/0xe0
[ 1578.769252] [<ffffffff8102d1ab>] system_call_fastpath+0x16/0x1b
[ 1578.769252] Code: 57 81 48 c7 c7 8e 94 57 81 e8 b3 19 02 00 c9 c3 90 55 89 d1 48 89 e5 48 89 f2 31 c0 48 89 fe 48 c7 c7 f0 d0 59 81 e8 15 6a 25 00 <0f> 0b eb fe 66 90 55 48 89 e5 41 57 41 56 49 89 ce 41 55 49 89
[ 1578.773074] RIP [<ffffffff81243ada>] assfail+0x1a/0x20
[ 1578.773074] RSP <ffff880121d4bc08>
[ 1578.773343] ---[ end trace 00b4a182e72dfb60 ]---
Will recompile without XFS DEBUG but FYI.
Justin.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: 2.6.31.6: XFS DEBUG: Assertions cause kernel OOPS.
2009-11-30 13:22 2.6.31.6: XFS DEBUG: Assertions cause kernel OOPS Justin Piszcz
@ 2009-11-30 13:59 ` Alex Elder
2009-11-30 14:15 ` Justin Piszcz
2009-11-30 23:38 ` Dave Chinner
1 sibling, 1 reply; 6+ messages in thread
From: Alex Elder @ 2009-11-30 13:59 UTC (permalink / raw)
To: Justin Piszcz; +Cc: xfs
Justin Piszcz wrote:
> Hi,
>
> I wanted to compile my kernel with DEBUG for XFS and also kernel frame pointers
> to catch any issues.
>
> However, DEBUG for XFS does more harm than good?
It should not. This simple patch would improve things
but I'll add to my to-do list a review of assertions
to find and fix things of this type.
-Alex
diff -Naur old/fs/xfs/xfs_bmap.c new/fs/xfs/xfs_bmap.c
--- old/fs/xfs/xfs_bmap.c 2009-11-30 07:55:59.000000000 -0600
+++ new/fs/xfs/xfs_bmap.c 2009-11-30 07:56:43.000000000 -0600
@@ -4843,6 +4843,7 @@
orig_mval = mval;
orig_nmap = *nmap;
#endif
+ ASSERT(nmap != NULL);
ASSERT(*nmap >= 1);
ASSERT(*nmap <= XFS_BMAP_MAX_NMAP || !(flags & XFS_BMAPI_WRITE));
whichfork = (flags & XFS_BMAPI_ATTRFORK) ?
> I tried to install bonnie++:
> # apt-get install -y bonnie++
>
> And then this happened:
>
> [ 1578.768558] Assertion failed: *nmap >= 1, file: fs/xfs/xfs_bmap.c, line: 4846
> [ 1578.768633] ------------[ cut here ]------------
> [ 1578.768690] kernel BUG at fs/xfs/support/debug.c:109!
> [ 1578.768742] invalid opcode: 0000 [#1] SMP
> [ 1578.768839] last sysfs file:
> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/host15/target15:0:0/15:0:0:0/queue_depth [
> 1578.768939] CPU 1 [ 1578.769016] Pid: 3273, comm: dpkg Not tainted 2.6.31.6 #6
> [ 1578.769083] RIP: 0010:[<ffffffff81243ada>] [<ffffffff81243ada>] assfail+0x1a/0x20
> [ 1578.769203] RSP: 0018:ffff880121d4bc08 EFLAGS: 00010296
> [ 1578.769252] RAX: 0000000000000054 RBX: 0000000000000000 RCX: ffffffff81364fb0
> [ 1578.769252] RDX: ffff880028038000 RSI: 0000000000000046 RDI: ffffffff816b4730
> [ 1578.769252] RBP: ffff880121d4bc08 R08: 0000000000000000 R09: 0000000000000006
> [ 1578.769252] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000800000000
> [ 1578.769252] R13: ffff88012a295580 R14: 0000000000000002 R15: ffff88012a522000
> [ 1578.769252] FS: 00007f2a7367d790(0000) GS:ffff880028038000(0000) knlGS:0000000000000000
> [ 1578.769252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 1578.769252] CR2: 000000000204e000 CR3: 0000000121d53000 CR4: 00000000000026e0
> [ 1578.769252] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 1578.769252] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 1578.769252] Process dpkg (pid: 3273, threadinfo ffff880121d4a000, task ffff88012b98b990)
> [ 1578.769252] Stack:
> [ 1578.769252] ffff880121d4bdb8 ffffffff811eb916 0000000121d4bc38 000000000204e000
> [ 1578.769252] <0> ffff8801244a85d0 0000000000000286 ffff880121d4bc68 ffffffff81260c51
> [ 1578.769252] <0> ffff880121d4bc68 0000000000000000 000000080204e000 00000000007ffff8
> [ 1578.769252] Call Trace:
> [ 1578.769252] [<ffffffff811eb916>] xfs_bmapi+0x66/0x1940
> [ 1578.769252] [<ffffffff81260c51>] ? __up_read+0x91/0xb0
> [ 1578.769252] [<ffffffff8123a0a7>] ? xfs_buf_free+0xc7/0x110
> [ 1578.769252] [<ffffffff8123a1e6>] ? xfs_buf_rele+0xf6/0x130
> [ 1578.769252] [<ffffffff8122e70a>] ? xfs_trans_brelse+0x19a/0x2a0
> [ 1578.769252] [<ffffffff811f7ac7>] ? xfs_da_brelse+0x77/0x120
> [ 1578.769252] [<ffffffff810bf63d>] ? filldir+0x9d/0xe0
> [ 1578.769252] [<ffffffff8120225a>] xfs_dir2_leaf_getdents+0x61a/0x780
> [ 1578.769252] [<ffffffff810bf5a0>] ? filldir+0x0/0xe0
> [ 1578.769252] [<ffffffff810bf5a0>] ? filldir+0x0/0xe0
> [ 1578.769252] [<ffffffff811fc6ec>] xfs_readdir+0x10c/0x110
> [ 1578.769252] [<ffffffff810bf5a0>] ? filldir+0x0/0xe0
> [ 1578.769252] [<ffffffff8123b4f4>] xfs_file_readdir+0x34/0x50
> [ 1578.769252] [<ffffffff810bf7f8>] vfs_readdir+0xa8/0xc0
> [ 1578.769252] [<ffffffff810bf963>] sys_getdents+0x83/0xe0
> [ 1578.769252] [<ffffffff8102d1ab>] system_call_fastpath+0x16/0x1b
> [ 1578.769252] Code: 57 81 48 c7 c7 8e 94 57 81 e8 b3 19 02 00 c9 c3 90 55 89 d1 48 89 e5 48 89
> f2 31 c0 48 89 fe 48 c7 c7 f0 d0 59 81 e8 15 6a 25 00 <0f> 0b eb fe 66 90 55 48 89 e5 41 57 41
> 56 49 89 ce 41 55 49 89 [ 1578.773074] RIP [<ffffffff81243ada>] assfail+0x1a/0x20 [
> 1578.773074] RSP <ffff880121d4bc08> [ 1578.773343] ---[ end trace 00b4a182e72dfb60 ]---
>
> Will recompile without XFS DEBUG but FYI.
>
> Justin.
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 6+ messages in thread* RE: 2.6.31.6: XFS DEBUG: Assertions cause kernel OOPS.
2009-11-30 13:59 ` Alex Elder
@ 2009-11-30 14:15 ` Justin Piszcz
0 siblings, 0 replies; 6+ messages in thread
From: Justin Piszcz @ 2009-11-30 14:15 UTC (permalink / raw)
To: Alex Elder; +Cc: xfs
On Mon, 30 Nov 2009, Alex Elder wrote:
> Justin Piszcz wrote:
>> Hi,
>>
>> I wanted to compile my kernel with DEBUG for XFS and also kernel frame pointers
>> to catch any issues.
>>
>> However, DEBUG for XFS does more harm than good?
>
> It should not. This simple patch would improve things
> but I'll add to my to-do list a review of assertions
> to find and fix things of this type.
>
> -Alex
>
> diff -Naur old/fs/xfs/xfs_bmap.c new/fs/xfs/xfs_bmap.c
> --- old/fs/xfs/xfs_bmap.c 2009-11-30 07:55:59.000000000 -0600
> +++ new/fs/xfs/xfs_bmap.c 2009-11-30 07:56:43.000000000 -0600
> @@ -4843,6 +4843,7 @@
> orig_mval = mval;
> orig_nmap = *nmap;
> #endif
> + ASSERT(nmap != NULL);
> ASSERT(*nmap >= 1);
> ASSERT(*nmap <= XFS_BMAP_MAX_NMAP || !(flags & XFS_BMAPI_WRITE));
> whichfork = (flags & XFS_BMAPI_ATTRFORK) ?
>
>
The patch failed against 2.6.31.6 but I added that line at that point anyway,
ok thanks will let you know if I see anything else with DEBUG enabled.
Justin.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.6.31.6: XFS DEBUG: Assertions cause kernel OOPS.
2009-11-30 13:22 2.6.31.6: XFS DEBUG: Assertions cause kernel OOPS Justin Piszcz
2009-11-30 13:59 ` Alex Elder
@ 2009-11-30 23:38 ` Dave Chinner
2009-12-01 18:39 ` Justin Piszcz
1 sibling, 1 reply; 6+ messages in thread
From: Dave Chinner @ 2009-11-30 23:38 UTC (permalink / raw)
To: Justin Piszcz; +Cc: xfs
On Mon, Nov 30, 2009 at 08:22:03AM -0500, Justin Piszcz wrote:
> Hi,
>
> I wanted to compile my kernel with DEBUG for XFS and also kernel frame pointers
> to catch any issues.
>
> However, DEBUG for XFS does more harm than good?
DEBUG is there for developers, not end users. Often the debug code
will assert fail or panic where the situation is not fatal but
indicative of some problem that should be looked into further.
> I tried to install bonnie++:
> # apt-get install -y bonnie++
>
> And then this happened:
>
> [ 1578.768558] Assertion failed: *nmap >= 1, file: fs/xfs/xfs_bmap.c, line: 4846
....
> [ 1578.769252] [<ffffffff811eb916>] xfs_bmapi+0x66/0x1940
> [ 1578.769252] [<ffffffff81260c51>] ? __up_read+0x91/0xb0
> [ 1578.769252] [<ffffffff8123a0a7>] ? xfs_buf_free+0xc7/0x110
> [ 1578.769252] [<ffffffff8123a1e6>] ? xfs_buf_rele+0xf6/0x130
> [ 1578.769252] [<ffffffff8122e70a>] ? xfs_trans_brelse+0x19a/0x2a0
> [ 1578.769252] [<ffffffff811f7ac7>] ? xfs_da_brelse+0x77/0x120
> [ 1578.769252] [<ffffffff810bf63d>] ? filldir+0x9d/0xe0
> [ 1578.769252] [<ffffffff8120225a>] xfs_dir2_leaf_getdents+0x61a/0x780
> [ 1578.769252] [<ffffffff810bf5a0>] ? filldir+0x0/0xe0
> [ 1578.769252] [<ffffffff810bf5a0>] ? filldir+0x0/0xe0
> [ 1578.769252] [<ffffffff811fc6ec>] xfs_readdir+0x10c/0x110
> [ 1578.769252] [<ffffffff810bf5a0>] ? filldir+0x0/0xe0
> [ 1578.769252] [<ffffffff8123b4f4>] xfs_file_readdir+0x34/0x50
> [ 1578.769252] [<ffffffff810bf7f8>] vfs_readdir+0xa8/0xc0
> [ 1578.769252] [<ffffffff810bf963>] sys_getdents+0x83/0xe0
> [ 1578.769252] [<ffffffff8102d1ab>] system_call_fastpath+0x16/0x1b
That is indicative of trying to map more blocks that we can fit
in the block map array during directory leaf readahead. That is
definitely not a problem for production systems, but definitely
indicates that the readahead loop termination has a bug in it
somewhere.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.6.31.6: XFS DEBUG: Assertions cause kernel OOPS.
2009-11-30 23:38 ` Dave Chinner
@ 2009-12-01 18:39 ` Justin Piszcz
2009-12-01 23:58 ` Dave Chinner
0 siblings, 1 reply; 6+ messages in thread
From: Justin Piszcz @ 2009-12-01 18:39 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
On Tue, 1 Dec 2009, Dave Chinner wrote:
> On Mon, Nov 30, 2009 at 08:22:03AM -0500, Justin Piszcz wrote:
>> Hi,
>>
>> I wanted to compile my kernel with DEBUG for XFS and also kernel frame pointers
>> to catch any issues.
>>
>> However, DEBUG for XFS does more harm than good?
>
> DEBUG is there for developers, not end users. Often the debug code
> will assert fail or panic where the situation is not fatal but
> indicative of some problem that should be looked into further.
Dave,
Recall that issue that I was having with asterisk installed? How should
one get more information on a filesystem lockup if we cannot have debug
enabled for the filesystem? Should there be a user-debug option which it
will include more verbosity if/when the xfs kernel processes lockup?
>
>> I tried to install bonnie++:
>> # apt-get install -y bonnie++
>>
>> And then this happened:
>>
>> [ 1578.768558] Assertion failed: *nmap >= 1, file: fs/xfs/xfs_bmap.c, line: 4846
> ....
>> [ 1578.769252] [<ffffffff811eb916>] xfs_bmapi+0x66/0x1940
>> [ 1578.769252] [<ffffffff81260c51>] ? __up_read+0x91/0xb0
>> [ 1578.769252] [<ffffffff8123a0a7>] ? xfs_buf_free+0xc7/0x110
>> [ 1578.769252] [<ffffffff8123a1e6>] ? xfs_buf_rele+0xf6/0x130
>> [ 1578.769252] [<ffffffff8122e70a>] ? xfs_trans_brelse+0x19a/0x2a0
>> [ 1578.769252] [<ffffffff811f7ac7>] ? xfs_da_brelse+0x77/0x120
>> [ 1578.769252] [<ffffffff810bf63d>] ? filldir+0x9d/0xe0
>> [ 1578.769252] [<ffffffff8120225a>] xfs_dir2_leaf_getdents+0x61a/0x780
>> [ 1578.769252] [<ffffffff810bf5a0>] ? filldir+0x0/0xe0
>> [ 1578.769252] [<ffffffff810bf5a0>] ? filldir+0x0/0xe0
>> [ 1578.769252] [<ffffffff811fc6ec>] xfs_readdir+0x10c/0x110
>> [ 1578.769252] [<ffffffff810bf5a0>] ? filldir+0x0/0xe0
>> [ 1578.769252] [<ffffffff8123b4f4>] xfs_file_readdir+0x34/0x50
>> [ 1578.769252] [<ffffffff810bf7f8>] vfs_readdir+0xa8/0xc0
>> [ 1578.769252] [<ffffffff810bf963>] sys_getdents+0x83/0xe0
>> [ 1578.769252] [<ffffffff8102d1ab>] system_call_fastpath+0x16/0x1b
>
> That is indicative of trying to map more blocks that we can fit
> in the block map array during directory leaf readahead. That is
> definitely not a problem for production systems, but definitely
> indicates that the readahead loop termination has a bug in it
> somewhere.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.6.31.6: XFS DEBUG: Assertions cause kernel OOPS.
2009-12-01 18:39 ` Justin Piszcz
@ 2009-12-01 23:58 ` Dave Chinner
0 siblings, 0 replies; 6+ messages in thread
From: Dave Chinner @ 2009-12-01 23:58 UTC (permalink / raw)
To: Justin Piszcz; +Cc: xfs
On Tue, Dec 01, 2009 at 01:39:06PM -0500, Justin Piszcz wrote:
>
>
> On Tue, 1 Dec 2009, Dave Chinner wrote:
>
>> On Mon, Nov 30, 2009 at 08:22:03AM -0500, Justin Piszcz wrote:
>>> Hi,
>>>
>>> I wanted to compile my kernel with DEBUG for XFS and also kernel frame pointers
>>> to catch any issues.
>>>
>>> However, DEBUG for XFS does more harm than good?
>>
>> DEBUG is there for developers, not end users. Often the debug code
>> will assert fail or panic where the situation is not fatal but
>> indicative of some problem that should be looked into further.
> Dave,
>
> Recall that issue that I was having with asterisk installed?
Yes, it appeared to be caused by log IO not completing, right?
> How should
> one get more information on a filesystem lockup if we cannot have debug
> enabled for the filesystem?
The debug code in XFS won't tell you anything extra about a lock-up
unless it trips an ASSERT before the lockup that is related to the
lock-up. My experience with hangs like the above are that they
generally are not related to the filesystem at all; most of the time
it is IO not being completed by the lower layers for some reason,
and no amount of filesystem level debug will help solve that
problem.
> Should there be a user-debug option which it
> will include more verbosity if/when the xfs kernel processes lockup?
See /proc/sys/fs/xfs/error_level. That can turn up the verbosity of
error reporting without needing DEBUG compiled in.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2009-12-01 23:57 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-11-30 13:22 2.6.31.6: XFS DEBUG: Assertions cause kernel OOPS Justin Piszcz
2009-11-30 13:59 ` Alex Elder
2009-11-30 14:15 ` Justin Piszcz
2009-11-30 23:38 ` Dave Chinner
2009-12-01 18:39 ` Justin Piszcz
2009-12-01 23:58 ` Dave Chinner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox