* xfstest 179 ASSERT
@ 2012-10-31 18:23 Mark Tinguely
2012-11-01 1:49 ` Dave Chinner
0 siblings, 1 reply; 3+ messages in thread
From: Mark Tinguely @ 2012-10-31 18:23 UTC (permalink / raw)
To: Dave Chinner, xfs-oss
OSS sources with the xfs: fix buffer shudown reference count mismatch
patch and xfstest 179.
xfstest 179 started to have various asserts starting with the "move the
workers" series, but mostly the b_hold count is zero assert.
Now that the b_hold count is fixed, the asert is:
XFS: Assertion failed: atomic_read(&pag->pag_ref) == 0,
file: /root/xfs/fs/xfs/xfs_mount.c, line: 273
PID: 6741 TASK: ffff880268f46540 CPU: 0 COMMAND: "umount"
#0 [ffff88034bba1a70] machine_kexec at ffffffff8102a71d
#1 [ffff88034bba1ae0] crash_kexec at ffffffff810a4703
#2 [ffff88034bba1bb0] oops_end at ffffffff81432098
#3 [ffff88034bba1be0] die at ffffffff81005763
#4 [ffff88034bba1c10] do_trap at ffffffff814319b3
#5 [ffff88034bba1c70] do_invalid_op at ffffffff81002f50
#6 [ffff88034bba1d10] invalid_op at ffffffff8143a09e
[exception RIP: assfail+29]
RIP: ffffffffa038e7dd RSP: ffff88034bba1dc8 RFLAGS: 00010292
RAX: 0000000000000065 RBX: 0000000000000000 RCX: 0000000000000de5
RDX: 0000000000002013 RSI: 0000000000000092 RDI: 0000000000000246
RBP: ffff88034bba1dc8 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000352 R11: 0000000000000351 R12: ffff88034d8af1f8
R13: ffff88034d8af000 R14: ffff88034d8af1e8 R15: ffff88034fbf28c0
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#7 [ffff88034bba1dd0] xfs_free_perag at ffffffffa03de5d8 [xfs]
#8 [ffff88034bba1e10] xfs_unmountfs at ffffffffa03e08bc [xfs]
#9 [ffff88034bba1e60] xfs_fs_put_super at ffffffffa0390e80 [xfs]
#10 [ffff88034bba1e80] generic_shutdown_super at ffffffff8114f37d
#11 [ffff88034bba1eb0] kill_block_super at ffffffff8114f42b
#12 [ffff88034bba1ed0] deactivate_locked_super at ffffffff8114f9e7
#13 [ffff88034bba1ef0] deactivate_super at ffffffff811502d9
#14 [ffff88034bba1f10] mntput_no_expire at ffffffff8116a821
#15 [ffff88034bba1f40] sys_umount at ffffffff8116bd21
#16 [ffff88034bba1f80] system_call_fastpath at ffffffff814390a9
RIP: 00007f64a30cf5d7 RSP: 00007ffffe2f3798 RFLAGS: 00010202
RAX: 00000000000000a6 RBX: ffffffff814390a9 RCX: 000000000000c0c8
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00007f64a3c0e0f0
RBP: 00007f64a3c0dff0 R8: 0000000000000000 R9: 00007f64a3a05435
R10: 00007ffffe2f35c0 R11: 0000000000000246 R12: 00007f64a3c0e0f0
R13: 00007f64a3c0e0d0 R14: 00007f64a3c0e0f0 R15: 0000000000000000
ORIG_RAX: 00000000000000a6 CS: 0033 SS: 002b
mount perag information:
m_perag_tree = {
height = 0x1,
gfp_mask = 0x20,
rnode = 0xffff8803124bb4b1
},
crash> radix_tree_node ffff8803124bb4b0
struct radix_tree_node {
height = 0x1,
count = 0x3,
{
parent = 0x0,
callback_head = {
next = 0x0,
func = 0xffffffff812384b0 <radix_tree_node_rcu_free>
}
},
slots = {0x0, 0xffff88034fd6f540, 0xffff88034fd6f840,
0xffff88034fd6f240, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
tags = {{0x0}, {0x0}, {0x0}}
}
The inodes at 0xffff88034fd6f540, 0xffff88034fd6f840, 0xffff88034fd6f240
don't look valid.
I haven't spent much time on this.
--Mark.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: xfstest 179 ASSERT
2012-10-31 18:23 xfstest 179 ASSERT Mark Tinguely
@ 2012-11-01 1:49 ` Dave Chinner
2012-11-01 13:08 ` Mark Tinguely
0 siblings, 1 reply; 3+ messages in thread
From: Dave Chinner @ 2012-11-01 1:49 UTC (permalink / raw)
To: Mark Tinguely; +Cc: xfs-oss
On Wed, Oct 31, 2012 at 01:23:54PM -0500, Mark Tinguely wrote:
> OSS sources with the xfs: fix buffer shudown reference count mismatch
> patch and xfstest 179.
>
> xfstest 179 started to have various asserts starting with the "move the
> workers" series, but mostly the b_hold count is zero assert.
>
> Now that the b_hold count is fixed, the asert is:
>
> XFS: Assertion failed: atomic_read(&pag->pag_ref) == 0,
> file: /root/xfs/fs/xfs/xfs_mount.c, line: 273
The only way you are going to track this down is through tracing
perag gets and puts, and finding the object that is missing a put.
There are already tracepoints for these and they tell you the caller
function, so that should be sufficient to find what function has a
get without a put...
The other possibility is that there are buffers that have not be
freed properly (or leaked), but IIRC this failure pre-exists
attaching the perag to buffers....
> mount perag information:
> m_perag_tree = {
> height = 0x1,
> gfp_mask = 0x20,
> rnode = 0xffff8803124bb4b1
> },
> crash> radix_tree_node ffff8803124bb4b0
> struct radix_tree_node {
> height = 0x1,
> count = 0x3,
> {
> parent = 0x0,
> callback_head = {
> next = 0x0,
> func = 0xffffffff812384b0 <radix_tree_node_rcu_free>
> }
> },
> slots = {0x0, 0xffff88034fd6f540, 0xffff88034fd6f840,
> 0xffff88034fd6f240, 0x0,
> 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
> 0x0, 0x0, 0x0,
> 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
> 0x0, 0x0, 0x0,
> 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
> 0x0, 0x0, 0x0,
> 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
> tags = {{0x0}, {0x0}, {0x0}}
> }
>
> The inodes at 0xffff88034fd6f540, 0xffff88034fd6f840, 0xffff88034fd6f240
> don't look valid.
Those aren't inodes based on the addresses - they are all in the
same page, so the size is at most 0x300 bytes (768 bytes). An XFS
inode is larger than this, but a radix tree node is smaller (560
bytes and fits 7 to a page on my x86_64 machines), so I'm not sure
what you have there. how big does /proc/slabinfo tell you a radix
tree node is?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: xfstest 179 ASSERT
2012-11-01 1:49 ` Dave Chinner
@ 2012-11-01 13:08 ` Mark Tinguely
0 siblings, 0 replies; 3+ messages in thread
From: Mark Tinguely @ 2012-11-01 13:08 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs-oss
On 10/31/12 20:49, Dave Chinner wrote:
> On Wed, Oct 31, 2012 at 01:23:54PM -0500, Mark Tinguely wrote:
>> OSS sources with the xfs: fix buffer shudown reference count mismatch
>> patch and xfstest 179.
>>
>> xfstest 179 started to have various asserts starting with the "move the
>> workers" series, but mostly the b_hold count is zero assert.
>>
>> Now that the b_hold count is fixed, the asert is:
>>
>> XFS: Assertion failed: atomic_read(&pag->pag_ref) == 0,
>> file: /root/xfs/fs/xfs/xfs_mount.c, line: 273
>
> The only way you are going to track this down is through tracing
> perag gets and puts, and finding the object that is missing a put.
> There are already tracepoints for these and they tell you the caller
> function, so that should be sufficient to find what function has a
> get without a put...
>
> The other possibility is that there are buffers that have not be
> freed properly (or leaked), but IIRC this failure pre-exists
> attaching the perag to buffers....
>
>> mount perag information:
>> m_perag_tree = {
>> height = 0x1,
>> gfp_mask = 0x20,
>> rnode = 0xffff8803124bb4b1
>> },
>> crash> radix_tree_node ffff8803124bb4b0
>> struct radix_tree_node {
>> height = 0x1,
>> count = 0x3,
>> {
>> parent = 0x0,
>> callback_head = {
>> next = 0x0,
>> func = 0xffffffff812384b0<radix_tree_node_rcu_free>
>> }
>> },
>> slots = {0x0, 0xffff88034fd6f540, 0xffff88034fd6f840,
>> 0xffff88034fd6f240, 0x0,
>> 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
>> 0x0, 0x0, 0x0,
>> 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
>> 0x0, 0x0, 0x0,
>> 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
>> 0x0, 0x0, 0x0,
>> 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
>> tags = {{0x0}, {0x0}, {0x0}}
>> }
>>
>> The inodes at 0xffff88034fd6f540, 0xffff88034fd6f840, 0xffff88034fd6f240
>> don't look valid.
>
> Those aren't inodes based on the addresses - they are all in the
> same page, so the size is at most 0x300 bytes (768 bytes). An XFS
> inode is larger than this, but a radix tree node is smaller (560
> bytes and fits 7 to a page on my x86_64 machines), so I'm not sure
> what you have there. how big does /proc/slabinfo tell you a radix
> tree node is?
>
> Cheers,
>
> Dave.
I have been spending too much time in earlier versions of XFS.
Last night, I had a good laugh at my mistake when I realized the perag
was not in an array...duh me, this is the radix tree of the perag
structures. I could not do a crash "kmem" command, that would have
given me a clue.
I *think* Ben is also seeing this assert on his 32 bit test machine.
Thanks for the advice, I will do some investigation this weekend.
--Mark.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2012-11-01 13:07 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-10-31 18:23 xfstest 179 ASSERT Mark Tinguely
2012-11-01 1:49 ` Dave Chinner
2012-11-01 13:08 ` Mark Tinguely
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox