From: David Teigland <teigland@redhat.com>
To: ocfs2-devel@oss.oracle.com
Subject: [Ocfs2-devel] [PATCH] ocfs2: fix __ocfs2_cluster_lock() dead lock
Date: Tue, 12 Jan 2010 13:47:13 -0600 [thread overview]
Message-ID: <20100112194713.GD24645@redhat.com> (raw)
In-Reply-To: <20100112015946.GE20285@mail.oracle.com>
On Mon, Jan 11, 2010 at 05:59:46PM -0800, Joel Becker wrote:
> I've attached the full patch with my changes. Dave, please test
> my version (the attached one) instead of Wengang's.
Your new patch fixes the mount, so I went on to test make_panic which is
the test we never got to work:
http://oss.oracle.com/pipermail/ocfs2-devel/2009-April/004313.html
https://bugzilla.novell.com/show_bug.cgi?id=492055
It ran on three nodes for several minutes; much longer than it ever had
before. It eventually triggered different BUG's on two of the nodes,
rather than just getting stuck as it used to. I wasn't watching, so I
don't know which of these came first.
One node had:
kernel BUG at fs/ocfs2/dlmglue.c:3567!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
CPU 3
Modules linked in: ocfs2_stack_user dlm ocfs2 jbd2 ocfs2_nodemanager
configfs ocfs2_stackglue ipt_REJECT xt_tcpudp iptable_filter ip_tables
x_tables bridge stp autofs4 sunrpc ipv6 iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi cpufreq_ondemand dm_multipath video output sbs sbshc
battery ac parport_pc lp parport sg serio_raw button tg3 libphy
i2c_nforce2 i2c_core pcspkr dm_snapshot dm_zero dm_mirror dm_region_hash
dm_log dm_mod qla2xxx scsi_transport_fc shpchp mptspi mptscsih mptbase
scsi_transport_spi sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 4019, comm: ocfs2dc Not tainted 2.6.32.3 #2 ProLiant DL145 G2
RIP: 0010:[<ffffffffa03a8148>] [<ffffffffa03a8148>]
ocfs2_ci_checkpointed+0x7c/0xcb [ocfs2]
RSP: 0018:ffff8800785b7dc0 EFLAGS: 00010002
RAX: 0000000000000001 RBX: 0000000000000411 RCX: ffff8800779b53b8
RDX: 000000000000d0cf RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffff8800785b7df0 R08: ffffffffa03a810a R09: ffffffffa03aa010
R10: ffff88013f421e68 R11: ffff8800839d36c0 R12: ffffffffffffffff
R13: ffff880078b73938 R14: 0000000000000000 R15: ffff880078b73368
FS: 00007f589e5126e0(0000) GS:ffff880083a00000(0000)
knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007f0598be6000 CR3: 0000000133608000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ocfs2dc (pid: 4019, threadinfo ffff8800785b6000, task
ffff8800779b4cc0)
Stack:
ffff88007856c000 0000000000000282 ffff880078b73368 0000000000000000
<0> ffff88007856c000 0000000000000282 ffff8800785b7e00 ffffffffa03ab4b0
<0> ffff8800785b7eb0 ffffffffa03aa16d ffff8800785b7eb0 ffffffff813528ed
Call Trace:
[<ffffffffa03ab4b0>] ocfs2_check_meta_downconvert+0x32/0x34 [ocfs2]
[<ffffffffa03aa16d>] ocfs2_downconvert_thread+0x470/0x869 [ocfs2]
[<ffffffff813528ed>] ? thread_return+0x3e/0xee
[<ffffffff8105ff63>] ? autoremove_wake_function+0x0/0x3d
[<ffffffffa03a9cfd>] ? ocfs2_downconvert_thread+0x0/0x869 [ocfs2]
[<ffffffff8105fe55>] kthread+0x82/0x8d
[<ffffffff8100cc1a>] child_rip+0xa/0x20
[<ffffffff8105fdb2>] ? kthreadd+0xc7/0xe8
[<ffffffff8105fdd3>] ? kthread+0x0/0x8d
[<ffffffff8100cc10>] ? child_rip+0x0/0x20
Code: e0 45 85 f6 74 0a 41 83 fe 03 74 04 0f 0b eb fe 49 29 dc 49 f7 d4 4c
89 e0 48 c1 e8 3f 41 83 bf a0 00 00 00 05 74 08 84 c0 74 08 <0f> 0b eb fe
84 c0 75 07 b8 01 00 00 00 eb 33 4c 89 ef e8 07 ac
RIP [<ffffffffa03a8148>] ocfs2_ci_checkpointed+0x7c/0xcb [ocfs2]
RSP <ffff8800785b7dc0>
Another node had:
(2881,2):ocfs2_inode_lock_update:2224 ERROR: bug expression:
inode->i_generation != le32_to_cpu(fe->i_generation)
(2881,2):ocfs2_inode_lock_update:2224 ERROR: Invalid dinode 420382 disk
generation: 1523484106 inode->i_generation: 1523484094
------------[ cut here ]------------
kernel BUG at fs/ocfs2/dlmglue.c:2224!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
CPU 2
Modules linked in: ocfs2_stack_user dlm ocfs2 jbd2 ocfs2_nodemanager
configfs ocfs2_stackglue ipt_REJECT xt_tcpudp iptable_filter ip_tables
x_tables bridge stp sunrpc ipv6 cpufreq_ondemand dm_multipath uinput
serio_raw pcspkr sg qla2xxx scsi_transport_fc tg3 libphy i2c_nforce2
i2c_core button dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod
shpchp mptspi mptscsih mptbase scsi_transport_spi sd_mod scsi_mod ext3 jbd
uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan]
Pid: 2881, comm: make_panic Not tainted 2.6.32.3 #1 ProLiant DL145 G2
RIP: 0010:[<ffffffffa032e9a0>] [<ffffffffa032e9a0>]
ocfs2_inode_lock_full_nested+0x850/0xd00 [ocfs2]
RSP: 0018:ffff88007b55bce8 EFLAGS: 00010296
RAX: 0000000000000085 RBX: ffff880067c95000 RCX: 000000000000be01
RDX: ffff880083800000 RSI: 0000000000000001 RDI: 0000000000000003
RBP: ffff88007b55bd58 R08: 0000000000000092 R09: 0000000000000000
R10: 0000000000000003 R11: 0000000000018600 R12: ffff880066a8b2d8
R13: ffff880066a8b6e8 R14: ffff880066a8b840 R15: ffff880066a8b040
FS: 00007fa9c96b66f0(0000) GS:ffff880083800000(0000)
knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f54f5021000 CR3: 000000007e82a000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process make_panic (pid: 2881, threadinfo ffff88007b55a000, task
ffff88007e9d40c0)
Stack:
ffffffff5ace85ca 000001015ace85be 00000000810f4897 0000000000000000
<0> 0000000000000000 000000017f59ece0 ffff880066a8b228 ffffffff81103794
<0> ffff880068122cb0 ffff880066a8b840 ffff880066a8b840 0000000000000026
Call Trace:
[<ffffffff81103794>] ? mntput_no_expire+0x29/0xfd
[<ffffffffa03354f1>] ocfs2_permission+0x78/0x16f [ocfs2]
[<ffffffff810f3ea5>] inode_permission+0x6e/0x9e
[<ffffffff810f593c>] may_open+0x9e/0x252
[<ffffffff810f8389>] do_filp_open+0x51f/0xa55
[<ffffffff81101c14>] ? alloc_fd+0x122/0x133
[<ffffffff810ea0b0>] do_sys_open+0x62/0x109
[<ffffffff810ea18a>] sys_open+0x20/0x22
[<ffffffff8100bc5b>] system_call_fastpath+0x16/0x1b
Code: ff 48 c7 c1 b0 eb 37 a0 48 c7 c7 0b 6d 38 a0 65 8b 14 25 08 cd 00 00
89 44 24 08 8b 43 08 48 63 d2 89 04 24 31 c0 e8 37 cc 02 e1 <0f> 0b eb fe
48 83 7b 48 00 75 0a f6 43 2c 01 0f 85 bc 00 00 00
RIP [<ffffffffa032e9a0>] ocfs2_inode_lock_full_nested+0x850/0xd00 [ocfs2]
RSP <ffff88007b55bce8>
next prev parent reply other threads:[~2010-01-12 19:47 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-01-06 8:34 [Ocfs2-devel] [PATCH] ocfs2: fix __ocfs2_cluster_lock() dead lock Wengang Wang
2010-01-07 2:00 ` Joel Becker
2010-01-07 5:22 ` Wengang Wang
2010-01-09 18:05 ` Wengang Wang
2010-01-09 18:24 ` Sunil Mushran
2010-01-10 10:05 ` Joel Becker
2010-01-11 1:55 ` Wengang Wang
2010-01-12 1:59 ` Joel Becker
2010-01-12 3:58 ` Wengang Wang
2010-01-12 11:18 ` Joel Becker
2010-01-12 12:04 ` Wengang Wang
2010-01-12 13:01 ` Joel Becker
2010-01-12 14:30 ` Wengang Wang
2010-01-12 19:14 ` Sunil Mushran
2010-01-13 6:15 ` Wengang Wang
2010-01-12 19:47 ` David Teigland [this message]
2010-01-13 3:20 ` Wengang Wang
2010-01-13 7:08 ` Wengang Wang
2010-01-13 7:57 ` Joel Becker
2010-01-13 9:23 ` Wengang Wang
2010-01-13 9:31 ` Joel Becker
2010-01-13 9:36 ` Wengang Wang
2010-01-13 19:14 ` Sunil Mushran
2010-01-14 1:01 ` Joel Becker
2010-01-14 4:12 ` Wengang Wang
2010-01-21 15:30 ` David Teigland
2010-01-21 15:41 ` Sunil Mushran
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20100112194713.GD24645@redhat.com \
--to=teigland@redhat.com \
--cc=ocfs2-devel@oss.oracle.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.