From: Steven Whitehouse <swhiteho@redhat.com>
To: cluster-devel.redhat.com
Subject: [Cluster-devel] 2.6.37 GFS/CLVM/DLM trouble II
Date: Mon, 21 Mar 2011 09:50:36 +0000 [thread overview]
Message-ID: <1300701036.2568.6.camel@dolmen> (raw)
In-Reply-To: <20110320190116.GA29048@nik-comp.lan>
Hi,
On Sun, 2011-03-20 at 20:01 +0100, Nikola Ciprich wrote:
> Hello Stephen et al,
>
> some time ago, I reported GFS2 hangs. You asked me to obtain DLM lock
> dumps, I weren't able to reproduce till now.
> Today, the on my testing machine, GFS got stuck again. I also noticed
> that clustered LVM is also stuck on it, so I guess the problem is
> somewhere in the DLM code, not GFS.
>
> Here are kernel backtraces:
>
> [182189.107631] INFO: task clvmd:17723 blocked for more than 120
> seconds.
> [182189.107633] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [182189.107634] clvmd D ffffffff8140a4c0 0 17723 1
> 0x00000000
> [182189.107637] ffff8800853c1ca0 0000000000000086 0000000000000000
> 00000000000116c0
> [182189.107641] ffff88013b7348d8 0000000000000001 ffff88013b734530
> ffff88013fcd0000
> [182189.107644] ffff8800853c1fd8 0000000000000001 0000000001c225b8
> ffff8800853c1c98
> [182189.107647] Call Trace:
> [182189.107651] [<ffffffff810d5025>] ?
> get_page_from_freelist+0x3b5/0x510
> [182189.107654] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107656] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107659] [<ffffffff8136d205>]
> rwsem_down_failed_common+0xb5/0x130
> [182189.107663] [<ffffffff8136d2b5>] rwsem_down_read_failed+0x15/0x17
> [182189.107665] [<ffffffff811d4c44>]
> call_rwsem_down_read_failed+0x14/0x30
> [182189.107668] [<ffffffff8136c65d>] ? down_read+0x2d/0x40
> [182189.107673] [<ffffffffa0548aa2>] dlm_user_request+0x42/0x260 [dlm]
> [182189.107676] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107679] [<ffffffff8110e23e>] ?
> kmem_cache_alloc_notrace+0x9e/0xc0
> [182189.107684] [<ffffffffa0551b04>] device_write+0x684/0x880 [dlm]
> [182189.107687] [<ffffffff811a9cde>] ?
> security_file_permission+0x1e/0x90
> [182189.107689] [<ffffffff8111a894>] ? rw_verify_area+0x74/0xf0
> [182189.107691] [<ffffffff8111aef9>] vfs_write+0xc9/0x190
> [182189.107694] [<ffffffff8111b640>] sys_write+0x50/0x90
> [182189.107697] [<ffffffff810024fb>] system_call_fastpath+0x16/0x1b
> [182189.107705] INFO: task gfs2_quotad:22599 blocked for more than 120
> seconds.
> [182189.107706] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [182189.107707] gfs2_quotad D ffffffff8140a4c0 0 22599 2
> 0x00000000
> [182189.107711] ffff880113d0ba88 0000000000000046 00000000000116c0
> 00000000000116c0
> [182189.107714] ffff8801141bb1c8 0000000000000002 ffff8801141bae20
> ffff88013fcd5c40
> [182189.107717] ffff880113d0bfd8 ffff880113d0b9b0 0000000081046cd4
> ffff88013fcd0000
> [182189.107720] Call Trace:
> [182189.107723] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107726] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107729] [<ffffffff8136d205>]
> rwsem_down_failed_common+0xb5/0x130
> [182189.107731] [<ffffffff81035fb1>] ? cpuacct_charge+0x61/0x70
> [182189.107734] [<ffffffff8136d2b5>] rwsem_down_read_failed+0x15/0x17
> [182189.107737] [<ffffffff811d4c44>]
> call_rwsem_down_read_failed+0x14/0x30
> [182189.107740] [<ffffffff8136c65d>] ? down_read+0x2d/0x40
> [182189.107745] [<ffffffffa0547039>] dlm_lock+0x59/0x180 [dlm]
> [182189.107747] [<ffffffff81045ae2>] ? update_curr+0xb2/0x170
> [182189.107750] [<ffffffff810374df>] ? hrtick_update+0x2f/0x40
> [182189.107760] [<ffffffffa05885d3>] gdlm_lock+0xd3/0x120 [gfs2]
> [182189.107769] [<ffffffffa05887f0>] ? gdlm_ast+0x0/0x160 [gfs2]
> [182189.107777] [<ffffffffa0588620>] ? gdlm_bast+0x0/0x50 [gfs2]
> [182189.107783] [<ffffffffa056a62c>] do_xmote+0x18c/0x280 [gfs2]
> [182189.107789] [<ffffffffa056a7b1>] run_queue+0x91/0x260 [gfs2]
> [182189.107796] [<ffffffffa056aac3>] gfs2_glock_nq+0xc3/0x3a0 [gfs2]
> [182189.107804] [<ffffffffa0584f49>] gfs2_statfs_sync+0x59/0x1a0 [gfs2]
> [182189.107812] [<ffffffffa0584f41>] ? gfs2_statfs_sync+0x51/0x1a0
> [gfs2]
> [182189.107815] [<ffffffff8103c64d>] ? sub_preempt_count+0x9d/0xd0
> [182189.107823] [<ffffffffa057dbf7>] quotad_check_timeo+0x57/0x90
> [gfs2]
> [182189.107831] [<ffffffffa057f637>] gfs2_quotad+0x207/0x240 [gfs2]
> [182189.107834] [<ffffffff8106b130>] ?
> autoremove_wake_function+0x0/0x40
> [182189.107837] [<ffffffff8136d77d>] ?
> _raw_spin_unlock_irqrestore+0x1d/0x50
> [182189.107846] [<ffffffffa057f430>] ? gfs2_quotad+0x0/0x240 [gfs2]
> [182189.107848] [<ffffffff8106ac06>] kthread+0x96/0xa0
> [182189.107851] [<ffffffff810032d4>] kernel_thread_helper+0x4/0x10
> [182189.107854] [<ffffffff8106ab70>] ? kthread+0x0/0xa0
> [182189.107857] [<ffffffff810032d0>] ? kernel_thread_helper+0x0/0x10
>
So there are two processes, both waiting on an rwsem which is somewhere
in dlm.
> and here debugfs DLM lock dumps:
>
This is a glock dump not a dlm lock dump.
> [root at vbox5 pcmk:lvs]# cat glocks
> G: s:EX n:2/20188 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:22/131464 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/25d24 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:25/154916 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/102b7 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/3017c f:yIq t:EX d:EX/0 a:1 r:3
> I: n:101/196988 t:8 f:0x00 d:0x00000000 s:941
> G: s:SH n:5/102b8 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/20185 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:19/131461 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/20189 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:2/18 f:Iq t:SH d:EX/0 a:0 r:3
> I: n:3/24 t:4 f:0x00 d:0x00000201 s:3864
> G: s:UN n:2/25d0c f: t:UN d:EX/0 a:0 r:2
> G: s:EX n:2/2017a f:yIq t:EX d:EX/0 a:1 r:3
> I: n:8/131450 t:8 f:0x00 d:0x00000000 s:1822
> G: s:EX n:2/2018b f:yIq t:EX d:EX/0 a:1 r:3
> I: n:25/131467 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/3017b f:yIq t:EX d:EX/0 a:1 r:3
> I: n:75/196987 t:8 f:0x00 d:0x00000000 s:1170
> G: s:SH n:5/3017f f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/20180 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d32 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/20184 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:18/131460 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/2018b f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/2018a f:yIq t:EX d:EX/0 a:1 r:3
> I: n:24/131466 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/10839 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:1/2 f:Iq t:SH d:EX/0 a:0 r:3
> G: s:UN n:2/102ab f:lIq t:EX d:EX/0 a:0 r:4
> H: s:EX f:cW e:0 p:22599 [gfs2_quotad] gfs2_statfs_sync+0x51/0x1a0
> [gfs2]
> G: s:EX n:2/1053a f:yIq t:EX d:EX/0 a:1 r:3
> I: n:3/66874 t:8 f:0x00 d:0x00000000 s:3126995
> G: s:SH n:5/25d30 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d30 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:37/154928 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/25d38 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/102ab f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d38 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:45/154936 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/25d26 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:27/154918 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:2/16 f:Iq t:SH d:EX/0 a:0 r:6
> H: s:SH f:H e:0 p:17736 [mc] gfs2_lookupi+0xbc/0x1c0 [gfs2]
> H: s:EX f:W e:0 p:17711 [flush-253:6] gfs2_write_inode+0x7a/0x170
> [gfs2]
> H: s:SH f:AW e:0 p:18238 [ls] gfs2_getattr+0x89/0xf0 [gfs2]
> I: n:1/22 t:4 f:0x00 d:0x00000001 s:3864
> G: s:EX n:2/20180 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:14/131456 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/3017c f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:4/0 f:Iq t:SH d:EX/0 a:0 r:2
> G: s:SH n:5/3017d f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/2017b f:yIq t:EX d:EX/0 a:1 r:3
> I: n:9/131451 t:8 f:0x00 d:0x00000000 s:1621
> G: s:SH n:5/25d24 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d3a f:yIq t:EX d:EX/0 a:1 r:3
> I: n:47/154938 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/10839 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:4/67641 t:4 f:0x00 d:0x00000001 s:3864
> G: s:EX n:2/25d11 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:6/154897 t:8 f:0x00 d:0x00000000 s:392
> G: s:SH n:2/19 f:Iq t:SH d:EX/0 a:0 r:4
> H: s:SH f:eEcH e:0 p:22575 [(ended)] init_journal+0x63f/0x9d0 [gfs2]
> I: n:4/25 t:8 f:0x01 d:0x00000200 s:134217728
> G: s:EX n:2/25d10 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:5/154896 t:8 f:0x00 d:0x00000000 s:1423
> G: s:SH n:5/17 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d10 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/1083a f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/2017c f:yIq t:EX d:EX/0 a:1 r:3
> I: n:10/131452 t:8 f:0x00 d:0x00000000 s:1621
> G: s:SH n:5/20186 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d2c f:yIq t:EX d:EX/0 a:1 r:3
> I: n:33/154924 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/25d2e f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d0f f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/1009d f:Iq t:SH d:EX/0 a:0 r:2
> G: s:SH n:5/2017c f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/102b9 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/2018a f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/102ac f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:UN n:2/102ac f: t:UN d:EX/0 a:0 r:2
> G: s:SH n:5/20185 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:2/1083a f:Iq t:SH d:EX/0 a:0 r:3
> I: n:5/67642 t:4 f:0x00 d:0x00000001 s:3864
> G: s:SH n:5/2017f f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/805b f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:UN n:2/805b f:Iq t:UN d:EX/0 a:0 r:2
> G: s:SH n:5/25d0e f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/3017d f:yIq t:EX d:EX/0 a:1 r:3
> I: n:103/196989 t:8 f:0x00 d:0x00000000 s:1065
> G: s:EX n:2/20187 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:21/131463 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/20186 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:20/131462 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/3017b f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/2017b f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d2a f:yIq t:EX d:EX/0 a:1 r:3
> I: n:31/154922 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/1053a f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/20181 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:15/131457 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/25d2e f:yIq t:EX d:EX/0 a:1 r:3
> I: n:35/154926 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/30179 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:73/196985 t:8 f:0x00 d:0x00000000 s:1084
> G: s:UN n:2/102b7 f: t:UN d:EX/0 a:0 r:2
> G: s:EX n:2/2017f f:yIq t:EX d:EX/0 a:1 r:3
> I: n:13/131455 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:2/102b8 f:Iq t:SH d:EX/0 a:0 r:3
> I: n:1/66232 t:4 f:0x00 d:0x00000001 s:3864
> G: s:SH n:1/1 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:eEH e:0 p:22575 [(ended)] gfs2_glock_nq_num+0x62/0x90 [gfs2]
> G: s:EX n:2/2017d f:yIq t:EX d:EX/0 a:1 r:3
> I: n:11/131453 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/20189 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:23/131465 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/25d34 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:41/154932 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/25d0f f:yIq t:EX d:EX/0 a:1 r:3
> I: n:4/154895 t:10 f:0x00 d:0x00000000 s:10
> G: s:EX n:2/25d28 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:29/154920 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/20182 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:16/131458 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/20182 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/3017f f:yIq t:EX d:EX/0 a:1 r:3
> I: n:85/196991 t:8 f:0x00 d:0x00000000 s:1102
> G: s:SH n:5/2017a f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d2c f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:UN n:1/3 f: t:UN d:EX/0 a:0 r:2
> G: s:SH n:5/2017d f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/20183 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:17/131459 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:2/1009d f:Iq t:SH d:EX/0 a:0 r:2
> G: s:EX n:2/100a0 f:Iq t:EX d:EX/0 a:0 r:4
> H: s:EX f:H e:0 p:22575 [(ended)] init_per_node+0x181/0x250 [gfs2]
> I: n:9/65696 t:8 f:0x00 d:0x00000200 s:1048576
> G: s:EX n:2/25d0e f:yIq t:EX d:EX/0 a:1 r:3
> I: n:3/154894 t:4 f:0x00 d:0x00000001 s:3864
> G: s:SH n:5/2017e f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d26 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:2/17 f:Iq t:SH d:EX/0 a:0 r:3
> I: n:2/23 t:4 f:0x00 d:0x00000201 s:3864
> G: s:SH n:5/25d11 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/1009f f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:9/0 f:Iq t:EX d:EX/0 a:0 r:3
> H: s:EX f:eH e:0 p:22575 [(ended)] gfs2_glock_nq_num+0x62/0x90 [gfs2]
> G: s:SH n:5/3017e f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/20184 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/20187 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/16 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d32 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:39/154930 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/25d34 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d36 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:43/154934 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/20183 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/1009f f:Iq t:EX d:EX/0 a:0 r:4
> H: s:EX f:H e:0 p:22575 [(ended)] init_per_node+0x14e/0x250 [gfs2]
> I: n:8/65695 t:8 f:0x00 d:0x00000201 s:24
> G: s:SH n:5/30179 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d3a f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:UN n:5/25d0c f:lq t:SH d:EX/0 a:0 r:4
> H: s:SH f:EW e:0 p:17736 [mc] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/18 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d28 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/100a0 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d36 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/20181 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/102b9 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:2/66233 t:8 f:0x00 d:0x00000000 s:2612512
> G: s:EX n:2/2017e f:yIq t:EX d:EX/0 a:1 r:3
> I: n:12/131454 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/20188 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/3017e f:yIq t:EX d:EX/0 a:1 r:3
> I: n:84/196990 t:8 f:0x00 d:0x00000000 s:1115
> G: s:SH n:5/19 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d2a f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
>
> The machine is SMP x86_64 running 2.6.37.4 now. DLM, CLVMD as well as
> GFS is handled by corosync/pacemaker cluster.
> Could somebody please help me to debug it? I can keep the machine in
> hung state for some time as it's testing box...
>
> Thanks a lot in advance!
>
> with best regards
>
> nik
>
>
Do you have any log messages relating to recovery? I'm wondering if that
might have failed and be the reason for these messages. It would be
useful to have a dump from gfs_control for example,
Steve.
next prev parent reply other threads:[~2011-03-21 9:50 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-03-20 19:01 [Cluster-devel] 2.6.37 GFS/CLVM/DLM trouble II Nikola Ciprich
2011-03-21 9:50 ` Steven Whitehouse [this message]
[not found] ` <20110321160804.GA2177@nik-comp.lan>
2011-03-21 16:47 ` Steven Whitehouse
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1300701036.2568.6.camel@dolmen \
--to=swhiteho@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.