From mboxrd@z Thu Jan  1 00:00:00 1970
From: Steven Whitehouse <swhiteho@redhat.com>
Date: Mon, 21 Mar 2011 09:50:36 +0000
Subject: [Cluster-devel] 2.6.37 GFS/CLVM/DLM trouble II
In-Reply-To: <20110320190116.GA29048@nik-comp.lan>
References: <20110320190116.GA29048@nik-comp.lan>
Message-ID: <1300701036.2568.6.camel@dolmen>
List-Id: <cluster-devel.redhat.com>
To: cluster-devel.redhat.com
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

Hi,

On Sun, 2011-03-20 at 20:01 +0100, Nikola Ciprich wrote:
> Hello Stephen et al,
> 
> some time ago, I reported GFS2 hangs. You asked me to obtain DLM lock
> dumps, I weren't able to reproduce till now.
> Today, the on my testing machine, GFS got stuck again. I also noticed
> that clustered LVM is also stuck on it, so I guess the problem is
> somewhere in the DLM code, not GFS.
> 
> Here are kernel backtraces:
> 
> [182189.107631] INFO: task clvmd:17723 blocked for more than 120
> seconds.
> [182189.107633] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [182189.107634] clvmd         D ffffffff8140a4c0     0 17723      1
> 0x00000000
> [182189.107637]  ffff8800853c1ca0 0000000000000086 0000000000000000
> 00000000000116c0
> [182189.107641]  ffff88013b7348d8 0000000000000001 ffff88013b734530
> ffff88013fcd0000
> [182189.107644]  ffff8800853c1fd8 0000000000000001 0000000001c225b8
> ffff8800853c1c98
> [182189.107647] Call Trace:
> [182189.107651]  [<ffffffff810d5025>] ?
> get_page_from_freelist+0x3b5/0x510
> [182189.107654]  [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107656]  [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107659]  [<ffffffff8136d205>]
> rwsem_down_failed_common+0xb5/0x130
> [182189.107663]  [<ffffffff8136d2b5>] rwsem_down_read_failed+0x15/0x17
> [182189.107665]  [<ffffffff811d4c44>]
> call_rwsem_down_read_failed+0x14/0x30
> [182189.107668]  [<ffffffff8136c65d>] ? down_read+0x2d/0x40
> [182189.107673]  [<ffffffffa0548aa2>] dlm_user_request+0x42/0x260 [dlm]
> [182189.107676]  [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107679]  [<ffffffff8110e23e>] ?
> kmem_cache_alloc_notrace+0x9e/0xc0
> [182189.107684]  [<ffffffffa0551b04>] device_write+0x684/0x880 [dlm]
> [182189.107687]  [<ffffffff811a9cde>] ?
> security_file_permission+0x1e/0x90
> [182189.107689]  [<ffffffff8111a894>] ? rw_verify_area+0x74/0xf0
> [182189.107691]  [<ffffffff8111aef9>] vfs_write+0xc9/0x190
> [182189.107694]  [<ffffffff8111b640>] sys_write+0x50/0x90
> [182189.107697]  [<ffffffff810024fb>] system_call_fastpath+0x16/0x1b
> [182189.107705] INFO: task gfs2_quotad:22599 blocked for more than 120
> seconds.
> [182189.107706] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [182189.107707] gfs2_quotad   D ffffffff8140a4c0     0 22599      2
> 0x00000000
> [182189.107711]  ffff880113d0ba88 0000000000000046 00000000000116c0
> 00000000000116c0
> [182189.107714]  ffff8801141bb1c8 0000000000000002 ffff8801141bae20
> ffff88013fcd5c40
> [182189.107717]  ffff880113d0bfd8 ffff880113d0b9b0 0000000081046cd4
> ffff88013fcd0000
> [182189.107720] Call Trace:
> [182189.107723]  [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107726]  [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107729]  [<ffffffff8136d205>]
> rwsem_down_failed_common+0xb5/0x130
> [182189.107731]  [<ffffffff81035fb1>] ? cpuacct_charge+0x61/0x70
> [182189.107734]  [<ffffffff8136d2b5>] rwsem_down_read_failed+0x15/0x17
> [182189.107737]  [<ffffffff811d4c44>]
> call_rwsem_down_read_failed+0x14/0x30
> [182189.107740]  [<ffffffff8136c65d>] ? down_read+0x2d/0x40
> [182189.107745]  [<ffffffffa0547039>] dlm_lock+0x59/0x180 [dlm]
> [182189.107747]  [<ffffffff81045ae2>] ? update_curr+0xb2/0x170
> [182189.107750]  [<ffffffff810374df>] ? hrtick_update+0x2f/0x40
> [182189.107760]  [<ffffffffa05885d3>] gdlm_lock+0xd3/0x120 [gfs2]
> [182189.107769]  [<ffffffffa05887f0>] ? gdlm_ast+0x0/0x160 [gfs2]
> [182189.107777]  [<ffffffffa0588620>] ? gdlm_bast+0x0/0x50 [gfs2]
> [182189.107783]  [<ffffffffa056a62c>] do_xmote+0x18c/0x280 [gfs2]
> [182189.107789]  [<ffffffffa056a7b1>] run_queue+0x91/0x260 [gfs2]
> [182189.107796]  [<ffffffffa056aac3>] gfs2_glock_nq+0xc3/0x3a0 [gfs2]
> [182189.107804]  [<ffffffffa0584f49>] gfs2_statfs_sync+0x59/0x1a0 [gfs2]
> [182189.107812]  [<ffffffffa0584f41>] ? gfs2_statfs_sync+0x51/0x1a0
> [gfs2]
> [182189.107815]  [<ffffffff8103c64d>] ? sub_preempt_count+0x9d/0xd0
> [182189.107823]  [<ffffffffa057dbf7>] quotad_check_timeo+0x57/0x90
> [gfs2]
> [182189.107831]  [<ffffffffa057f637>] gfs2_quotad+0x207/0x240 [gfs2]
> [182189.107834]  [<ffffffff8106b130>] ?
> autoremove_wake_function+0x0/0x40
> [182189.107837]  [<ffffffff8136d77d>] ?
> _raw_spin_unlock_irqrestore+0x1d/0x50
> [182189.107846]  [<ffffffffa057f430>] ? gfs2_quotad+0x0/0x240 [gfs2]
> [182189.107848]  [<ffffffff8106ac06>] kthread+0x96/0xa0
> [182189.107851]  [<ffffffff810032d4>] kernel_thread_helper+0x4/0x10
> [182189.107854]  [<ffffffff8106ab70>] ? kthread+0x0/0xa0
> [182189.107857]  [<ffffffff810032d0>] ? kernel_thread_helper+0x0/0x10
> 
So there are two processes, both waiting on an rwsem which is somewhere
in dlm.

> and here debugfs DLM lock dumps:
> 
This is a glock dump not a dlm lock dump.

> [root at vbox5 pcmk:lvs]# cat glocks 
> G:  s:EX n:2/20188 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:22/131464 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:EX n:2/25d24 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:25/154916 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:SH n:5/102b7 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/3017c f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:101/196988 t:8 f:0x00 d:0x00000000 s:941
> G:  s:SH n:5/102b8 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/20185 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:19/131461 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:SH n:5/20189 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:2/18 f:Iq t:SH d:EX/0 a:0 r:3
>  I: n:3/24 t:4 f:0x00 d:0x00000201 s:3864
> G:  s:UN n:2/25d0c f: t:UN d:EX/0 a:0 r:2
> G:  s:EX n:2/2017a f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:8/131450 t:8 f:0x00 d:0x00000000 s:1822
> G:  s:EX n:2/2018b f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:25/131467 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:EX n:2/3017b f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:75/196987 t:8 f:0x00 d:0x00000000 s:1170
> G:  s:SH n:5/3017f f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/20180 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/25d32 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/20184 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:18/131460 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:SH n:5/2018b f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/2018a f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:24/131466 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:SH n:5/10839 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:1/2 f:Iq t:SH d:EX/0 a:0 r:3
> G:  s:UN n:2/102ab f:lIq t:EX d:EX/0 a:0 r:4
>  H: s:EX f:cW e:0 p:22599 [gfs2_quotad] gfs2_statfs_sync+0x51/0x1a0
> [gfs2]
> G:  s:EX n:2/1053a f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:3/66874 t:8 f:0x00 d:0x00000000 s:3126995
> G:  s:SH n:5/25d30 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/25d30 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:37/154928 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:SH n:5/25d38 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/102ab f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/25d38 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:45/154936 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:EX n:2/25d26 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:27/154918 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:SH n:2/16 f:Iq t:SH d:EX/0 a:0 r:6
>  H: s:SH f:H e:0 p:17736 [mc] gfs2_lookupi+0xbc/0x1c0 [gfs2]
>  H: s:EX f:W e:0 p:17711 [flush-253:6] gfs2_write_inode+0x7a/0x170
> [gfs2]
>  H: s:SH f:AW e:0 p:18238 [ls] gfs2_getattr+0x89/0xf0 [gfs2]
>  I: n:1/22 t:4 f:0x00 d:0x00000001 s:3864
> G:  s:EX n:2/20180 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:14/131456 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:SH n:5/3017c f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:4/0 f:Iq t:SH d:EX/0 a:0 r:2
> G:  s:SH n:5/3017d f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/2017b f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:9/131451 t:8 f:0x00 d:0x00000000 s:1621
> G:  s:SH n:5/25d24 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/25d3a f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:47/154938 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:EX n:2/10839 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:4/67641 t:4 f:0x00 d:0x00000001 s:3864
> G:  s:EX n:2/25d11 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:6/154897 t:8 f:0x00 d:0x00000000 s:392
> G:  s:SH n:2/19 f:Iq t:SH d:EX/0 a:0 r:4
>  H: s:SH f:eEcH e:0 p:22575 [(ended)] init_journal+0x63f/0x9d0 [gfs2]
>  I: n:4/25 t:8 f:0x01 d:0x00000200 s:134217728
> G:  s:EX n:2/25d10 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:5/154896 t:8 f:0x00 d:0x00000000 s:1423
> G:  s:SH n:5/17 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/25d10 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/1083a f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/2017c f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:10/131452 t:8 f:0x00 d:0x00000000 s:1621
> G:  s:SH n:5/20186 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/25d2c f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:33/154924 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:SH n:5/25d2e f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/25d0f f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/1009d f:Iq t:SH d:EX/0 a:0 r:2
> G:  s:SH n:5/2017c f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/102b9 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/2018a f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/102ac f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:UN n:2/102ac f: t:UN d:EX/0 a:0 r:2
> G:  s:SH n:5/20185 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:2/1083a f:Iq t:SH d:EX/0 a:0 r:3
>  I: n:5/67642 t:4 f:0x00 d:0x00000001 s:3864
> G:  s:SH n:5/2017f f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/805b f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:UN n:2/805b f:Iq t:UN d:EX/0 a:0 r:2
> G:  s:SH n:5/25d0e f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/3017d f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:103/196989 t:8 f:0x00 d:0x00000000 s:1065
> G:  s:EX n:2/20187 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:21/131463 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:EX n:2/20186 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:20/131462 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:SH n:5/3017b f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/2017b f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/25d2a f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:31/154922 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:SH n:5/1053a f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/20181 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:15/131457 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:EX n:2/25d2e f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:35/154926 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:EX n:2/30179 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:73/196985 t:8 f:0x00 d:0x00000000 s:1084
> G:  s:UN n:2/102b7 f: t:UN d:EX/0 a:0 r:2
> G:  s:EX n:2/2017f f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:13/131455 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:SH n:2/102b8 f:Iq t:SH d:EX/0 a:0 r:3
>  I: n:1/66232 t:4 f:0x00 d:0x00000001 s:3864
> G:  s:SH n:1/1 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:eEH e:0 p:22575 [(ended)] gfs2_glock_nq_num+0x62/0x90 [gfs2]
> G:  s:EX n:2/2017d f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:11/131453 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:EX n:2/20189 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:23/131465 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:EX n:2/25d34 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:41/154932 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:EX n:2/25d0f f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:4/154895 t:10 f:0x00 d:0x00000000 s:10
> G:  s:EX n:2/25d28 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:29/154920 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:EX n:2/20182 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:16/131458 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:SH n:5/20182 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/3017f f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:85/196991 t:8 f:0x00 d:0x00000000 s:1102
> G:  s:SH n:5/2017a f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/25d2c f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:UN n:1/3 f: t:UN d:EX/0 a:0 r:2
> G:  s:SH n:5/2017d f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/20183 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:17/131459 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:SH n:2/1009d f:Iq t:SH d:EX/0 a:0 r:2
> G:  s:EX n:2/100a0 f:Iq t:EX d:EX/0 a:0 r:4
>  H: s:EX f:H e:0 p:22575 [(ended)] init_per_node+0x181/0x250 [gfs2]
>  I: n:9/65696 t:8 f:0x00 d:0x00000200 s:1048576
> G:  s:EX n:2/25d0e f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:3/154894 t:4 f:0x00 d:0x00000001 s:3864
> G:  s:SH n:5/2017e f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/25d26 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:2/17 f:Iq t:SH d:EX/0 a:0 r:3
>  I: n:2/23 t:4 f:0x00 d:0x00000201 s:3864
> G:  s:SH n:5/25d11 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/1009f f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:9/0 f:Iq t:EX d:EX/0 a:0 r:3
>  H: s:EX f:eH e:0 p:22575 [(ended)] gfs2_glock_nq_num+0x62/0x90 [gfs2]
> G:  s:SH n:5/3017e f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/20184 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/20187 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/16 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/25d32 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:39/154930 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:SH n:5/25d34 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/25d36 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:43/154934 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:SH n:5/20183 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/1009f f:Iq t:EX d:EX/0 a:0 r:4
>  H: s:EX f:H e:0 p:22575 [(ended)] init_per_node+0x14e/0x250 [gfs2]
>  I: n:8/65695 t:8 f:0x00 d:0x00000201 s:24
> G:  s:SH n:5/30179 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/25d3a f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:UN n:5/25d0c f:lq t:SH d:EX/0 a:0 r:4
>  H: s:SH f:EW e:0 p:17736 [mc] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/18 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/25d28 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/100a0 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/25d36 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/20181 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/102b9 f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:2/66233 t:8 f:0x00 d:0x00000000 s:2612512
> G:  s:EX n:2/2017e f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:12/131454 t:8 f:0x00 d:0x00000000 s:1434
> G:  s:SH n:5/20188 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:EX n:2/3017e f:yIq t:EX d:EX/0 a:1 r:3
>  I: n:84/196990 t:8 f:0x00 d:0x00000000 s:1115
> G:  s:SH n:5/19 f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G:  s:SH n:5/25d2a f:Iq t:SH d:EX/0 a:0 r:3
>  H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> 
> The machine is SMP x86_64 running 2.6.37.4 now. DLM, CLVMD as well as
> GFS is handled by corosync/pacemaker cluster.
> Could somebody please help me to debug it? I can keep the machine in
> hung state for some time as it's testing box...
> 
> Thanks a lot in advance!
> 
> with best regards
> 
> nik
> 
> 
Do you have any log messages relating to recovery? I'm wondering if that
might have failed and be the reason for these messages. It would be
useful to have a dump from gfs_control for example,

Steve.