From mboxrd@z Thu Jan 1 00:00:00 1970 From: Nikola Ciprich Date: Sun, 20 Mar 2011 20:01:16 +0100 Subject: [Cluster-devel] 2.6.37 GFS/CLVM/DLM trouble II Message-ID: <20110320190116.GA29048@nik-comp.lan> List-Id: To: cluster-devel.redhat.com MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hello Stephen et al, some time ago, I reported GFS2 hangs. You asked me to obtain DLM lock dumps, I weren't able to reproduce till now. Today, the on my testing machine, GFS got stuck again. I also noticed that clustered LVM is also stuck on it, so I guess the problem is somewhere in the DLM code, not GFS. Here are kernel backtraces: [182189.107631] INFO: task clvmd:17723 blocked for more than 120 seconds. [182189.107633] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [182189.107634] clvmd D ffffffff8140a4c0 0 17723 1 0x00000000 [182189.107637] ffff8800853c1ca0 0000000000000086 0000000000000000 00000000000116c0 [182189.107641] ffff88013b7348d8 0000000000000001 ffff88013b734530 ffff88013fcd0000 [182189.107644] ffff8800853c1fd8 0000000000000001 0000000001c225b8 ffff8800853c1c98 [182189.107647] Call Trace: [182189.107651] [] ? get_page_from_freelist+0x3b5/0x510 [182189.107654] [] ? get_parent_ip+0x11/0x50 [182189.107656] [] ? get_parent_ip+0x11/0x50 [182189.107659] [] rwsem_down_failed_common+0xb5/0x130 [182189.107663] [] rwsem_down_read_failed+0x15/0x17 [182189.107665] [] call_rwsem_down_read_failed+0x14/0x30 [182189.107668] [] ? down_read+0x2d/0x40 [182189.107673] [] dlm_user_request+0x42/0x260 [dlm] [182189.107676] [] ? get_parent_ip+0x11/0x50 [182189.107679] [] ? kmem_cache_alloc_notrace+0x9e/0xc0 [182189.107684] [] device_write+0x684/0x880 [dlm] [182189.107687] [] ? security_file_permission+0x1e/0x90 [182189.107689] [] ? rw_verify_area+0x74/0xf0 [182189.107691] [] vfs_write+0xc9/0x190 [182189.107694] [] sys_write+0x50/0x90 [182189.107697] [] system_call_fastpath+0x16/0x1b [182189.107705] INFO: task gfs2_quotad:22599 blocked for more than 120 seconds. [182189.107706] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [182189.107707] gfs2_quotad D ffffffff8140a4c0 0 22599 2 0x00000000 [182189.107711] ffff880113d0ba88 0000000000000046 00000000000116c0 00000000000116c0 [182189.107714] ffff8801141bb1c8 0000000000000002 ffff8801141bae20 ffff88013fcd5c40 [182189.107717] ffff880113d0bfd8 ffff880113d0b9b0 0000000081046cd4 ffff88013fcd0000 [182189.107720] Call Trace: [182189.107723] [] ? get_parent_ip+0x11/0x50 [182189.107726] [] ? get_parent_ip+0x11/0x50 [182189.107729] [] rwsem_down_failed_common+0xb5/0x130 [182189.107731] [] ? cpuacct_charge+0x61/0x70 [182189.107734] [] rwsem_down_read_failed+0x15/0x17 [182189.107737] [] call_rwsem_down_read_failed+0x14/0x30 [182189.107740] [] ? down_read+0x2d/0x40 [182189.107745] [] dlm_lock+0x59/0x180 [dlm] [182189.107747] [] ? update_curr+0xb2/0x170 [182189.107750] [] ? hrtick_update+0x2f/0x40 [182189.107760] [] gdlm_lock+0xd3/0x120 [gfs2] [182189.107769] [] ? gdlm_ast+0x0/0x160 [gfs2] [182189.107777] [] ? gdlm_bast+0x0/0x50 [gfs2] [182189.107783] [] do_xmote+0x18c/0x280 [gfs2] [182189.107789] [] run_queue+0x91/0x260 [gfs2] [182189.107796] [] gfs2_glock_nq+0xc3/0x3a0 [gfs2] [182189.107804] [] gfs2_statfs_sync+0x59/0x1a0 [gfs2] [182189.107812] [] ? gfs2_statfs_sync+0x51/0x1a0 [gfs2] [182189.107815] [] ? sub_preempt_count+0x9d/0xd0 [182189.107823] [] quotad_check_timeo+0x57/0x90 [gfs2] [182189.107831] [] gfs2_quotad+0x207/0x240 [gfs2] [182189.107834] [] ? autoremove_wake_function+0x0/0x40 [182189.107837] [] ? _raw_spin_unlock_irqrestore+0x1d/0x50 [182189.107846] [] ? gfs2_quotad+0x0/0x240 [gfs2] [182189.107848] [] kthread+0x96/0xa0 [182189.107851] [] kernel_thread_helper+0x4/0x10 [182189.107854] [] ? kthread+0x0/0xa0 [182189.107857] [] ? kernel_thread_helper+0x0/0x10 and here debugfs DLM lock dumps: [root at vbox5 pcmk:lvs]# cat glocks G: s:EX n:2/20188 f:yIq t:EX d:EX/0 a:1 r:3 I: n:22/131464 t:8 f:0x00 d:0x00000000 s:1434 G: s:EX n:2/25d24 f:yIq t:EX d:EX/0 a:1 r:3 I: n:25/154916 t:8 f:0x00 d:0x00000000 s:1434 G: s:SH n:5/102b7 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/3017c f:yIq t:EX d:EX/0 a:1 r:3 I: n:101/196988 t:8 f:0x00 d:0x00000000 s:941 G: s:SH n:5/102b8 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/20185 f:yIq t:EX d:EX/0 a:1 r:3 I: n:19/131461 t:8 f:0x00 d:0x00000000 s:1434 G: s:SH n:5/20189 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:2/18 f:Iq t:SH d:EX/0 a:0 r:3 I: n:3/24 t:4 f:0x00 d:0x00000201 s:3864 G: s:UN n:2/25d0c f: t:UN d:EX/0 a:0 r:2 G: s:EX n:2/2017a f:yIq t:EX d:EX/0 a:1 r:3 I: n:8/131450 t:8 f:0x00 d:0x00000000 s:1822 G: s:EX n:2/2018b f:yIq t:EX d:EX/0 a:1 r:3 I: n:25/131467 t:8 f:0x00 d:0x00000000 s:1434 G: s:EX n:2/3017b f:yIq t:EX d:EX/0 a:1 r:3 I: n:75/196987 t:8 f:0x00 d:0x00000000 s:1170 G: s:SH n:5/3017f f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/20180 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/25d32 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/20184 f:yIq t:EX d:EX/0 a:1 r:3 I: n:18/131460 t:8 f:0x00 d:0x00000000 s:1434 G: s:SH n:5/2018b f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/2018a f:yIq t:EX d:EX/0 a:1 r:3 I: n:24/131466 t:8 f:0x00 d:0x00000000 s:1434 G: s:SH n:5/10839 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:1/2 f:Iq t:SH d:EX/0 a:0 r:3 G: s:UN n:2/102ab f:lIq t:EX d:EX/0 a:0 r:4 H: s:EX f:cW e:0 p:22599 [gfs2_quotad] gfs2_statfs_sync+0x51/0x1a0 [gfs2] G: s:EX n:2/1053a f:yIq t:EX d:EX/0 a:1 r:3 I: n:3/66874 t:8 f:0x00 d:0x00000000 s:3126995 G: s:SH n:5/25d30 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/25d30 f:yIq t:EX d:EX/0 a:1 r:3 I: n:37/154928 t:8 f:0x00 d:0x00000000 s:1434 G: s:SH n:5/25d38 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/102ab f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/25d38 f:yIq t:EX d:EX/0 a:1 r:3 I: n:45/154936 t:8 f:0x00 d:0x00000000 s:1434 G: s:EX n:2/25d26 f:yIq t:EX d:EX/0 a:1 r:3 I: n:27/154918 t:8 f:0x00 d:0x00000000 s:1434 G: s:SH n:2/16 f:Iq t:SH d:EX/0 a:0 r:6 H: s:SH f:H e:0 p:17736 [mc] gfs2_lookupi+0xbc/0x1c0 [gfs2] H: s:EX f:W e:0 p:17711 [flush-253:6] gfs2_write_inode+0x7a/0x170 [gfs2] H: s:SH f:AW e:0 p:18238 [ls] gfs2_getattr+0x89/0xf0 [gfs2] I: n:1/22 t:4 f:0x00 d:0x00000001 s:3864 G: s:EX n:2/20180 f:yIq t:EX d:EX/0 a:1 r:3 I: n:14/131456 t:8 f:0x00 d:0x00000000 s:1434 G: s:SH n:5/3017c f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:4/0 f:Iq t:SH d:EX/0 a:0 r:2 G: s:SH n:5/3017d f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/2017b f:yIq t:EX d:EX/0 a:1 r:3 I: n:9/131451 t:8 f:0x00 d:0x00000000 s:1621 G: s:SH n:5/25d24 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/25d3a f:yIq t:EX d:EX/0 a:1 r:3 I: n:47/154938 t:8 f:0x00 d:0x00000000 s:1434 G: s:EX n:2/10839 f:yIq t:EX d:EX/0 a:1 r:3 I: n:4/67641 t:4 f:0x00 d:0x00000001 s:3864 G: s:EX n:2/25d11 f:yIq t:EX d:EX/0 a:1 r:3 I: n:6/154897 t:8 f:0x00 d:0x00000000 s:392 G: s:SH n:2/19 f:Iq t:SH d:EX/0 a:0 r:4 H: s:SH f:eEcH e:0 p:22575 [(ended)] init_journal+0x63f/0x9d0 [gfs2] I: n:4/25 t:8 f:0x01 d:0x00000200 s:134217728 G: s:EX n:2/25d10 f:yIq t:EX d:EX/0 a:1 r:3 I: n:5/154896 t:8 f:0x00 d:0x00000000 s:1423 G: s:SH n:5/17 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/25d10 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/1083a f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/2017c f:yIq t:EX d:EX/0 a:1 r:3 I: n:10/131452 t:8 f:0x00 d:0x00000000 s:1621 G: s:SH n:5/20186 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/25d2c f:yIq t:EX d:EX/0 a:1 r:3 I: n:33/154924 t:8 f:0x00 d:0x00000000 s:1434 G: s:SH n:5/25d2e f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/25d0f f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/1009d f:Iq t:SH d:EX/0 a:0 r:2 G: s:SH n:5/2017c f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/102b9 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/2018a f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/102ac f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:UN n:2/102ac f: t:UN d:EX/0 a:0 r:2 G: s:SH n:5/20185 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:2/1083a f:Iq t:SH d:EX/0 a:0 r:3 I: n:5/67642 t:4 f:0x00 d:0x00000001 s:3864 G: s:SH n:5/2017f f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/805b f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:UN n:2/805b f:Iq t:UN d:EX/0 a:0 r:2 G: s:SH n:5/25d0e f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/3017d f:yIq t:EX d:EX/0 a:1 r:3 I: n:103/196989 t:8 f:0x00 d:0x00000000 s:1065 G: s:EX n:2/20187 f:yIq t:EX d:EX/0 a:1 r:3 I: n:21/131463 t:8 f:0x00 d:0x00000000 s:1434 G: s:EX n:2/20186 f:yIq t:EX d:EX/0 a:1 r:3 I: n:20/131462 t:8 f:0x00 d:0x00000000 s:1434 G: s:SH n:5/3017b f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/2017b f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/25d2a f:yIq t:EX d:EX/0 a:1 r:3 I: n:31/154922 t:8 f:0x00 d:0x00000000 s:1434 G: s:SH n:5/1053a f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/20181 f:yIq t:EX d:EX/0 a:1 r:3 I: n:15/131457 t:8 f:0x00 d:0x00000000 s:1434 G: s:EX n:2/25d2e f:yIq t:EX d:EX/0 a:1 r:3 I: n:35/154926 t:8 f:0x00 d:0x00000000 s:1434 G: s:EX n:2/30179 f:yIq t:EX d:EX/0 a:1 r:3 I: n:73/196985 t:8 f:0x00 d:0x00000000 s:1084 G: s:UN n:2/102b7 f: t:UN d:EX/0 a:0 r:2 G: s:EX n:2/2017f f:yIq t:EX d:EX/0 a:1 r:3 I: n:13/131455 t:8 f:0x00 d:0x00000000 s:1434 G: s:SH n:2/102b8 f:Iq t:SH d:EX/0 a:0 r:3 I: n:1/66232 t:4 f:0x00 d:0x00000001 s:3864 G: s:SH n:1/1 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:eEH e:0 p:22575 [(ended)] gfs2_glock_nq_num+0x62/0x90 [gfs2] G: s:EX n:2/2017d f:yIq t:EX d:EX/0 a:1 r:3 I: n:11/131453 t:8 f:0x00 d:0x00000000 s:1434 G: s:EX n:2/20189 f:yIq t:EX d:EX/0 a:1 r:3 I: n:23/131465 t:8 f:0x00 d:0x00000000 s:1434 G: s:EX n:2/25d34 f:yIq t:EX d:EX/0 a:1 r:3 I: n:41/154932 t:8 f:0x00 d:0x00000000 s:1434 G: s:EX n:2/25d0f f:yIq t:EX d:EX/0 a:1 r:3 I: n:4/154895 t:10 f:0x00 d:0x00000000 s:10 G: s:EX n:2/25d28 f:yIq t:EX d:EX/0 a:1 r:3 I: n:29/154920 t:8 f:0x00 d:0x00000000 s:1434 G: s:EX n:2/20182 f:yIq t:EX d:EX/0 a:1 r:3 I: n:16/131458 t:8 f:0x00 d:0x00000000 s:1434 G: s:SH n:5/20182 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/3017f f:yIq t:EX d:EX/0 a:1 r:3 I: n:85/196991 t:8 f:0x00 d:0x00000000 s:1102 G: s:SH n:5/2017a f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/25d2c f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:UN n:1/3 f: t:UN d:EX/0 a:0 r:2 G: s:SH n:5/2017d f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/20183 f:yIq t:EX d:EX/0 a:1 r:3 I: n:17/131459 t:8 f:0x00 d:0x00000000 s:1434 G: s:SH n:2/1009d f:Iq t:SH d:EX/0 a:0 r:2 G: s:EX n:2/100a0 f:Iq t:EX d:EX/0 a:0 r:4 H: s:EX f:H e:0 p:22575 [(ended)] init_per_node+0x181/0x250 [gfs2] I: n:9/65696 t:8 f:0x00 d:0x00000200 s:1048576 G: s:EX n:2/25d0e f:yIq t:EX d:EX/0 a:1 r:3 I: n:3/154894 t:4 f:0x00 d:0x00000001 s:3864 G: s:SH n:5/2017e f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/25d26 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:2/17 f:Iq t:SH d:EX/0 a:0 r:3 I: n:2/23 t:4 f:0x00 d:0x00000201 s:3864 G: s:SH n:5/25d11 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/1009f f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:9/0 f:Iq t:EX d:EX/0 a:0 r:3 H: s:EX f:eH e:0 p:22575 [(ended)] gfs2_glock_nq_num+0x62/0x90 [gfs2] G: s:SH n:5/3017e f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/20184 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/20187 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/16 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/25d32 f:yIq t:EX d:EX/0 a:1 r:3 I: n:39/154930 t:8 f:0x00 d:0x00000000 s:1434 G: s:SH n:5/25d34 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/25d36 f:yIq t:EX d:EX/0 a:1 r:3 I: n:43/154934 t:8 f:0x00 d:0x00000000 s:1434 G: s:SH n:5/20183 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/1009f f:Iq t:EX d:EX/0 a:0 r:4 H: s:EX f:H e:0 p:22575 [(ended)] init_per_node+0x14e/0x250 [gfs2] I: n:8/65695 t:8 f:0x00 d:0x00000201 s:24 G: s:SH n:5/30179 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/25d3a f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:UN n:5/25d0c f:lq t:SH d:EX/0 a:0 r:4 H: s:SH f:EW e:0 p:17736 [mc] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/18 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/25d28 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/100a0 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/25d36 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/20181 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/102b9 f:yIq t:EX d:EX/0 a:1 r:3 I: n:2/66233 t:8 f:0x00 d:0x00000000 s:2612512 G: s:EX n:2/2017e f:yIq t:EX d:EX/0 a:1 r:3 I: n:12/131454 t:8 f:0x00 d:0x00000000 s:1434 G: s:SH n:5/20188 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:EX n:2/3017e f:yIq t:EX d:EX/0 a:1 r:3 I: n:84/196990 t:8 f:0x00 d:0x00000000 s:1115 G: s:SH n:5/19 f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] G: s:SH n:5/25d2a f:Iq t:SH d:EX/0 a:0 r:3 H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2] The machine is SMP x86_64 running 2.6.37.4 now. DLM, CLVMD as well as GFS is handled by corosync/pacemaker cluster. Could somebody please help me to debug it? I can keep the machine in hung state for some time as it's testing box... Thanks a lot in advance! with best regards nik -- ------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis at linuxbox.cz -------------------------------------