* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20. @ 2007-02-08 12:15 Zbyszek Żółkiewski 2007-02-08 12:57 ` Steven Whitehouse 0 siblings, 1 reply; 11+ messages in thread From: Zbyszek Żółkiewski @ 2007-02-08 12:15 UTC (permalink / raw) To: cluster-devel.redhat.com Hi, I have recently setup cluster on Debian systems and i got issues related with kernel panic: systems affected: Debian 4.0 (testing) and Debian 3.1 (r4) gcc: 3.4 and 4.1.2 kernels: 2.6.19 and 2.6.20 cluster from latest cvs command invoked: mount -t gfs2 /dev/sdb1 /mnt/ docs that i followed : http://sources.redhat.com/cgi-bin/cvsweb.cgi/cluster/doc/usage.txt?cvsroot=cluster any clue? here are detailed logs: Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: add member 1 Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: total members 1 error 0 Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: dlm_recover_directory Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: dlm_recover_directory 0 entries Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: recover 1 done: 0 ms Feb 8 13:05:18 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: Joined cluster. Now mounting FS... Feb 8 13:05:21 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: jid=0, already locked for use Feb 8 13:05:21 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: jid=0: Looking at journal... Feb 8 13:05:21 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: jid=0: Done Feb 8 13:05:21 xmpp-alt2 kernel: ------------[ cut here ]------------ Feb 8 13:05:21 xmpp-alt2 kernel: kernel BUG at fs/gfs2/glock.c:738! Feb 8 13:05:21 xmpp-alt2 kernel: invalid opcode: 0000 [#1] Feb 8 13:05:21 xmpp-alt2 kernel: Modules linked in: lock_dlm dlm configfs gfs2 Feb 8 13:05:21 xmpp-alt2 kernel: CPU: 0 Feb 8 13:05:21 xmpp-alt2 kernel: EIP: 0060:[<f890f11e>] Not tainted VLI Feb 8 13:05:21 xmpp-alt2 kernel: EFLAGS: 00000282 (2.6.20-xmpp2 #1) Feb 8 13:05:21 xmpp-alt2 kernel: EIP is at gfs2_glmutex_unlock+0x18/0x1c [gfs2] Feb 8 13:05:21 xmpp-alt2 kernel: eax: f3aa1bbc ebx: f3aa1bec ecx: 00007080 edx: f3aa1bbc Feb 8 13:05:22 xmpp-alt2 kernel: esi: f3aa1b78 edi: f15a3f88 ebp: f15a3f94 esp: f15a3f64 Feb 8 13:05:22 xmpp-alt2 kernel: ds: 007b es: 007b ss: 0068 Feb 8 13:05:22 xmpp-alt2 kernel: Process gfs2_glockd (pid: 7013, ti=f15a2000 task=f3ad9a70 task.ti=f15a2000) Feb 8 13:05:22 xmpp-alt2 kernel: Stack: f89108e5 f34f6000 f34f6364 f8907861 00000000 f3ad9a70 c0123580 f15a3fa0 Feb 8 13:05:22 xmpp-alt2 kernel: f15a3fa0 f15a3fac c010f84f 00000000 00000000 f3ad9a70 c0123580 f15a3fa0 Feb 8 13:05:22 xmpp-alt2 kernel: f15a3fa0 000004c1 f9a6da64 0024d65b f348ddc4 f34f6000 f8907830 fffffffc Feb 8 13:05:22 xmpp-alt2 kernel: Call Trace: Feb 8 13:05:22 xmpp-alt2 kernel: [<f89108e5>] gfs2_reclaim_glock+0x8d/0x8f [gfs2] Feb 8 13:05:22 xmpp-alt2 kernel: [<f8907861>] gfs2_glockd+0x31/0xe4 [gfs2] Feb 8 13:05:22 xmpp-alt2 kernel: [<c0123580>] autoremove_wake_function+0x0/0x43 Feb 8 13:05:22 xmpp-alt2 kernel: [<c010f84f>] __wake_up_common+0x33/0x56 Feb 8 13:05:22 xmpp-alt2 kernel: [<c0123580>] autoremove_wake_function+0x0/0x43 Feb 8 13:05:22 xmpp-alt2 kernel: [<f8907830>] gfs2_glockd+0x0/0xe4 [gfs2] Feb 8 13:05:22 xmpp-alt2 kernel: [<c012320b>] kthread+0x9d/0xce Feb 8 13:05:22 xmpp-alt2 kernel: [<c012316e>] kthread+0x0/0xce Feb 8 13:05:22 xmpp-alt2 kernel: [<c01034df>] kernel_thread_helper+0x7/0x10 Feb 8 13:05:22 xmpp-alt2 kernel: ======================= Feb 8 13:05:22 xmpp-alt2 kernel: Code: c3 65 a1 08 00 00 00 89 42 24 8b 04 24 89 42 28 89 c8 c3 0f ba 70 08 01 c7 40 24 00 00 00 00 c7 40 28 00 00 00 00 e8 7a fe ff ff <0f> 0b eb fe 55 89 d5 57 89 c7 56 31 f6 53 8d 58 3c 83 ec 04 8b Feb 8 13:05:22 xmpp-alt2 kernel: EIP: [<f890f11e>] gfs2_glmutex_unlock+0x18/0x1c [gfs2] SS:ESP 0068:f15a3f64 Feb 8 13:05:33 xmpp-alt2 kernel: <0>------------[ cut here ]------------ Feb 8 13:05:33 xmpp-alt2 kernel: kernel BUG at fs/gfs2/glock.c:738! Feb 8 13:05:33 xmpp-alt2 kernel: invalid opcode: 0000 [#2] Feb 8 13:05:33 xmpp-alt2 kernel: Modules linked in: lock_dlm dlm configfs gfs2 Feb 8 13:05:33 xmpp-alt2 kernel: CPU: 0 Feb 8 13:05:33 xmpp-alt2 kernel: EIP: 0060:[<f890f11e>] Not tainted VLI Feb 8 13:05:33 xmpp-alt2 kernel: EFLAGS: 00000282 (2.6.20-xmpp2 #1) Feb 8 13:05:34 xmpp-alt2 kernel: EIP is at gfs2_glmutex_unlock+0x18/0x1c [gfs2] Feb 8 13:05:34 xmpp-alt2 kernel: eax: f3aa1bbc ebx: f3aa1b78 ecx: 00007080 edx: f3aa1bbc Feb 8 13:05:34 xmpp-alt2 kernel: esi: f34f6000 edi: f3aa1b78 ebp: 00000001 esp: f34f5f98 Feb 8 13:05:34 xmpp-alt2 kernel: ds: 007b es: 007b ss: 0068 Feb 8 13:05:34 xmpp-alt2 kernel: Process gfs2_scand (pid: 7012, ti=f34f4000 task=f3bef030 task.ti=f34f4000) Feb 8 13:05:34 xmpp-alt2 kernel: Stack: f8910940 f8910942 000002be f34f6000 f8907800 fffffffc f89109aa f34f6000 Feb 8 13:05:34 xmpp-alt2 kernel: f34f6000 f890780c f348ddc4 c012320b 00000001 ffffffff ffffffff c012316e Feb 8 13:05:34 xmpp-alt2 kernel: 00000000 00000000 00000000 c01034df f348ddbc 00000000 00000000 00000000 Feb 8 13:05:34 xmpp-alt2 kernel: Call Trace: Feb 8 13:05:34 xmpp-alt2 kernel: [<f8910940>] examine_bucket+0x59/0x5b [gfs2] Feb 8 13:05:34 xmpp-alt2 kernel: [<f8910942>] scan_glock+0x0/0x51 [gfs2] Feb 8 13:05:34 xmpp-alt2 kernel: [<f8907800>] gfs2_scand+0x0/0x30 [gfs2] Feb 8 13:05:34 xmpp-alt2 kernel: [<f89109aa>] gfs2_scand_internal+0x17/0x22 [gfs2] Feb 8 13:05:34 xmpp-alt2 kernel: [<f890780c>] gfs2_scand+0xc/0x30 [gfs2] Feb 8 13:05:34 xmpp-alt2 kernel: [<c012320b>] kthread+0x9d/0xce Feb 8 13:05:34 xmpp-alt2 kernel: [<c012316e>] kthread+0x0/0xce Feb 8 13:05:34 xmpp-alt2 kernel: [<c01034df>] kernel_thread_helper+0x7/0x10 Feb 8 13:05:34 xmpp-alt2 kernel: ======================= Feb 8 13:05:34 xmpp-alt2 kernel: Code: c3 65 a1 08 00 00 00 89 42 24 8b 04 24 89 42 28 89 c8 c3 0f ba 70 08 01 c7 40 24 00 00 00 00 c7 40 28 00 00 00 00 e8 7a fe ff ff <0f> 0b eb fe 55 89 d5 57 89 c7 56 31 f6 53 8d 58 3c 83 ec 04 8b Feb 8 13:05:34 xmpp-alt2 kernel: EIP: [<f890f11e>] gfs2_glmutex_unlock+0x18/0x1c [gfs2] SS:ESP 0068:f34f5f98 -- pozdrawiam, Zbyszek ???kiewski -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20070208/9fb2115b/attachment.htm> ^ permalink raw reply [flat|nested] 11+ messages in thread
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20. 2007-02-08 12:15 [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20 Zbyszek Żółkiewski @ 2007-02-08 12:57 ` Steven Whitehouse [not found] ` <f7ca4caa0702080502m723d4bc3r3dbdb9302a942bff@mail.gmail.com> 0 siblings, 1 reply; 11+ messages in thread From: Steven Whitehouse @ 2007-02-08 12:57 UTC (permalink / raw) To: cluster-devel.redhat.com Hi, On Thu, 2007-02-08 at 13:15 +0100, Zbyszek ???kiewski wrote: > Hi, > > I have recently setup cluster on Debian systems and i got issues > related with kernel panic: > systems affected: Debian 4.0 (testing) and Debian 3.1 (r4) > gcc: 3.4 and 4.1.2 > kernels: 2.6.19 and 2.6.20 > cluster from latest cvs > > command invoked: mount -t gfs2 /dev/sdb1 /mnt/ > docs that i followed : > http://sources.redhat.com/cgi-bin/cvsweb.cgi/cluster/doc/usage.txt?cvsroot=cluster > > any clue? > It looks like the mount was successful, but that the problem occurred right after mount. Just as a sanity check does the same thing happen if you use lock_nolock? There have been a number of bug fixes since 2.6.20 which went into Linus' kernel yesterday, so you might want to try the latest upstream kernel, but I don't recognise this problem as being something we've seen before, Steve. > here are detailed logs: > > Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: add member 1 > Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: total members 1 error 0 > Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: dlm_recover_directory > Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: dlm_recover_directory 0 > entries > Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: recover 1 done: 0 ms > Feb 8 13:05:18 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: Joined > cluster. Now mounting FS... > Feb 8 13:05:21 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: jid=0, > already locked for use > Feb 8 13:05:21 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: jid=0: > Looking at journal... > Feb 8 13:05:21 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2: test.0: jid=0: > Done > Feb 8 13:05:21 xmpp-alt2 kernel: ------------[ cut here ]------------ > Feb 8 13:05:21 xmpp-alt2 kernel: kernel BUG at fs/gfs2/glock.c:738! > Feb 8 13:05:21 xmpp-alt2 kernel: invalid opcode: 0000 [#1] > Feb 8 13:05:21 xmpp-alt2 kernel: Modules linked in: lock_dlm dlm > configfs gfs2 > Feb 8 13:05:21 xmpp-alt2 kernel: CPU: 0 > Feb 8 13:05:21 xmpp-alt2 kernel: EIP: 0060:[<f890f11e>] Not > tainted VLI > Feb 8 13:05:21 xmpp-alt2 kernel: EFLAGS: 00000282 (2.6.20-xmpp2 #1) > Feb 8 13:05:21 xmpp-alt2 kernel: EIP is at gfs2_glmutex_unlock > +0x18/0x1c [gfs2] > Feb 8 13:05:21 xmpp-alt2 kernel: eax: f3aa1bbc ebx: f3aa1bec ecx: > 00007080 edx: f3aa1bbc > Feb 8 13:05:22 xmpp-alt2 kernel: esi: f3aa1b78 edi: f15a3f88 ebp: > f15a3f94 esp: f15a3f64 > Feb 8 13:05:22 xmpp-alt2 kernel: ds: 007b es: 007b ss: 0068 > Feb 8 13:05:22 xmpp-alt2 kernel: Process gfs2_glockd (pid: 7013, > ti=f15a2000 task=f3ad9a70 task.ti=f15a2000) > Feb 8 13:05:22 xmpp-alt2 kernel: Stack: f89108e5 f34f6000 f34f6364 > f8907861 00000000 f3ad9a70 c0123580 f15a3fa0 > Feb 8 13:05:22 xmpp-alt2 kernel: f15a3fa0 f15a3fac c010f84f > 00000000 00000000 f3ad9a70 c0123580 f15a3fa0 > Feb 8 13:05:22 xmpp-alt2 kernel: f15a3fa0 000004c1 f9a6da64 > 0024d65b f348ddc4 f34f6000 f8907830 fffffffc > Feb 8 13:05:22 xmpp-alt2 kernel: Call Trace: > Feb 8 13:05:22 xmpp-alt2 kernel: [<f89108e5>] gfs2_reclaim_glock > +0x8d/0x8f [gfs2] > Feb 8 13:05:22 xmpp-alt2 kernel: [<f8907861>] gfs2_glockd+0x31/0xe4 > [gfs2] > Feb 8 13:05:22 xmpp-alt2 kernel: [<c0123580>] > autoremove_wake_function+0x0/0x43 > Feb 8 13:05:22 xmpp-alt2 kernel: [<c010f84f>] __wake_up_common > +0x33/0x56 > Feb 8 13:05:22 xmpp-alt2 kernel: [<c0123580>] > autoremove_wake_function+0x0/0x43 > Feb 8 13:05:22 xmpp-alt2 kernel: [<f8907830>] gfs2_glockd+0x0/0xe4 > [gfs2] > Feb 8 13:05:22 xmpp-alt2 kernel: [<c012320b>] kthread+0x9d/0xce > Feb 8 13:05:22 xmpp-alt2 kernel: [<c012316e>] kthread+0x0/0xce > Feb 8 13:05:22 xmpp-alt2 kernel: [<c01034df>] kernel_thread_helper > +0x7/0x10 > Feb 8 13:05:22 xmpp-alt2 kernel: ======================= > Feb 8 13:05:22 xmpp-alt2 kernel: Code: c3 65 a1 08 00 00 00 89 42 24 > 8b 04 24 89 42 28 89 c8 c3 0f ba 70 08 01 c7 40 24 00 00 00 00 c7 40 > 28 00 00 00 00 e8 7a fe ff ff <0f> 0b eb fe 55 89 d5 57 89 c7 56 31 f6 > 53 8d 58 3c 83 ec 04 8b > Feb 8 13:05:22 xmpp-alt2 kernel: EIP: [<f890f11e>] > gfs2_glmutex_unlock+0x18/0x1c [gfs2] SS:ESP 0068:f15a3f64 > Feb 8 13:05:33 xmpp-alt2 kernel: <0>------------[ cut > here ]------------ > Feb 8 13:05:33 xmpp-alt2 kernel: kernel BUG at fs/gfs2/glock.c:738! > Feb 8 13:05:33 xmpp-alt2 kernel: invalid opcode: 0000 [#2] > Feb 8 13:05:33 xmpp-alt2 kernel: Modules linked in: lock_dlm dlm > configfs gfs2 > Feb 8 13:05:33 xmpp-alt2 kernel: CPU: 0 > Feb 8 13:05:33 xmpp-alt2 kernel: EIP: 0060:[<f890f11e>] Not > tainted VLI > Feb 8 13:05:33 xmpp-alt2 kernel: EFLAGS: 00000282 (2.6.20-xmpp2 #1) > Feb 8 13:05:34 xmpp-alt2 kernel: EIP is at gfs2_glmutex_unlock > +0x18/0x1c [gfs2] > Feb 8 13:05:34 xmpp-alt2 kernel: eax: f3aa1bbc ebx: f3aa1b78 ecx: > 00007080 edx: f3aa1bbc > Feb 8 13:05:34 xmpp-alt2 kernel: esi: f34f6000 edi: f3aa1b78 ebp: > 00000001 esp: f34f5f98 > Feb 8 13:05:34 xmpp-alt2 kernel: ds: 007b es: 007b ss: 0068 > Feb 8 13:05:34 xmpp-alt2 kernel: Process gfs2_scand (pid: 7012, > ti=f34f4000 task=f3bef030 task.ti=f34f4000) > Feb 8 13:05:34 xmpp-alt2 kernel: Stack: f8910940 f8910942 000002be > f34f6000 f8907800 fffffffc f89109aa f34f6000 > Feb 8 13:05:34 xmpp-alt2 kernel: f34f6000 f890780c f348ddc4 > c012320b 00000001 ffffffff ffffffff c012316e > Feb 8 13:05:34 xmpp-alt2 kernel: 00000000 00000000 00000000 > c01034df f348ddbc 00000000 00000000 00000000 > Feb 8 13:05:34 xmpp-alt2 kernel: Call Trace: > Feb 8 13:05:34 xmpp-alt2 kernel: [<f8910940>] examine_bucket > +0x59/0x5b [gfs2] > Feb 8 13:05:34 xmpp-alt2 kernel: [<f8910942>] scan_glock+0x0/0x51 > [gfs2] > Feb 8 13:05:34 xmpp-alt2 kernel: [<f8907800>] gfs2_scand+0x0/0x30 > [gfs2] > Feb 8 13:05:34 xmpp-alt2 kernel: [<f89109aa>] gfs2_scand_internal > +0x17/0x22 [gfs2] > Feb 8 13:05:34 xmpp-alt2 kernel: [<f890780c>] gfs2_scand+0xc/0x30 > [gfs2] > Feb 8 13:05:34 xmpp-alt2 kernel: [<c012320b>] kthread+0x9d/0xce > Feb 8 13:05:34 xmpp-alt2 kernel: [<c012316e>] kthread+0x0/0xce > Feb 8 13:05:34 xmpp-alt2 kernel: [<c01034df>] kernel_thread_helper > +0x7/0x10 > Feb 8 13:05:34 xmpp-alt2 kernel: ======================= > Feb 8 13:05:34 xmpp-alt2 kernel: Code: c3 65 a1 08 00 00 00 89 42 24 > 8b 04 24 89 42 28 89 c8 c3 0f ba 70 08 01 c7 40 24 00 00 00 00 c7 40 > 28 00 00 00 00 e8 7a fe ff ff <0f> 0b eb fe 55 89 d5 57 89 c7 56 31 f6 > 53 8d 58 3c 83 ec 04 8b > Feb 8 13:05:34 xmpp-alt2 kernel: EIP: [<f890f11e>] > gfs2_glmutex_unlock+0x18/0x1c [gfs2] SS:ESP 0068:f34f5f98 > > > -- > pozdrawiam, > Zbyszek ???kiewski ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <f7ca4caa0702080502m723d4bc3r3dbdb9302a942bff@mail.gmail.com>]
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20. [not found] ` <f7ca4caa0702080502m723d4bc3r3dbdb9302a942bff@mail.gmail.com> @ 2007-02-08 13:04 ` Zbyszek Żółkiewski 2007-02-08 13:26 ` Steven Whitehouse 0 siblings, 1 reply; 11+ messages in thread From: Zbyszek Żółkiewski @ 2007-02-08 13:04 UTC (permalink / raw) To: cluster-devel.redhat.com sorry - mail went only to Steven, now to group.... On 2/8/07, Zbyszek ???kiewski <zbyszek@toliman.pl> wrote: > > well, thanks for answer, i have tried with nolock, and result is as > follow: > of course i made mkfs -t gfs2 -p lock_nolock -t xmpp-alt2:test -j 1 > /dev/sdb1 and then: > mount -t gfs2 /dev/sdb1 /mnt -v > > and yes - the device is mounted, > (the changes to kernel you was talking about: you mean: git1 for 2.6.20?) > > > Feb 8 13:52:46 xmpp-alt2 kernel: Lock_Nolock (built Feb 8 2007 13:52:20) > installed > Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=: Trying to join cluster > "lock_nolock", "xmpp-alt2:test" > Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: Joined > cluster. Now mounting FS... > Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: jid=0, > already locked for use > Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2: test.0: jid=0: > Looking at journal... > Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: jid=0: Done > Feb 8 13:58:13 xmpp-alt2 kernel: ------------[ cut here ]------------ > Feb 8 13:58:13 xmpp-alt2 kernel: kernel BUG at fs/gfs2/glock.c:738! > Feb 8 13:58:13 xmpp-alt2 kernel: invalid opcode: 0000 [#1] > Feb 8 13:58:13 xmpp-alt2 kernel: Modules linked in: lock_nolock lock_dlm > gfs2 dlm configfs > Feb 8 13:58:13 xmpp-alt2 kernel: CPU: 0 > Feb 8 13:58:13 xmpp-alt2 kernel: EIP: 0060:[<f895011e>] Not tainted > VLI > Feb 8 13:58:13 xmpp-alt2 kernel: EFLAGS: 00000282 (2.6.20-xmpp2 #1) > Feb 8 13:58:13 xmpp-alt2 kernel: EIP is at gfs2_glmutex_unlock+0x18/0x1c > [gfs2] > Feb 8 13:58:13 xmpp-alt2 kernel: eax: f5085bbc ebx: f5085bec ecx: > f5875000 edx: f5085bbc > Feb 8 13:58:13 xmpp-alt2 kernel: esi: f5085b78 edi: f575ff88 ebp: > f575ff94 esp: f575ff64 > Feb 8 13:58:13 xmpp-alt2 kernel: ds: 007b es: 007b ss: 0068 > Feb 8 13:58:13 xmpp-alt2 kernel: Process gfs2_glockd (pid: 2645, > ti=f575e000 task=f58d3550 task.ti=f575e000) > Feb 8 13:58:13 xmpp-alt2 kernel: Stack: f89518e5 f5875000 f5875364 > f8948861 00000000 f58d3550 c0123580 f575ffa0 > Feb 8 13:58:13 xmpp-alt2 kernel: f575ffa0 f575ffac c010f84f > 00000000 00000000 f58d3550 c0123580 f575ffa0 > Feb 8 13:58:13 xmpp-alt2 kernel: f575ffa0 000004f8 b716dc5d > 0024d983 f5011dc4 f5875000 f8948830 fffffffc > Feb 8 13:58:13 xmpp-alt2 kernel: Call Trace: > Feb 8 13:58:13 xmpp-alt2 kernel: [<f89518e5>] > gfs2_reclaim_glock+0x8d/0x8f [gfs2] > Feb 8 13:58:13 xmpp-alt2 kernel: [<f8948861>] gfs2_glockd+0x31/0xe4 > [gfs2] > Feb 8 13:58:13 xmpp-alt2 kernel: [<c0123580>] > autoremove_wake_function+0x0/0x43 > Feb 8 13:58:13 xmpp-alt2 kernel: [<c010f84f>] __wake_up_common+0x33/0x56 > > Feb 8 13:58:13 xmpp-alt2 kernel: [<c0123580>] > autoremove_wake_function+0x0/0x43 > Feb 8 13:58:13 xmpp-alt2 kernel: [<f8948830>] gfs2_glockd+0x0/0xe4 > [gfs2] > Feb 8 13:58:13 xmpp-alt2 kernel: [<c012320b>] kthread+0x9d/0xce > Feb 8 13:58:13 xmpp-alt2 kernel: [<c012316e>] kthread+0x0/0xce > Feb 8 13:58:13 xmpp-alt2 kernel: [<c01034df>] > kernel_thread_helper+0x7/0x10 > Feb 8 13:58:13 xmpp-alt2 kernel: ======================= > Feb 8 13:58:13 xmpp-alt2 kernel: Code: c3 65 a1 08 00 00 00 89 42 24 8b > 04 24 89 42 28 89 c8 c3 0f ba 70 08 01 c7 40 24 00 00 00 00 c7 40 28 00 00 > 00 00 e8 7a fe ff ff <0f> 0b eb fe 55 89 d5 57 89 c7 56 31 f6 53 8d 58 3c 83 > ec 04 8b > Feb 8 13:58:13 xmpp-alt2 kernel: EIP: [<f895011e>] > gfs2_glmutex_unlock+0x18/0x1c [gfs2] SS:ESP 0068:f575ff64 > Feb 8 13:58:28 xmpp-alt2 kernel: <0>------------[ cut here ]------------ > Feb 8 13:58:28 xmpp-alt2 kernel: kernel BUG at fs/gfs2/glock.c:738! > Feb 8 13:58:28 xmpp-alt2 kernel: invalid opcode: 0000 [#2] > Feb 8 13:58:28 xmpp-alt2 kernel: Modules linked in: lock_nolock lock_dlm > gfs2 dlm configfs > Feb 8 13:58:28 xmpp-alt2 kernel: CPU: 0 > Feb 8 13:58:28 xmpp-alt2 kernel: EIP: 0060:[<f895011e>] Not tainted > VLI > Feb 8 13:58:28 xmpp-alt2 kernel: EFLAGS: 00000282 (2.6.20-xmpp2 #1) > Feb 8 13:58:28 xmpp-alt2 kernel: EIP is at gfs2_glmutex_unlock+0x18/0x1c > [gfs2] > Feb 8 13:58:28 xmpp-alt2 kernel: eax: f5085bbc ebx: f5085b78 ecx: > f5875000 edx: f5085bbc > Feb 8 13:58:28 xmpp-alt2 kernel: esi: f5875000 edi: f5085b78 ebp: > 00000001 esp: f577df98 > Feb 8 13:58:28 xmpp-alt2 kernel: ds: 007b es: 007b ss: 0068 > Feb 8 13:58:28 xmpp-alt2 kernel: Process gfs2_scand (pid: 2644, > ti=f577c000 task=f58d3a70 task.ti=f577c000) > Feb 8 13:58:28 xmpp-alt2 kernel: Stack: f8951940 f8951942 000001e0 > f5875000 f8948800 fffffffc f89519aa f5875000 > Feb 8 13:58:28 xmpp-alt2 kernel: f5875000 f894880c f5011dc4 > c012320b 00000001 ffffffff ffffffff c012316e > Feb 8 13:58:28 xmpp-alt2 kernel: 00000000 00000000 00000000 > c01034df f5011dbc 00000000 00000000 00000000 > Feb 8 13:58:28 xmpp-alt2 kernel: Call Trace: > Feb 8 13:58:28 xmpp-alt2 kernel: [<f8951940>] examine_bucket+0x59/0x5b > [gfs2] > Feb 8 13:58:28 xmpp-alt2 kernel: [<f8951942>] scan_glock+0x0/0x51 [gfs2] > Feb 8 13:58:28 xmpp-alt2 kernel: [<f8948800>] gfs2_scand+0x0/0x30 [gfs2] > Feb 8 13:58:28 xmpp-alt2 kernel: [<f89519aa>] > gfs2_scand_internal+0x17/0x22 [gfs2] > Feb 8 13:58:28 xmpp-alt2 kernel: [<f894880c>] gfs2_scand+0xc/0x30 [gfs2] > Feb 8 13:58:28 xmpp-alt2 kernel: [<c012320b>] kthread+0x9d/0xce > Feb 8 13:58:28 xmpp-alt2 kernel: [<c012316e>] kthread+0x0/0xce > Feb 8 13:58:28 xmpp-alt2 kernel: [<c01034df>] > kernel_thread_helper+0x7/0x10 > Feb 8 13:58:28 xmpp-alt2 kernel: ======================= > Feb 8 13:58:28 xmpp-alt2 kernel: Code: c3 65 a1 08 00 00 00 89 42 24 8b > 04 24 89 42 28 89 c8 c3 0f ba 70 08 01 c7 40 24 00 00 00 00 c7 40 28 00 00 > 00 00 e8 7a fe ff ff <0f> 0b eb fe 55 89 d5 57 89 c7 56 31 f6 53 8d 58 3c 83 > ec 04 8b > Feb 8 13:58:28 xmpp-alt2 kernel: EIP: [<f895011e>] > gfs2_glmutex_unlock+0x18/0x1c [gfs2] SS:ESP 0068:f577df98 > > > > > > On 2/8/07, Steven Whitehouse <swhiteho@redhat.com> wrote: > > > > Hi, > > > > On Thu, 2007-02-08 at 13:15 +0100, Zbyszek ???kiewski wrote: > > > Hi, > > > > > > I have recently setup cluster on Debian systems and i got issues > > > related with kernel panic: > > > systems affected: Debian 4.0 (testing) and Debian 3.1 (r4) > > > gcc: 3.4 and 4.1.2 > > > kernels: 2.6.19 and 2.6.20 > > > cluster from latest cvs > > > > > > command invoked: mount -t gfs2 /dev/sdb1 /mnt/ > > > docs that i followed : > > > > > http://sources.redhat.com/cgi-bin/cvsweb.cgi/cluster/doc/usage.txt?cvsroot=cluster > > > > > > any clue? > > > > > It looks like the mount was successful, but that the problem occurred > > right after mount. Just as a sanity check does the same thing happen if > > you use lock_nolock? > > > > There have been a number of bug fixes since 2.6.20 which went into > > Linus' kernel yesterday, so you might want to try the latest upstream > > kernel, but I don't recognise this problem as being something we've seen > > before, > > > > Steve. > > > > > > -- > pozdrawiam, > Zbyszek ???kiewski -- pozdrawiam, Zbyszek ???kiewski -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20070208/84892e46/attachment.htm> ^ permalink raw reply [flat|nested] 11+ messages in thread
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20. 2007-02-08 13:04 ` Zbyszek Żółkiewski @ 2007-02-08 13:26 ` Steven Whitehouse 2007-02-08 14:02 ` Zbyszek Żółkiewski 0 siblings, 1 reply; 11+ messages in thread From: Steven Whitehouse @ 2007-02-08 13:26 UTC (permalink / raw) To: cluster-devel.redhat.com Hi, On Thu, 2007-02-08 at 14:04 +0100, Zbyszek ???kiewski wrote: > sorry - mail went only to Steven, now to group.... > > > On 2/8/07, Zbyszek ???kiewski <zbyszek@toliman.pl> wrote: > well, thanks for answer, i have tried with nolock, and result > is as follow: > of course i made mkfs -t gfs2 -p lock_nolock -t xmpp-alt2:test > -j 1 /dev/sdb1 and then: > mount -t gfs2 /dev/sdb1 /mnt -v > > and yes - the device is mounted, It looks like what is happening is that a glmutex_unlock() is discovering that its spinlock has been dropped by glock.c:run_queue() which should be impossible, so something odd is happening here I think. The daemons implicated in this are there to demote unused locks on a periodic basis, so its presumably one of the locks used during mounting of the filesystem thats at fault. > (the changes to kernel you was talking about: you mean: git1 > for 2.6.20?) > I'm not sure if its in git1 or not, I suspect it will be git2 since it was only yesterday that the patches went in. Linus' current git tree seems to be broken (both gitweb and direct via the git tools) otherwise I'd post a URL to the changes. In the mean time you can find them in my -nmw tree which will get updated just as soon as git it working again at kernel.org, Steve. > > Feb 8 13:52:46 xmpp-alt2 kernel: Lock_Nolock (built Feb 8 > 2007 13:52:20) installed > Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=: Trying to join > cluster "lock_nolock", "xmpp-alt2:test" > Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: > Joined cluster. Now mounting FS... > Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: > jid=0, already locked for use > Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2: > test.0: jid=0: Looking at journal... > Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: > jid=0: Done > Feb 8 13:58:13 xmpp-alt2 kernel: ------------[ cut > here ]------------ > Feb 8 13:58:13 xmpp-alt2 kernel: kernel BUG at > fs/gfs2/glock.c:738! > Feb 8 13:58:13 xmpp-alt2 kernel: invalid opcode: 0000 [#1] > Feb 8 13:58:13 xmpp-alt2 kernel: Modules linked in: > lock_nolock lock_dlm gfs2 dlm configfs > Feb 8 13:58:13 xmpp-alt2 kernel: CPU: 0 > Feb 8 13:58:13 xmpp-alt2 kernel: EIP: 0060:[<f895011e>] > Not tainted VLI > Feb 8 13:58:13 xmpp-alt2 kernel: EFLAGS: 00000282 > (2.6.20-xmpp2 #1) > Feb 8 13:58:13 xmpp-alt2 kernel: EIP is at > gfs2_glmutex_unlock+0x18/0x1c [gfs2] > Feb 8 13:58:13 xmpp-alt2 kernel: eax: f5085bbc ebx: > f5085bec ecx: f5875000 edx: f5085bbc > Feb 8 13:58:13 xmpp-alt2 kernel: esi: f5085b78 edi: > f575ff88 ebp: f575ff94 esp: f575ff64 > Feb 8 13:58:13 xmpp-alt2 kernel: ds: 007b es: 007b ss: > 0068 > Feb 8 13:58:13 xmpp-alt2 kernel: Process gfs2_glockd (pid: > 2645, ti=f575e000 task=f58d3550 task.ti=f575e000) > Feb 8 13:58:13 xmpp-alt2 kernel: Stack: f89518e5 f5875000 > f5875364 f8948861 00000000 f58d3550 c0123580 f575ffa0 > Feb 8 13:58:13 xmpp-alt2 kernel: f575ffa0 f575ffac > c010f84f 00000000 00000000 f58d3550 c0123580 f575ffa0 > Feb 8 13:58:13 xmpp-alt2 kernel: f575ffa0 000004f8 > b716dc5d 0024d983 f5011dc4 f5875000 f8948830 fffffffc > Feb 8 13:58:13 xmpp-alt2 kernel: Call Trace: > Feb 8 13:58:13 xmpp-alt2 kernel: [<f89518e5>] > gfs2_reclaim_glock+0x8d/0x8f [gfs2] > Feb 8 13:58:13 xmpp-alt2 kernel: [<f8948861>] gfs2_glockd > +0x31/0xe4 [gfs2] > Feb 8 13:58:13 xmpp-alt2 kernel: [<c0123580>] > autoremove_wake_function+0x0/0x43 > Feb 8 13:58:13 xmpp-alt2 kernel: [<c010f84f>] > __wake_up_common+0x33/0x56 > Feb 8 13:58:13 xmpp-alt2 kernel: [<c0123580>] > autoremove_wake_function+0x0/0x43 > Feb 8 13:58:13 xmpp-alt2 kernel: [<f8948830>] gfs2_glockd > +0x0/0xe4 [gfs2] > Feb 8 13:58:13 xmpp-alt2 kernel: [<c012320b>] kthread > +0x9d/0xce > Feb 8 13:58:13 xmpp-alt2 kernel: [<c012316e>] kthread > +0x0/0xce > Feb 8 13:58:13 xmpp-alt2 kernel: [<c01034df>] > kernel_thread_helper+0x7/0x10 > Feb 8 13:58:13 xmpp-alt2 kernel: ======================= > Feb 8 13:58:13 xmpp-alt2 kernel: Code: c3 65 a1 08 00 00 00 > 89 42 24 8b 04 24 89 42 28 89 c8 c3 0f ba 70 08 01 c7 40 24 00 > 00 00 00 c7 40 28 00 00 00 00 e8 7a fe ff ff <0f> 0b eb fe 55 > 89 d5 57 89 c7 56 31 f6 53 8d 58 3c 83 ec 04 8b > Feb 8 13:58:13 xmpp-alt2 kernel: EIP: [<f895011e>] > gfs2_glmutex_unlock+0x18/0x1c [gfs2] SS:ESP 0068:f575ff64 > Feb 8 13:58:28 xmpp-alt2 kernel: <0>------------[ cut > here ]------------ > Feb 8 13:58:28 xmpp-alt2 kernel: kernel BUG at > fs/gfs2/glock.c:738! > Feb 8 13:58:28 xmpp-alt2 kernel: invalid opcode: 0000 [#2] > Feb 8 13:58:28 xmpp-alt2 kernel: Modules linked in: > lock_nolock lock_dlm gfs2 dlm configfs > Feb 8 13:58:28 xmpp-alt2 kernel: CPU: 0 > Feb 8 13:58:28 xmpp-alt2 kernel: EIP: 0060:[<f895011e>] > Not tainted VLI > Feb 8 13:58:28 xmpp-alt2 kernel: EFLAGS: 00000282 > (2.6.20-xmpp2 #1) > Feb 8 13:58:28 xmpp-alt2 kernel: EIP is at > gfs2_glmutex_unlock+0x18/0x1c [gfs2] > Feb 8 13:58:28 xmpp-alt2 kernel: eax: f5085bbc ebx: > f5085b78 ecx: f5875000 edx: f5085bbc > Feb 8 13:58:28 xmpp-alt2 kernel: esi: f5875000 edi: > f5085b78 ebp: 00000001 esp: f577df98 > Feb 8 13:58:28 xmpp-alt2 kernel: ds: 007b es: 007b ss: > 0068 > Feb 8 13:58:28 xmpp-alt2 kernel: Process gfs2_scand (pid: > 2644, ti=f577c000 task=f58d3a70 task.ti=f577c000) > Feb 8 13:58:28 xmpp-alt2 kernel: Stack: f8951940 f8951942 > 000001e0 f5875000 f8948800 fffffffc f89519aa f5875000 > Feb 8 13:58:28 xmpp-alt2 kernel: f5875000 f894880c > f5011dc4 c012320b 00000001 ffffffff ffffffff c012316e > Feb 8 13:58:28 xmpp-alt2 kernel: 00000000 00000000 > 00000000 c01034df f5011dbc 00000000 00000000 00000000 > Feb 8 13:58:28 xmpp-alt2 kernel: Call Trace: > Feb 8 13:58:28 xmpp-alt2 kernel: [<f8951940>] examine_bucket > +0x59/0x5b [gfs2] > Feb 8 13:58:28 xmpp-alt2 kernel: [<f8951942>] scan_glock > +0x0/0x51 [gfs2] > Feb 8 13:58:28 xmpp-alt2 kernel: [<f8948800>] gfs2_scand > +0x0/0x30 [gfs2] > Feb 8 13:58:28 xmpp-alt2 kernel: [<f89519aa>] > gfs2_scand_internal+0x17/0x22 [gfs2] > Feb 8 13:58:28 xmpp-alt2 kernel: [<f894880c>] gfs2_scand > +0xc/0x30 [gfs2] > Feb 8 13:58:28 xmpp-alt2 kernel: [<c012320b>] kthread > +0x9d/0xce > Feb 8 13:58:28 xmpp-alt2 kernel: [<c012316e>] kthread > +0x0/0xce > Feb 8 13:58:28 xmpp-alt2 kernel: [<c01034df>] > kernel_thread_helper+0x7/0x10 > Feb 8 13:58:28 xmpp-alt2 kernel: ======================= > Feb 8 13:58:28 xmpp-alt2 kernel: Code: c3 65 a1 08 00 00 00 > 89 42 24 8b 04 24 89 42 28 89 c8 c3 0f ba 70 08 01 c7 40 24 00 > 00 00 00 c7 40 28 00 00 00 00 e8 7a fe ff ff <0f> 0b eb fe 55 > 89 d5 57 89 c7 56 31 f6 53 8d 58 3c 83 ec 04 8b > Feb 8 13:58:28 xmpp-alt2 kernel: EIP: [<f895011e>] > gfs2_glmutex_unlock+0x18/0x1c [gfs2] SS:ESP 0068:f577df98 > > > > > > On 2/8/07, Steven Whitehouse <swhiteho@redhat.com> wrote: > Hi, > > On Thu, 2007-02-08 at 13:15 +0100, Zbyszek ???kiewski > wrote: > > Hi, > > > > I have recently setup cluster on Debian systems and > i got issues > > related with kernel panic: > > systems affected: Debian 4.0 (testing) and Debian > 3.1 (r4) > > gcc: 3.4 and 4.1.2 > > kernels: 2.6.19 and 2.6.20 > > cluster from latest cvs > > > > command invoked: mount -t gfs2 /dev/sdb1 /mnt/ > > docs that i followed : > > > http://sources.redhat.com/cgi-bin/cvsweb.cgi/cluster/doc/usage.txt?cvsroot=cluster > > > > any clue? > > > It looks like the mount was successful, but that the > problem occurred > right after mount. Just as a sanity check does the > same thing happen if > you use lock_nolock? > > There have been a number of bug fixes since 2.6.20 > which went into > Linus' kernel yesterday, so you might want to try the > latest upstream > kernel, but I don't recognise this problem as being > something we've seen > before, > > Steve. > > > > -- > pozdrawiam, > Zbyszek ???kiewski > > > > -- > pozdrawiam, > Zbyszek ???kiewski ^ permalink raw reply [flat|nested] 11+ messages in thread
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20. 2007-02-08 13:26 ` Steven Whitehouse @ 2007-02-08 14:02 ` Zbyszek Żółkiewski 2007-02-08 16:26 ` Steven Whitehouse 0 siblings, 1 reply; 11+ messages in thread From: Zbyszek Żółkiewski @ 2007-02-08 14:02 UTC (permalink / raw) To: cluster-devel.redhat.com ok , i have build kernel from your git ( git://git.kernel.org/pub/scm/linux/kernel/git/steve/gfs2-2.6-nmw.git) Linux version 2.6.20-xmpp2-ga2cf8222-dirty (root at xmpp-alt2) (gcc version 3.4.4 20050314 (prerelease) (Debian 3.4.3-13sarge1)) #1 Thu Feb 8 14:51:21 CET 2007 and there is the same problem; kernel BUG at fs/gfs2/glock.c:704! invalid opcode: 0000 [#2] Modules linked in: lock_nolock lock_dlm gfs2 dlm configfs CPU: 0 EIP: 0060:[<f8950173>] Not tainted VLI EFLAGS: 00000282 (2.6.20-xmpp2-ga2cf8222-dirty #1) EIP is at gfs2_glmutex_unlock+0x18/0x1c [gfs2] and so one.... ok so waiting patiently for solution.... On 2/8/07, Steven Whitehouse <swhiteho@redhat.com> wrote: > > Hi, > > On Thu, 2007-02-08 at 14:04 +0100, Zbyszek ???kiewski wrote: > > sorry - mail went only to Steven, now to group.... > > > > > > On 2/8/07, Zbyszek ???kiewski <zbyszek@toliman.pl> wrote: > > well, thanks for answer, i have tried with nolock, and result > > is as follow: > > of course i made mkfs -t gfs2 -p lock_nolock -t xmpp-alt2:test > > -j 1 /dev/sdb1 and then: > > mount -t gfs2 /dev/sdb1 /mnt -v > > > > and yes - the device is mounted, > It looks like what is happening is that a glmutex_unlock() is > discovering that its spinlock has been dropped by glock.c:run_queue() > which should be impossible, so something odd is happening here I think. > > The daemons implicated in this are there to demote unused locks on a > periodic basis, so its presumably one of the locks used during mounting > of the filesystem thats at fault. > > > (the changes to kernel you was talking about: you mean: git1 > > for 2.6.20?) > > > I'm not sure if its in git1 or not, I suspect it will be git2 since it > was only yesterday that the patches went in. Linus' current git tree > seems to be broken (both gitweb and direct via the git tools) otherwise > I'd post a URL to the changes. In the mean time you can find them in my > -nmw tree which will get updated just as soon as git it working again at > kernel.org, > > Steve. > > > -- pozdrawiam, Zbyszek ???kiewski -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20070208/748a6f60/attachment.htm> ^ permalink raw reply [flat|nested] 11+ messages in thread
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20. 2007-02-08 14:02 ` Zbyszek Żółkiewski @ 2007-02-08 16:26 ` Steven Whitehouse 2007-02-08 17:21 ` Zbyszek Żółkiewski 0 siblings, 1 reply; 11+ messages in thread From: Steven Whitehouse @ 2007-02-08 16:26 UTC (permalink / raw) To: cluster-devel.redhat.com Hi, I cannot reproduce what you are seeing, but try the attached patch to see if we can narrow this down, Steve. On Thu, 2007-02-08 at 15:02 +0100, Zbyszek ???kiewski wrote: > ok , i have build kernel from your git ( > git://git.kernel.org/pub/scm/linux/kernel/git/steve/gfs2-2.6-nmw.git) > Linux version 2.6.20-xmpp2-ga2cf8222-dirty (root at xmpp-alt2) (gcc > version 3.4.4 20050314 (prerelease) (Debian 3.4.3-13sarge1)) #1 Thu > Feb 8 14:51:21 CET 2007 > > and there is the same problem; > kernel BUG at fs/gfs2/glock.c:704! > invalid opcode: 0000 [#2] > Modules linked in: lock_nolock lock_dlm gfs2 dlm configfs > CPU: 0 > EIP: 0060:[<f8950173>] Not tainted VLI > EFLAGS: 00000282 (2.6.20-xmpp2-ga2cf8222-dirty #1) > EIP is at gfs2_glmutex_unlock+0x18/0x1c [gfs2] > > and so one.... > > ok so waiting patiently for solution.... > > > > > On 2/8/07, Steven Whitehouse <swhiteho@redhat.com> wrote: > Hi, > > On Thu, 2007-02-08 at 14:04 +0100, Zbyszek ???kiewski wrote: > > sorry - mail went only to Steven, now to group.... > > > > > > On 2/8/07, Zbyszek ???kiewski <zbyszek@toliman.pl> wrote: > > well, thanks for answer, i have tried with nolock, > and result > > is as follow: > > of course i made mkfs -t gfs2 -p lock_nolock -t > xmpp-alt2:test > > -j 1 /dev/sdb1 and then: > > mount -t gfs2 /dev/sdb1 /mnt -v > > > > and yes - the device is mounted, > It looks like what is happening is that a glmutex_unlock() is > discovering that its spinlock has been dropped by > glock.c:run_queue() > which should be impossible, so something odd is happening here > I think. > > The daemons implicated in this are there to demote unused > locks on a > periodic basis, so its presumably one of the locks used during > mounting > of the filesystem thats at fault. > > > (the changes to kernel you was talking about: you > mean: git1 > > for 2.6.20?) > > > I'm not sure if its in git1 or not, I suspect it will be git2 > since it > was only yesterday that the patches went in. Linus' current > git tree > seems to be broken (both gitweb and direct via the git tools) > otherwise > I'd post a URL to the changes. In the mean time you can find > them in my > -nmw tree which will get updated just as soon as git it > working again at > kernel.org, > > Steve. > > > > > -- > pozdrawiam, > Zbyszek ???kiewski -------------- next part -------------- A non-text attachment was scrubbed... Name: test.diff Type: text/x-patch Size: 1259 bytes Desc: not available URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20070208/ca40ccac/attachment.bin> ^ permalink raw reply [flat|nested] 11+ messages in thread
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20. 2007-02-08 16:26 ` Steven Whitehouse @ 2007-02-08 17:21 ` Zbyszek Żółkiewski 2007-02-09 10:32 ` Steven Whitehouse 0 siblings, 1 reply; 11+ messages in thread From: Zbyszek Żółkiewski @ 2007-02-08 17:21 UTC (permalink / raw) To: cluster-devel.redhat.com are you using the same glibc and distro? well i have patched gfs2 and here is output: Feb 8 18:16:28 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: Joined cluster. Now mounting FS... Feb 8 18:16:28 xmpp-alt2 kernel: ------------[ cut here ]------------ Feb 8 18:16:28 xmpp-alt2 kernel: kernel BUG at fs/gfs2/glock.c:632! Feb 8 18:16:28 xmpp-alt2 kernel: invalid opcode: 0000 [#1] Feb 8 18:16:28 xmpp-alt2 kernel: Modules linked in: lock_nolock lock_dlm gfs2 dlm configfs Feb 8 18:16:28 xmpp-alt2 kernel: CPU: 0 Feb 8 18:16:28 xmpp-alt2 kernel: EIP: 0060:[<f894ff98>] Not tainted VLI Feb 8 18:16:28 xmpp-alt2 kernel: EFLAGS: 00000292 (2.6.20-xmpp2 #3) Feb 8 18:16:28 xmpp-alt2 kernel: EIP is at run_queue+0x0/0x4 [gfs2] Feb 8 18:16:28 xmpp-alt2 kernel: eax: f5f119a4 ebx: f5ef9e34 ecx: f5f119e8 edx: f5f119e8 Feb 8 18:16:28 xmpp-alt2 kernel: esi: f5f119a4 edi: 00000000 ebp: f5e7c000 esp: f5ef9dd4 Feb 8 18:16:28 xmpp-alt2 kernel: ds: 007b es: 007b ss: 0068 Feb 8 18:16:28 xmpp-alt2 kernel: Process mount.gfs2 (pid: 1907, ti=f5ef8000 task=f5e1b550 task.ti=f5ef8000) Feb 8 18:16:28 xmpp-alt2 kernel: Stack: f8950d03 00000000 f5e7c000 00000000 f5ef9e34 f895101f f5ef9e34 00000001 Feb 8 18:16:28 xmpp-alt2 kernel: f5ef9df8 f5f119a4 f5eff550 f895c359 f89687a0 00000001 00000404 f5ef9e34 Feb 8 18:16:28 xmpp-alt2 kernel: 00000000 f5e7c000 f5c6ae00 00000000 f895d2bc f5ef9e7c c02d572f f5ef9e5c Feb 8 18:16:28 xmpp-alt2 kernel: Call Trace: Feb 8 18:16:28 xmpp-alt2 kernel: [<f8950d03>] gfs2_glock_nq+0x42/0x63 [gfs2] Feb 8 18:16:28 xmpp-alt2 kernel: [<f895101f>] gfs2_glock_nq_num+0x4e/0x6f [gfs2] Feb 8 18:16:28 xmpp-alt2 kernel: [<f895c359>] init_locking+0xe1/0x2b0 [gfs2] Feb 8 18:16:28 xmpp-alt2 kernel: [<f895d2bc>] fill_super+0xd6/0x231 [gfs2] Feb 8 18:16:28 xmpp-alt2 kernel: [<f8951016>] gfs2_glock_nq_num+0x45/0x6f [gfs2] Feb 8 18:16:28 xmpp-alt2 kernel: [<c014b975>] get_sb_bdev+0xe1/0x119 Feb 8 18:16:28 xmpp-alt2 kernel: [<f895d43e>] gfs2_get_sb+0x27/0x4a [gfs2] Feb 8 18:16:28 xmpp-alt2 kernel: [<f895d1e6>] fill_super+0x0/0x231 [gfs2] Feb 8 18:16:28 xmpp-alt2 kernel: [<c014bb98>] vfs_kern_mount+0x4c/0x99 Feb 8 18:16:28 xmpp-alt2 kernel: [<c014bc1a>] do_kern_mount+0x35/0x50 Feb 8 18:16:28 xmpp-alt2 kernel: [<c015d4b5>] do_new_mount+0x6b/0xb7 Feb 8 18:16:28 xmpp-alt2 kernel: [<c015db41>] do_mount+0x1ca/0x1ec Feb 8 18:16:28 xmpp-alt2 kernel: [<c0133feb>] __alloc_pages+0x54/0x2d4 Feb 8 18:16:28 xmpp-alt2 kernel: [<c015d921>] copy_mount_options+0x56/0xac Feb 8 18:16:28 xmpp-alt2 kernel: [<c015de04>] sys_mount+0x74/0xab Feb 8 18:16:28 xmpp-alt2 kernel: [<c01029c4>] syscall_call+0x7/0xb Feb 8 18:16:28 xmpp-alt2 kernel: ======================= Feb 8 18:16:28 xmpp-alt2 kernel: Code: 08 8b 48 04 8b 00 89 48 04 89 01 89 5b 04 89 1b b8 07 00 00 00 0f b3 42 08 89 d8 e8 5b fd ff ff 89 d8 e8 07 79 7f c7 31 c0 5b c3 <0f> 0b eb fe 56 31 c9 31 d2 53 89 c3 83 ec 34 8d 74 24 04 89 34 Feb 8 18:16:28 xmpp-alt2 kernel: EIP: [<f894ff98>] run_queue+0x0/0x4 [gfs2] SS:ESP 0068:f5ef9dd4 Feb 8 18:16:42 xmpp-alt2 kernel: <0>------------[ cut here ]------------ Feb 8 18:16:42 xmpp-alt2 kernel: kernel BUG at fs/gfs2/glock.c:632! Feb 8 18:16:42 xmpp-alt2 kernel: invalid opcode: 0000 [#2] Feb 8 18:16:42 xmpp-alt2 kernel: Modules linked in: lock_nolock lock_dlm gfs2 dlm configfs Feb 8 18:16:42 xmpp-alt2 kernel: CPU: 0 Feb 8 18:16:42 xmpp-alt2 kernel: EIP: 0060:[<f894ff98>] Not tainted VLI Feb 8 18:16:42 xmpp-alt2 kernel: EFLAGS: 00000247 (2.6.20-xmpp2 #3) Feb 8 18:16:42 xmpp-alt2 kernel: EIP is at run_queue+0x0/0x4 [gfs2] Feb 8 18:16:42 xmpp-alt2 kernel: eax: f5f119a4 ebx: f5f119a4 ecx: 00000001 edx: f5f119a4 Feb 8 18:16:42 xmpp-alt2 kernel: esi: f5e7c000 edi: f5f119a4 ebp: 00000001 esp: f5f33f94 Feb 8 18:16:42 xmpp-alt2 kernel: ds: 007b es: 007b ss: 0068 Feb 8 18:16:42 xmpp-alt2 kernel: Process gfs2_scand (pid: 1925, ti=f5f32000 task=f5effa70 task.ti=f5f32000) Feb 8 18:16:42 xmpp-alt2 kernel: Stack: f895005f f8951881 f8951883 00001990 f5e7c000 f8948800 fffffffc f89518eb Feb 8 18:16:42 xmpp-alt2 kernel: f5e7c000 f5e7c000 f894880c f5ef9dc4 c012320b 00000001 ffffffff ffffffff Feb 8 18:16:42 xmpp-alt2 kernel: c012316e 00000000 00000000 00000000 c01034df f5ef9dbc 00000000 00000000 Feb 8 18:16:42 xmpp-alt2 kernel: Call Trace: Feb 8 18:16:42 xmpp-alt2 kernel: [<f895005f>] gfs2_glmutex_unlock+0x18/0x1c [gfs2] Feb 8 18:16:42 xmpp-alt2 kernel: [<f8951881>] examine_bucket+0x59/0x5b [gfs2] Feb 8 18:16:42 xmpp-alt2 kernel: [<f8951883>] scan_glock+0x0/0x51 [gfs2] Feb 8 18:16:42 xmpp-alt2 kernel: [<f8948800>] gfs2_scand+0x0/0x30 [gfs2] Feb 8 18:16:42 xmpp-alt2 kernel: [<f89518eb>] gfs2_scand_internal+0x17/0x22 [gfs2] Feb 8 18:16:42 xmpp-alt2 kernel: [<f894880c>] gfs2_scand+0xc/0x30 [gfs2] Feb 8 18:16:42 xmpp-alt2 kernel: [<c012320b>] kthread+0x9d/0xce Feb 8 18:16:42 xmpp-alt2 kernel: [<c012316e>] kthread+0x0/0xce Feb 8 18:16:42 xmpp-alt2 kernel: [<c01034df>] kernel_thread_helper+0x7/0x10 Feb 8 18:16:42 xmpp-alt2 kernel: ======================= Feb 8 18:16:42 xmpp-alt2 kernel: Code: 08 8b 48 04 8b 00 89 48 04 89 01 89 5b 04 89 1b b8 07 00 00 00 0f b3 42 08 89 d8 e8 5b fd ff ff 89 d8 e8 07 79 7f c7 31 c0 5b c3 <0f> 0b eb fe 56 31 c9 31 d2 53 89 c3 83 ec 34 8d 74 24 04 89 34 Feb 8 18:16:42 xmpp-alt2 kernel: EIP: [<f894ff98>] run_queue+0x0/0x4 [gfs2] SS:ESP 0068:f5f33f94 and? On 2/8/07, Steven Whitehouse <swhiteho@redhat.com> wrote: > > Hi, > > I cannot reproduce what you are seeing, but try the attached patch to > see if we can narrow this down, > > Steve. > > On Thu, 2007-02-08 at 15:02 +0100, Zbyszek ???kiewski wrote: > > ok , i have build kernel from your git ( > > git://git.kernel.org/pub/scm/linux/kernel/git/steve/gfs2-2.6-nmw.git) > > Linux version 2.6.20-xmpp2-ga2cf8222-dirty (root at xmpp-alt2) (gcc > > version 3.4.4 20050314 (prerelease) (Debian 3.4.3-13sarge1)) #1 Thu > > Feb 8 14:51:21 CET 2007 > > > > and there is the same problem; > > kernel BUG at fs/gfs2/glock.c:704! > > invalid opcode: 0000 [#2] > > Modules linked in: lock_nolock lock_dlm gfs2 dlm configfs > > CPU: 0 > > EIP: 0060:[<f8950173>] Not tainted VLI > > EFLAGS: 00000282 (2.6.20-xmpp2-ga2cf8222-dirty #1) > > EIP is at gfs2_glmutex_unlock+0x18/0x1c [gfs2] > > > > and so one.... > > > > ok so waiting patiently for solution.... > > > > > > > > > > On 2/8/07, Steven Whitehouse <swhiteho@redhat.com> wrote: > > Hi, > > > > On Thu, 2007-02-08 at 14:04 +0100, Zbyszek ???kiewski wrote: > > > sorry - mail went only to Steven, now to group.... > > > > > > > > > On 2/8/07, Zbyszek ???kiewski <zbyszek@toliman.pl> wrote: > > > well, thanks for answer, i have tried with nolock, > > and result > > > is as follow: > > > of course i made mkfs -t gfs2 -p lock_nolock -t > > xmpp-alt2:test > > > -j 1 /dev/sdb1 and then: > > > mount -t gfs2 /dev/sdb1 /mnt -v > > > > > > and yes - the device is mounted, > > It looks like what is happening is that a glmutex_unlock() is > > discovering that its spinlock has been dropped by > > glock.c:run_queue() > > which should be impossible, so something odd is happening here > > I think. > > > > The daemons implicated in this are there to demote unused > > locks on a > > periodic basis, so its presumably one of the locks used during > > mounting > > of the filesystem thats at fault. > > > > > (the changes to kernel you was talking about: you > > mean: git1 > > > for 2.6.20?) > > > > > I'm not sure if its in git1 or not, I suspect it will be git2 > > since it > > was only yesterday that the patches went in. Linus' current > > git tree > > seems to be broken (both gitweb and direct via the git tools) > > otherwise > > I'd post a URL to the changes. In the mean time you can find > > them in my > > -nmw tree which will get updated just as soon as git it > > working again at > > kernel.org, > > > > Steve. > > > > > > > > > > -- > > pozdrawiam, > > Zbyszek ???kiewski > > -- pozdrawiam, Zbyszek ???kiewski -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20070208/c0060ea1/attachment.htm> ^ permalink raw reply [flat|nested] 11+ messages in thread
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20. 2007-02-08 17:21 ` Zbyszek Żółkiewski @ 2007-02-09 10:32 ` Steven Whitehouse 2007-02-09 10:34 ` Zbyszek Żółkiewski 0 siblings, 1 reply; 11+ messages in thread From: Steven Whitehouse @ 2007-02-09 10:32 UTC (permalink / raw) To: cluster-devel.redhat.com Hi, On Thu, 2007-02-08 at 18:21 +0100, Zbyszek ???kiewski wrote: > are you using the same glibc and distro? > I don't think glibc will make a difference here. I'm using FC-6 but with the upstream kernel. > well i have patched gfs2 and here is output: > Thanks for the report, I presume that line 632 is, in your glock.c the BUG_ON which is directly after the call to rq_promote? If so then we've narrowed it down to that function. Having said that, even after looking at every case of lock & unlock of gl_spin I still can't see any unbalanced pairs of locks. rq_promote does, in one case drop that spin lock, but it does also take it again before exiting the function. By adding some more BUG_ON()s into rq_promote, it might be possible to track it closer to the source. Are you using preempt btw? That ought to be fine if you are, but its always helpful to know when looking at locking problems, Steve. ^ permalink raw reply [flat|nested] 11+ messages in thread
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20. 2007-02-09 10:32 ` Steven Whitehouse @ 2007-02-09 10:34 ` Zbyszek Żółkiewski 2007-02-09 11:41 ` Steven Whitehouse 0 siblings, 1 reply; 11+ messages in thread From: Zbyszek Żółkiewski @ 2007-02-09 10:34 UTC (permalink / raw) To: cluster-devel.redhat.com well yes the line is: static void run_queue(struct gfs2_glock *gl) { struct gfs2_holder *gh; int blocked = 1; BUG_ON(!spin_is_locked(&gl->gl_spin)); < ------------------------------ line 632 for (;;) { ....... and i don;t use preemption (No Forced....)... i was playing with various options but the bug still appears, no metter how i setup gfs, on what distro (debian stable/eth).... On 2/9/07, Steven Whitehouse <swhiteho@redhat.com> wrote: > > Hi, > > On Thu, 2007-02-08 at 18:21 +0100, Zbyszek ???kiewski wrote: > > are you using the same glibc and distro? > > > I don't think glibc will make a difference here. I'm using FC-6 but with > the upstream kernel. > > > well i have patched gfs2 and here is output: > > > Thanks for the report, I presume that line 632 is, in your glock.c the > BUG_ON which is directly after the call to rq_promote? If so then we've > narrowed it down to that function. Having said that, even after looking > at every case of lock & unlock of gl_spin I still can't see any > unbalanced pairs of locks. > > rq_promote does, in one case drop that spin lock, but it does also take > it again before exiting the function. By adding some more BUG_ON()s into > rq_promote, it might be possible to track it closer to the source. > > Are you using preempt btw? That ought to be fine if you are, but its > always helpful to know when looking at locking problems, > > Steve. > > > -- pozdrawiam, Zbyszek ???kiewski -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20070209/438bfa28/attachment.htm> ^ permalink raw reply [flat|nested] 11+ messages in thread
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20. 2007-02-09 10:34 ` Zbyszek Żółkiewski @ 2007-02-09 11:41 ` Steven Whitehouse 2007-02-09 12:01 ` Zbyszek Żółkiewski 0 siblings, 1 reply; 11+ messages in thread From: Steven Whitehouse @ 2007-02-09 11:41 UTC (permalink / raw) To: cluster-devel.redhat.com Hi, Ok, that confuses me even more then, as thats not my expected location of this problem. Looking back through the call chain it seems that there is no obvious way in which this spinlock could become unlocked. I'll have to think about this some more and get back to you I'm afraid, Steve. On Fri, 2007-02-09 at 11:34 +0100, Zbyszek ???kiewski wrote: > well yes the line is: > > static void run_queue(struct gfs2_glock *gl) > { > struct gfs2_holder *gh; > int blocked = 1; > > BUG_ON(!spin_is_locked(&gl->gl_spin)); < > ------------------------------ line 632 > for (;;) { > ....... > > and i don;t use preemption (No Forced....)... i was playing with > various options but the bug still appears, no metter how i setup gfs, > on what distro (debian stable/eth).... > > > > > > On 2/9/07, Steven Whitehouse <swhiteho@redhat.com> wrote: > Hi, > > On Thu, 2007-02-08 at 18:21 +0100, Zbyszek ???kiewski wrote: > > are you using the same glibc and distro? > > > I don't think glibc will make a difference here. I'm using > FC-6 but with > the upstream kernel. > > > well i have patched gfs2 and here is output: > > > Thanks for the report, I presume that line 632 is, in your > glock.c the > BUG_ON which is directly after the call to rq_promote? If so > then we've > narrowed it down to that function. Having said that, even > after looking > at every case of lock & unlock of gl_spin I still can't see > any > unbalanced pairs of locks. > > rq_promote does, in one case drop that spin lock, but it does > also take > it again before exiting the function. By adding some more > BUG_ON()s into > rq_promote, it might be possible to track it closer to the > source. > > Are you using preempt btw? That ought to be fine if you are, > but its > always helpful to know when looking at locking problems, > > Steve. > > > > > > -- > pozdrawiam, > Zbyszek ???kiewski ^ permalink raw reply [flat|nested] 11+ messages in thread
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20. 2007-02-09 11:41 ` Steven Whitehouse @ 2007-02-09 12:01 ` Zbyszek Żółkiewski 0 siblings, 0 replies; 11+ messages in thread From: Zbyszek Żółkiewski @ 2007-02-09 12:01 UTC (permalink / raw) To: cluster-devel.redhat.com ok then, let me know if you find something... On 2/9/07, Steven Whitehouse <swhiteho@redhat.com> wrote: > > Hi, > > Ok, that confuses me even more then, as thats not my expected location > of this problem. Looking back through the call chain it seems that there > is no obvious way in which this spinlock could become unlocked. > > I'll have to think about this some more and get back to you I'm afraid, > > Steve. > > On Fri, 2007-02-09 at 11:34 +0100, Zbyszek ???kiewski wrote: > > well yes the line is: > > > > static void run_queue(struct gfs2_glock *gl) > > { > > struct gfs2_holder *gh; > > int blocked = 1; > > > > BUG_ON(!spin_is_locked(&gl->gl_spin)); < > > ------------------------------ line 632 > > for (;;) { > > ....... > > > > and i don;t use preemption (No Forced....)... i was playing with > > various options but the bug still appears, no metter how i setup gfs, > > on what distro (debian stable/eth).... > > > > > > > > > > > > On 2/9/07, Steven Whitehouse <swhiteho@redhat.com> wrote: > > Hi, > > > > On Thu, 2007-02-08 at 18:21 +0100, Zbyszek ???kiewski wrote: > > > are you using the same glibc and distro? > > > > > I don't think glibc will make a difference here. I'm using > > FC-6 but with > > the upstream kernel. > > > > > well i have patched gfs2 and here is output: > > > > > Thanks for the report, I presume that line 632 is, in your > > glock.c the > > BUG_ON which is directly after the call to rq_promote? If so > > then we've > > narrowed it down to that function. Having said that, even > > after looking > > at every case of lock & unlock of gl_spin I still can't see > > any > > unbalanced pairs of locks. > > > > rq_promote does, in one case drop that spin lock, but it does > > also take > > it again before exiting the function. By adding some more > > BUG_ON()s into > > rq_promote, it might be possible to track it closer to the > > source. > > > > Are you using preempt btw? That ought to be fine if you are, > > but its > > always helpful to know when looking at locking problems, > > > > Steve. > > > > > > > > > > > > -- > > pozdrawiam, > > Zbyszek ???kiewski > > -- pozdrawiam, Zbyszek ???kiewski -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20070209/79b7de07/attachment.htm> ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2007-02-09 12:01 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2007-02-08 12:15 [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20 Zbyszek Żółkiewski
2007-02-08 12:57 ` Steven Whitehouse
[not found] ` <f7ca4caa0702080502m723d4bc3r3dbdb9302a942bff@mail.gmail.com>
2007-02-08 13:04 ` Zbyszek Żółkiewski
2007-02-08 13:26 ` Steven Whitehouse
2007-02-08 14:02 ` Zbyszek Żółkiewski
2007-02-08 16:26 ` Steven Whitehouse
2007-02-08 17:21 ` Zbyszek Żółkiewski
2007-02-09 10:32 ` Steven Whitehouse
2007-02-09 10:34 ` Zbyszek Żółkiewski
2007-02-09 11:41 ` Steven Whitehouse
2007-02-09 12:01 ` Zbyszek Żółkiewski
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).