* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20.
@ 2007-02-08 12:15 Zbyszek Żółkiewski
2007-02-08 12:57 ` Steven Whitehouse
0 siblings, 1 reply; 11+ messages in thread
From: Zbyszek Żółkiewski @ 2007-02-08 12:15 UTC (permalink / raw)
To: cluster-devel.redhat.com
Hi,
I have recently setup cluster on Debian systems and i got issues related
with kernel panic:
systems affected: Debian 4.0 (testing) and Debian 3.1 (r4)
gcc: 3.4 and 4.1.2
kernels: 2.6.19 and 2.6.20
cluster from latest cvs
command invoked: mount -t gfs2 /dev/sdb1 /mnt/
docs that i followed :
http://sources.redhat.com/cgi-bin/cvsweb.cgi/cluster/doc/usage.txt?cvsroot=cluster
any clue?
here are detailed logs:
Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: add member 1
Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: total members 1 error 0
Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: dlm_recover_directory
Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: dlm_recover_directory 0 entries
Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: recover 1 done: 0 ms
Feb 8 13:05:18 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: Joined
cluster. Now mounting FS...
Feb 8 13:05:21 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: jid=0,
already locked for use
Feb 8 13:05:21 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: jid=0:
Looking at journal...
Feb 8 13:05:21 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: jid=0: Done
Feb 8 13:05:21 xmpp-alt2 kernel: ------------[ cut here ]------------
Feb 8 13:05:21 xmpp-alt2 kernel: kernel BUG at fs/gfs2/glock.c:738!
Feb 8 13:05:21 xmpp-alt2 kernel: invalid opcode: 0000 [#1]
Feb 8 13:05:21 xmpp-alt2 kernel: Modules linked in: lock_dlm dlm configfs
gfs2
Feb 8 13:05:21 xmpp-alt2 kernel: CPU: 0
Feb 8 13:05:21 xmpp-alt2 kernel: EIP: 0060:[<f890f11e>] Not tainted
VLI
Feb 8 13:05:21 xmpp-alt2 kernel: EFLAGS: 00000282 (2.6.20-xmpp2 #1)
Feb 8 13:05:21 xmpp-alt2 kernel: EIP is at gfs2_glmutex_unlock+0x18/0x1c
[gfs2]
Feb 8 13:05:21 xmpp-alt2 kernel: eax: f3aa1bbc ebx: f3aa1bec ecx:
00007080 edx: f3aa1bbc
Feb 8 13:05:22 xmpp-alt2 kernel: esi: f3aa1b78 edi: f15a3f88 ebp:
f15a3f94 esp: f15a3f64
Feb 8 13:05:22 xmpp-alt2 kernel: ds: 007b es: 007b ss: 0068
Feb 8 13:05:22 xmpp-alt2 kernel: Process gfs2_glockd (pid: 7013,
ti=f15a2000 task=f3ad9a70 task.ti=f15a2000)
Feb 8 13:05:22 xmpp-alt2 kernel: Stack: f89108e5 f34f6000 f34f6364 f8907861
00000000 f3ad9a70 c0123580 f15a3fa0
Feb 8 13:05:22 xmpp-alt2 kernel: f15a3fa0 f15a3fac c010f84f 00000000
00000000 f3ad9a70 c0123580 f15a3fa0
Feb 8 13:05:22 xmpp-alt2 kernel: f15a3fa0 000004c1 f9a6da64 0024d65b
f348ddc4 f34f6000 f8907830 fffffffc
Feb 8 13:05:22 xmpp-alt2 kernel: Call Trace:
Feb 8 13:05:22 xmpp-alt2 kernel: [<f89108e5>] gfs2_reclaim_glock+0x8d/0x8f
[gfs2]
Feb 8 13:05:22 xmpp-alt2 kernel: [<f8907861>] gfs2_glockd+0x31/0xe4 [gfs2]
Feb 8 13:05:22 xmpp-alt2 kernel: [<c0123580>]
autoremove_wake_function+0x0/0x43
Feb 8 13:05:22 xmpp-alt2 kernel: [<c010f84f>] __wake_up_common+0x33/0x56
Feb 8 13:05:22 xmpp-alt2 kernel: [<c0123580>]
autoremove_wake_function+0x0/0x43
Feb 8 13:05:22 xmpp-alt2 kernel: [<f8907830>] gfs2_glockd+0x0/0xe4 [gfs2]
Feb 8 13:05:22 xmpp-alt2 kernel: [<c012320b>] kthread+0x9d/0xce
Feb 8 13:05:22 xmpp-alt2 kernel: [<c012316e>] kthread+0x0/0xce
Feb 8 13:05:22 xmpp-alt2 kernel: [<c01034df>]
kernel_thread_helper+0x7/0x10
Feb 8 13:05:22 xmpp-alt2 kernel: =======================
Feb 8 13:05:22 xmpp-alt2 kernel: Code: c3 65 a1 08 00 00 00 89 42 24 8b 04
24 89 42 28 89 c8 c3 0f ba 70 08 01 c7 40 24 00 00 00 00 c7 40 28 00 00 00
00 e8 7a fe ff ff <0f> 0b eb fe 55 89 d5 57 89 c7 56 31 f6 53 8d 58 3c 83 ec
04 8b
Feb 8 13:05:22 xmpp-alt2 kernel: EIP: [<f890f11e>]
gfs2_glmutex_unlock+0x18/0x1c [gfs2] SS:ESP 0068:f15a3f64
Feb 8 13:05:33 xmpp-alt2 kernel: <0>------------[ cut here ]------------
Feb 8 13:05:33 xmpp-alt2 kernel: kernel BUG at fs/gfs2/glock.c:738!
Feb 8 13:05:33 xmpp-alt2 kernel: invalid opcode: 0000 [#2]
Feb 8 13:05:33 xmpp-alt2 kernel: Modules linked in: lock_dlm dlm configfs
gfs2
Feb 8 13:05:33 xmpp-alt2 kernel: CPU: 0
Feb 8 13:05:33 xmpp-alt2 kernel: EIP: 0060:[<f890f11e>] Not tainted
VLI
Feb 8 13:05:33 xmpp-alt2 kernel: EFLAGS: 00000282 (2.6.20-xmpp2 #1)
Feb 8 13:05:34 xmpp-alt2 kernel: EIP is at gfs2_glmutex_unlock+0x18/0x1c
[gfs2]
Feb 8 13:05:34 xmpp-alt2 kernel: eax: f3aa1bbc ebx: f3aa1b78 ecx:
00007080 edx: f3aa1bbc
Feb 8 13:05:34 xmpp-alt2 kernel: esi: f34f6000 edi: f3aa1b78 ebp:
00000001 esp: f34f5f98
Feb 8 13:05:34 xmpp-alt2 kernel: ds: 007b es: 007b ss: 0068
Feb 8 13:05:34 xmpp-alt2 kernel: Process gfs2_scand (pid: 7012, ti=f34f4000
task=f3bef030 task.ti=f34f4000)
Feb 8 13:05:34 xmpp-alt2 kernel: Stack: f8910940 f8910942 000002be f34f6000
f8907800 fffffffc f89109aa f34f6000
Feb 8 13:05:34 xmpp-alt2 kernel: f34f6000 f890780c f348ddc4 c012320b
00000001 ffffffff ffffffff c012316e
Feb 8 13:05:34 xmpp-alt2 kernel: 00000000 00000000 00000000 c01034df
f348ddbc 00000000 00000000 00000000
Feb 8 13:05:34 xmpp-alt2 kernel: Call Trace:
Feb 8 13:05:34 xmpp-alt2 kernel: [<f8910940>] examine_bucket+0x59/0x5b
[gfs2]
Feb 8 13:05:34 xmpp-alt2 kernel: [<f8910942>] scan_glock+0x0/0x51 [gfs2]
Feb 8 13:05:34 xmpp-alt2 kernel: [<f8907800>] gfs2_scand+0x0/0x30 [gfs2]
Feb 8 13:05:34 xmpp-alt2 kernel: [<f89109aa>]
gfs2_scand_internal+0x17/0x22 [gfs2]
Feb 8 13:05:34 xmpp-alt2 kernel: [<f890780c>] gfs2_scand+0xc/0x30 [gfs2]
Feb 8 13:05:34 xmpp-alt2 kernel: [<c012320b>] kthread+0x9d/0xce
Feb 8 13:05:34 xmpp-alt2 kernel: [<c012316e>] kthread+0x0/0xce
Feb 8 13:05:34 xmpp-alt2 kernel: [<c01034df>]
kernel_thread_helper+0x7/0x10
Feb 8 13:05:34 xmpp-alt2 kernel: =======================
Feb 8 13:05:34 xmpp-alt2 kernel: Code: c3 65 a1 08 00 00 00 89 42 24 8b 04
24 89 42 28 89 c8 c3 0f ba 70 08 01 c7 40 24 00 00 00 00 c7 40 28 00 00 00
00 e8 7a fe ff ff <0f> 0b eb fe 55 89 d5 57 89 c7 56 31 f6 53 8d 58 3c 83 ec
04 8b
Feb 8 13:05:34 xmpp-alt2 kernel: EIP: [<f890f11e>]
gfs2_glmutex_unlock+0x18/0x1c [gfs2] SS:ESP 0068:f34f5f98
--
pozdrawiam,
Zbyszek ???kiewski
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20070208/9fb2115b/attachment.htm>
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20.
2007-02-08 12:15 [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20 Zbyszek Żółkiewski
@ 2007-02-08 12:57 ` Steven Whitehouse
[not found] ` <f7ca4caa0702080502m723d4bc3r3dbdb9302a942bff@mail.gmail.com>
0 siblings, 1 reply; 11+ messages in thread
From: Steven Whitehouse @ 2007-02-08 12:57 UTC (permalink / raw)
To: cluster-devel.redhat.com
Hi,
On Thu, 2007-02-08 at 13:15 +0100, Zbyszek ???kiewski wrote:
> Hi,
>
> I have recently setup cluster on Debian systems and i got issues
> related with kernel panic:
> systems affected: Debian 4.0 (testing) and Debian 3.1 (r4)
> gcc: 3.4 and 4.1.2
> kernels: 2.6.19 and 2.6.20
> cluster from latest cvs
>
> command invoked: mount -t gfs2 /dev/sdb1 /mnt/
> docs that i followed :
> http://sources.redhat.com/cgi-bin/cvsweb.cgi/cluster/doc/usage.txt?cvsroot=cluster
>
> any clue?
>
It looks like the mount was successful, but that the problem occurred
right after mount. Just as a sanity check does the same thing happen if
you use lock_nolock?
There have been a number of bug fixes since 2.6.20 which went into
Linus' kernel yesterday, so you might want to try the latest upstream
kernel, but I don't recognise this problem as being something we've seen
before,
Steve.
> here are detailed logs:
>
> Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: add member 1
> Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: total members 1 error 0
> Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: dlm_recover_directory
> Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: dlm_recover_directory 0
> entries
> Feb 8 13:05:18 xmpp-alt2 kernel: dlm: test: recover 1 done: 0 ms
> Feb 8 13:05:18 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: Joined
> cluster. Now mounting FS...
> Feb 8 13:05:21 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: jid=0,
> already locked for use
> Feb 8 13:05:21 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: jid=0:
> Looking at journal...
> Feb 8 13:05:21 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2: test.0: jid=0:
> Done
> Feb 8 13:05:21 xmpp-alt2 kernel: ------------[ cut here ]------------
> Feb 8 13:05:21 xmpp-alt2 kernel: kernel BUG at fs/gfs2/glock.c:738!
> Feb 8 13:05:21 xmpp-alt2 kernel: invalid opcode: 0000 [#1]
> Feb 8 13:05:21 xmpp-alt2 kernel: Modules linked in: lock_dlm dlm
> configfs gfs2
> Feb 8 13:05:21 xmpp-alt2 kernel: CPU: 0
> Feb 8 13:05:21 xmpp-alt2 kernel: EIP: 0060:[<f890f11e>] Not
> tainted VLI
> Feb 8 13:05:21 xmpp-alt2 kernel: EFLAGS: 00000282 (2.6.20-xmpp2 #1)
> Feb 8 13:05:21 xmpp-alt2 kernel: EIP is at gfs2_glmutex_unlock
> +0x18/0x1c [gfs2]
> Feb 8 13:05:21 xmpp-alt2 kernel: eax: f3aa1bbc ebx: f3aa1bec ecx:
> 00007080 edx: f3aa1bbc
> Feb 8 13:05:22 xmpp-alt2 kernel: esi: f3aa1b78 edi: f15a3f88 ebp:
> f15a3f94 esp: f15a3f64
> Feb 8 13:05:22 xmpp-alt2 kernel: ds: 007b es: 007b ss: 0068
> Feb 8 13:05:22 xmpp-alt2 kernel: Process gfs2_glockd (pid: 7013,
> ti=f15a2000 task=f3ad9a70 task.ti=f15a2000)
> Feb 8 13:05:22 xmpp-alt2 kernel: Stack: f89108e5 f34f6000 f34f6364
> f8907861 00000000 f3ad9a70 c0123580 f15a3fa0
> Feb 8 13:05:22 xmpp-alt2 kernel: f15a3fa0 f15a3fac c010f84f
> 00000000 00000000 f3ad9a70 c0123580 f15a3fa0
> Feb 8 13:05:22 xmpp-alt2 kernel: f15a3fa0 000004c1 f9a6da64
> 0024d65b f348ddc4 f34f6000 f8907830 fffffffc
> Feb 8 13:05:22 xmpp-alt2 kernel: Call Trace:
> Feb 8 13:05:22 xmpp-alt2 kernel: [<f89108e5>] gfs2_reclaim_glock
> +0x8d/0x8f [gfs2]
> Feb 8 13:05:22 xmpp-alt2 kernel: [<f8907861>] gfs2_glockd+0x31/0xe4
> [gfs2]
> Feb 8 13:05:22 xmpp-alt2 kernel: [<c0123580>]
> autoremove_wake_function+0x0/0x43
> Feb 8 13:05:22 xmpp-alt2 kernel: [<c010f84f>] __wake_up_common
> +0x33/0x56
> Feb 8 13:05:22 xmpp-alt2 kernel: [<c0123580>]
> autoremove_wake_function+0x0/0x43
> Feb 8 13:05:22 xmpp-alt2 kernel: [<f8907830>] gfs2_glockd+0x0/0xe4
> [gfs2]
> Feb 8 13:05:22 xmpp-alt2 kernel: [<c012320b>] kthread+0x9d/0xce
> Feb 8 13:05:22 xmpp-alt2 kernel: [<c012316e>] kthread+0x0/0xce
> Feb 8 13:05:22 xmpp-alt2 kernel: [<c01034df>] kernel_thread_helper
> +0x7/0x10
> Feb 8 13:05:22 xmpp-alt2 kernel: =======================
> Feb 8 13:05:22 xmpp-alt2 kernel: Code: c3 65 a1 08 00 00 00 89 42 24
> 8b 04 24 89 42 28 89 c8 c3 0f ba 70 08 01 c7 40 24 00 00 00 00 c7 40
> 28 00 00 00 00 e8 7a fe ff ff <0f> 0b eb fe 55 89 d5 57 89 c7 56 31 f6
> 53 8d 58 3c 83 ec 04 8b
> Feb 8 13:05:22 xmpp-alt2 kernel: EIP: [<f890f11e>]
> gfs2_glmutex_unlock+0x18/0x1c [gfs2] SS:ESP 0068:f15a3f64
> Feb 8 13:05:33 xmpp-alt2 kernel: <0>------------[ cut
> here ]------------
> Feb 8 13:05:33 xmpp-alt2 kernel: kernel BUG at fs/gfs2/glock.c:738!
> Feb 8 13:05:33 xmpp-alt2 kernel: invalid opcode: 0000 [#2]
> Feb 8 13:05:33 xmpp-alt2 kernel: Modules linked in: lock_dlm dlm
> configfs gfs2
> Feb 8 13:05:33 xmpp-alt2 kernel: CPU: 0
> Feb 8 13:05:33 xmpp-alt2 kernel: EIP: 0060:[<f890f11e>] Not
> tainted VLI
> Feb 8 13:05:33 xmpp-alt2 kernel: EFLAGS: 00000282 (2.6.20-xmpp2 #1)
> Feb 8 13:05:34 xmpp-alt2 kernel: EIP is at gfs2_glmutex_unlock
> +0x18/0x1c [gfs2]
> Feb 8 13:05:34 xmpp-alt2 kernel: eax: f3aa1bbc ebx: f3aa1b78 ecx:
> 00007080 edx: f3aa1bbc
> Feb 8 13:05:34 xmpp-alt2 kernel: esi: f34f6000 edi: f3aa1b78 ebp:
> 00000001 esp: f34f5f98
> Feb 8 13:05:34 xmpp-alt2 kernel: ds: 007b es: 007b ss: 0068
> Feb 8 13:05:34 xmpp-alt2 kernel: Process gfs2_scand (pid: 7012,
> ti=f34f4000 task=f3bef030 task.ti=f34f4000)
> Feb 8 13:05:34 xmpp-alt2 kernel: Stack: f8910940 f8910942 000002be
> f34f6000 f8907800 fffffffc f89109aa f34f6000
> Feb 8 13:05:34 xmpp-alt2 kernel: f34f6000 f890780c f348ddc4
> c012320b 00000001 ffffffff ffffffff c012316e
> Feb 8 13:05:34 xmpp-alt2 kernel: 00000000 00000000 00000000
> c01034df f348ddbc 00000000 00000000 00000000
> Feb 8 13:05:34 xmpp-alt2 kernel: Call Trace:
> Feb 8 13:05:34 xmpp-alt2 kernel: [<f8910940>] examine_bucket
> +0x59/0x5b [gfs2]
> Feb 8 13:05:34 xmpp-alt2 kernel: [<f8910942>] scan_glock+0x0/0x51
> [gfs2]
> Feb 8 13:05:34 xmpp-alt2 kernel: [<f8907800>] gfs2_scand+0x0/0x30
> [gfs2]
> Feb 8 13:05:34 xmpp-alt2 kernel: [<f89109aa>] gfs2_scand_internal
> +0x17/0x22 [gfs2]
> Feb 8 13:05:34 xmpp-alt2 kernel: [<f890780c>] gfs2_scand+0xc/0x30
> [gfs2]
> Feb 8 13:05:34 xmpp-alt2 kernel: [<c012320b>] kthread+0x9d/0xce
> Feb 8 13:05:34 xmpp-alt2 kernel: [<c012316e>] kthread+0x0/0xce
> Feb 8 13:05:34 xmpp-alt2 kernel: [<c01034df>] kernel_thread_helper
> +0x7/0x10
> Feb 8 13:05:34 xmpp-alt2 kernel: =======================
> Feb 8 13:05:34 xmpp-alt2 kernel: Code: c3 65 a1 08 00 00 00 89 42 24
> 8b 04 24 89 42 28 89 c8 c3 0f ba 70 08 01 c7 40 24 00 00 00 00 c7 40
> 28 00 00 00 00 e8 7a fe ff ff <0f> 0b eb fe 55 89 d5 57 89 c7 56 31 f6
> 53 8d 58 3c 83 ec 04 8b
> Feb 8 13:05:34 xmpp-alt2 kernel: EIP: [<f890f11e>]
> gfs2_glmutex_unlock+0x18/0x1c [gfs2] SS:ESP 0068:f34f5f98
>
>
> --
> pozdrawiam,
> Zbyszek ???kiewski
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20.
[not found] ` <f7ca4caa0702080502m723d4bc3r3dbdb9302a942bff@mail.gmail.com>
@ 2007-02-08 13:04 ` Zbyszek Żółkiewski
2007-02-08 13:26 ` Steven Whitehouse
0 siblings, 1 reply; 11+ messages in thread
From: Zbyszek Żółkiewski @ 2007-02-08 13:04 UTC (permalink / raw)
To: cluster-devel.redhat.com
sorry - mail went only to Steven, now to group....
On 2/8/07, Zbyszek ???kiewski <zbyszek@toliman.pl> wrote:
>
> well, thanks for answer, i have tried with nolock, and result is as
> follow:
> of course i made mkfs -t gfs2 -p lock_nolock -t xmpp-alt2:test -j 1
> /dev/sdb1 and then:
> mount -t gfs2 /dev/sdb1 /mnt -v
>
> and yes - the device is mounted,
> (the changes to kernel you was talking about: you mean: git1 for 2.6.20?)
>
>
> Feb 8 13:52:46 xmpp-alt2 kernel: Lock_Nolock (built Feb 8 2007 13:52:20)
> installed
> Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=: Trying to join cluster
> "lock_nolock", "xmpp-alt2:test"
> Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: Joined
> cluster. Now mounting FS...
> Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: jid=0,
> already locked for use
> Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2: test.0: jid=0:
> Looking at journal...
> Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: jid=0: Done
> Feb 8 13:58:13 xmpp-alt2 kernel: ------------[ cut here ]------------
> Feb 8 13:58:13 xmpp-alt2 kernel: kernel BUG at fs/gfs2/glock.c:738!
> Feb 8 13:58:13 xmpp-alt2 kernel: invalid opcode: 0000 [#1]
> Feb 8 13:58:13 xmpp-alt2 kernel: Modules linked in: lock_nolock lock_dlm
> gfs2 dlm configfs
> Feb 8 13:58:13 xmpp-alt2 kernel: CPU: 0
> Feb 8 13:58:13 xmpp-alt2 kernel: EIP: 0060:[<f895011e>] Not tainted
> VLI
> Feb 8 13:58:13 xmpp-alt2 kernel: EFLAGS: 00000282 (2.6.20-xmpp2 #1)
> Feb 8 13:58:13 xmpp-alt2 kernel: EIP is at gfs2_glmutex_unlock+0x18/0x1c
> [gfs2]
> Feb 8 13:58:13 xmpp-alt2 kernel: eax: f5085bbc ebx: f5085bec ecx:
> f5875000 edx: f5085bbc
> Feb 8 13:58:13 xmpp-alt2 kernel: esi: f5085b78 edi: f575ff88 ebp:
> f575ff94 esp: f575ff64
> Feb 8 13:58:13 xmpp-alt2 kernel: ds: 007b es: 007b ss: 0068
> Feb 8 13:58:13 xmpp-alt2 kernel: Process gfs2_glockd (pid: 2645,
> ti=f575e000 task=f58d3550 task.ti=f575e000)
> Feb 8 13:58:13 xmpp-alt2 kernel: Stack: f89518e5 f5875000 f5875364
> f8948861 00000000 f58d3550 c0123580 f575ffa0
> Feb 8 13:58:13 xmpp-alt2 kernel: f575ffa0 f575ffac c010f84f
> 00000000 00000000 f58d3550 c0123580 f575ffa0
> Feb 8 13:58:13 xmpp-alt2 kernel: f575ffa0 000004f8 b716dc5d
> 0024d983 f5011dc4 f5875000 f8948830 fffffffc
> Feb 8 13:58:13 xmpp-alt2 kernel: Call Trace:
> Feb 8 13:58:13 xmpp-alt2 kernel: [<f89518e5>]
> gfs2_reclaim_glock+0x8d/0x8f [gfs2]
> Feb 8 13:58:13 xmpp-alt2 kernel: [<f8948861>] gfs2_glockd+0x31/0xe4
> [gfs2]
> Feb 8 13:58:13 xmpp-alt2 kernel: [<c0123580>]
> autoremove_wake_function+0x0/0x43
> Feb 8 13:58:13 xmpp-alt2 kernel: [<c010f84f>] __wake_up_common+0x33/0x56
>
> Feb 8 13:58:13 xmpp-alt2 kernel: [<c0123580>]
> autoremove_wake_function+0x0/0x43
> Feb 8 13:58:13 xmpp-alt2 kernel: [<f8948830>] gfs2_glockd+0x0/0xe4
> [gfs2]
> Feb 8 13:58:13 xmpp-alt2 kernel: [<c012320b>] kthread+0x9d/0xce
> Feb 8 13:58:13 xmpp-alt2 kernel: [<c012316e>] kthread+0x0/0xce
> Feb 8 13:58:13 xmpp-alt2 kernel: [<c01034df>]
> kernel_thread_helper+0x7/0x10
> Feb 8 13:58:13 xmpp-alt2 kernel: =======================
> Feb 8 13:58:13 xmpp-alt2 kernel: Code: c3 65 a1 08 00 00 00 89 42 24 8b
> 04 24 89 42 28 89 c8 c3 0f ba 70 08 01 c7 40 24 00 00 00 00 c7 40 28 00 00
> 00 00 e8 7a fe ff ff <0f> 0b eb fe 55 89 d5 57 89 c7 56 31 f6 53 8d 58 3c 83
> ec 04 8b
> Feb 8 13:58:13 xmpp-alt2 kernel: EIP: [<f895011e>]
> gfs2_glmutex_unlock+0x18/0x1c [gfs2] SS:ESP 0068:f575ff64
> Feb 8 13:58:28 xmpp-alt2 kernel: <0>------------[ cut here ]------------
> Feb 8 13:58:28 xmpp-alt2 kernel: kernel BUG at fs/gfs2/glock.c:738!
> Feb 8 13:58:28 xmpp-alt2 kernel: invalid opcode: 0000 [#2]
> Feb 8 13:58:28 xmpp-alt2 kernel: Modules linked in: lock_nolock lock_dlm
> gfs2 dlm configfs
> Feb 8 13:58:28 xmpp-alt2 kernel: CPU: 0
> Feb 8 13:58:28 xmpp-alt2 kernel: EIP: 0060:[<f895011e>] Not tainted
> VLI
> Feb 8 13:58:28 xmpp-alt2 kernel: EFLAGS: 00000282 (2.6.20-xmpp2 #1)
> Feb 8 13:58:28 xmpp-alt2 kernel: EIP is at gfs2_glmutex_unlock+0x18/0x1c
> [gfs2]
> Feb 8 13:58:28 xmpp-alt2 kernel: eax: f5085bbc ebx: f5085b78 ecx:
> f5875000 edx: f5085bbc
> Feb 8 13:58:28 xmpp-alt2 kernel: esi: f5875000 edi: f5085b78 ebp:
> 00000001 esp: f577df98
> Feb 8 13:58:28 xmpp-alt2 kernel: ds: 007b es: 007b ss: 0068
> Feb 8 13:58:28 xmpp-alt2 kernel: Process gfs2_scand (pid: 2644,
> ti=f577c000 task=f58d3a70 task.ti=f577c000)
> Feb 8 13:58:28 xmpp-alt2 kernel: Stack: f8951940 f8951942 000001e0
> f5875000 f8948800 fffffffc f89519aa f5875000
> Feb 8 13:58:28 xmpp-alt2 kernel: f5875000 f894880c f5011dc4
> c012320b 00000001 ffffffff ffffffff c012316e
> Feb 8 13:58:28 xmpp-alt2 kernel: 00000000 00000000 00000000
> c01034df f5011dbc 00000000 00000000 00000000
> Feb 8 13:58:28 xmpp-alt2 kernel: Call Trace:
> Feb 8 13:58:28 xmpp-alt2 kernel: [<f8951940>] examine_bucket+0x59/0x5b
> [gfs2]
> Feb 8 13:58:28 xmpp-alt2 kernel: [<f8951942>] scan_glock+0x0/0x51 [gfs2]
> Feb 8 13:58:28 xmpp-alt2 kernel: [<f8948800>] gfs2_scand+0x0/0x30 [gfs2]
> Feb 8 13:58:28 xmpp-alt2 kernel: [<f89519aa>]
> gfs2_scand_internal+0x17/0x22 [gfs2]
> Feb 8 13:58:28 xmpp-alt2 kernel: [<f894880c>] gfs2_scand+0xc/0x30 [gfs2]
> Feb 8 13:58:28 xmpp-alt2 kernel: [<c012320b>] kthread+0x9d/0xce
> Feb 8 13:58:28 xmpp-alt2 kernel: [<c012316e>] kthread+0x0/0xce
> Feb 8 13:58:28 xmpp-alt2 kernel: [<c01034df>]
> kernel_thread_helper+0x7/0x10
> Feb 8 13:58:28 xmpp-alt2 kernel: =======================
> Feb 8 13:58:28 xmpp-alt2 kernel: Code: c3 65 a1 08 00 00 00 89 42 24 8b
> 04 24 89 42 28 89 c8 c3 0f ba 70 08 01 c7 40 24 00 00 00 00 c7 40 28 00 00
> 00 00 e8 7a fe ff ff <0f> 0b eb fe 55 89 d5 57 89 c7 56 31 f6 53 8d 58 3c 83
> ec 04 8b
> Feb 8 13:58:28 xmpp-alt2 kernel: EIP: [<f895011e>]
> gfs2_glmutex_unlock+0x18/0x1c [gfs2] SS:ESP 0068:f577df98
>
>
>
>
>
> On 2/8/07, Steven Whitehouse <swhiteho@redhat.com> wrote:
> >
> > Hi,
> >
> > On Thu, 2007-02-08 at 13:15 +0100, Zbyszek ???kiewski wrote:
> > > Hi,
> > >
> > > I have recently setup cluster on Debian systems and i got issues
> > > related with kernel panic:
> > > systems affected: Debian 4.0 (testing) and Debian 3.1 (r4)
> > > gcc: 3.4 and 4.1.2
> > > kernels: 2.6.19 and 2.6.20
> > > cluster from latest cvs
> > >
> > > command invoked: mount -t gfs2 /dev/sdb1 /mnt/
> > > docs that i followed :
> > >
> > http://sources.redhat.com/cgi-bin/cvsweb.cgi/cluster/doc/usage.txt?cvsroot=cluster
> > >
> > > any clue?
> > >
> > It looks like the mount was successful, but that the problem occurred
> > right after mount. Just as a sanity check does the same thing happen if
> > you use lock_nolock?
> >
> > There have been a number of bug fixes since 2.6.20 which went into
> > Linus' kernel yesterday, so you might want to try the latest upstream
> > kernel, but I don't recognise this problem as being something we've seen
> > before,
> >
> > Steve.
> >
> >
>
> --
> pozdrawiam,
> Zbyszek ???kiewski
--
pozdrawiam,
Zbyszek ???kiewski
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20070208/84892e46/attachment.htm>
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20.
2007-02-08 13:04 ` Zbyszek Żółkiewski
@ 2007-02-08 13:26 ` Steven Whitehouse
2007-02-08 14:02 ` Zbyszek Żółkiewski
0 siblings, 1 reply; 11+ messages in thread
From: Steven Whitehouse @ 2007-02-08 13:26 UTC (permalink / raw)
To: cluster-devel.redhat.com
Hi,
On Thu, 2007-02-08 at 14:04 +0100, Zbyszek ???kiewski wrote:
> sorry - mail went only to Steven, now to group....
>
>
> On 2/8/07, Zbyszek ???kiewski <zbyszek@toliman.pl> wrote:
> well, thanks for answer, i have tried with nolock, and result
> is as follow:
> of course i made mkfs -t gfs2 -p lock_nolock -t xmpp-alt2:test
> -j 1 /dev/sdb1 and then:
> mount -t gfs2 /dev/sdb1 /mnt -v
>
> and yes - the device is mounted,
It looks like what is happening is that a glmutex_unlock() is
discovering that its spinlock has been dropped by glock.c:run_queue()
which should be impossible, so something odd is happening here I think.
The daemons implicated in this are there to demote unused locks on a
periodic basis, so its presumably one of the locks used during mounting
of the filesystem thats at fault.
> (the changes to kernel you was talking about: you mean: git1
> for 2.6.20?)
>
I'm not sure if its in git1 or not, I suspect it will be git2 since it
was only yesterday that the patches went in. Linus' current git tree
seems to be broken (both gitweb and direct via the git tools) otherwise
I'd post a URL to the changes. In the mean time you can find them in my
-nmw tree which will get updated just as soon as git it working again at
kernel.org,
Steve.
>
> Feb 8 13:52:46 xmpp-alt2 kernel: Lock_Nolock (built Feb 8
> 2007 13:52:20) installed
> Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=: Trying to join
> cluster "lock_nolock", "xmpp-alt2:test"
> Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0:
> Joined cluster. Now mounting FS...
> Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0:
> jid=0, already locked for use
> Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:
> test.0: jid=0: Looking at journal...
> Feb 8 13:58:13 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0:
> jid=0: Done
> Feb 8 13:58:13 xmpp-alt2 kernel: ------------[ cut
> here ]------------
> Feb 8 13:58:13 xmpp-alt2 kernel: kernel BUG at
> fs/gfs2/glock.c:738!
> Feb 8 13:58:13 xmpp-alt2 kernel: invalid opcode: 0000 [#1]
> Feb 8 13:58:13 xmpp-alt2 kernel: Modules linked in:
> lock_nolock lock_dlm gfs2 dlm configfs
> Feb 8 13:58:13 xmpp-alt2 kernel: CPU: 0
> Feb 8 13:58:13 xmpp-alt2 kernel: EIP: 0060:[<f895011e>]
> Not tainted VLI
> Feb 8 13:58:13 xmpp-alt2 kernel: EFLAGS: 00000282
> (2.6.20-xmpp2 #1)
> Feb 8 13:58:13 xmpp-alt2 kernel: EIP is at
> gfs2_glmutex_unlock+0x18/0x1c [gfs2]
> Feb 8 13:58:13 xmpp-alt2 kernel: eax: f5085bbc ebx:
> f5085bec ecx: f5875000 edx: f5085bbc
> Feb 8 13:58:13 xmpp-alt2 kernel: esi: f5085b78 edi:
> f575ff88 ebp: f575ff94 esp: f575ff64
> Feb 8 13:58:13 xmpp-alt2 kernel: ds: 007b es: 007b ss:
> 0068
> Feb 8 13:58:13 xmpp-alt2 kernel: Process gfs2_glockd (pid:
> 2645, ti=f575e000 task=f58d3550 task.ti=f575e000)
> Feb 8 13:58:13 xmpp-alt2 kernel: Stack: f89518e5 f5875000
> f5875364 f8948861 00000000 f58d3550 c0123580 f575ffa0
> Feb 8 13:58:13 xmpp-alt2 kernel: f575ffa0 f575ffac
> c010f84f 00000000 00000000 f58d3550 c0123580 f575ffa0
> Feb 8 13:58:13 xmpp-alt2 kernel: f575ffa0 000004f8
> b716dc5d 0024d983 f5011dc4 f5875000 f8948830 fffffffc
> Feb 8 13:58:13 xmpp-alt2 kernel: Call Trace:
> Feb 8 13:58:13 xmpp-alt2 kernel: [<f89518e5>]
> gfs2_reclaim_glock+0x8d/0x8f [gfs2]
> Feb 8 13:58:13 xmpp-alt2 kernel: [<f8948861>] gfs2_glockd
> +0x31/0xe4 [gfs2]
> Feb 8 13:58:13 xmpp-alt2 kernel: [<c0123580>]
> autoremove_wake_function+0x0/0x43
> Feb 8 13:58:13 xmpp-alt2 kernel: [<c010f84f>]
> __wake_up_common+0x33/0x56
> Feb 8 13:58:13 xmpp-alt2 kernel: [<c0123580>]
> autoremove_wake_function+0x0/0x43
> Feb 8 13:58:13 xmpp-alt2 kernel: [<f8948830>] gfs2_glockd
> +0x0/0xe4 [gfs2]
> Feb 8 13:58:13 xmpp-alt2 kernel: [<c012320b>] kthread
> +0x9d/0xce
> Feb 8 13:58:13 xmpp-alt2 kernel: [<c012316e>] kthread
> +0x0/0xce
> Feb 8 13:58:13 xmpp-alt2 kernel: [<c01034df>]
> kernel_thread_helper+0x7/0x10
> Feb 8 13:58:13 xmpp-alt2 kernel: =======================
> Feb 8 13:58:13 xmpp-alt2 kernel: Code: c3 65 a1 08 00 00 00
> 89 42 24 8b 04 24 89 42 28 89 c8 c3 0f ba 70 08 01 c7 40 24 00
> 00 00 00 c7 40 28 00 00 00 00 e8 7a fe ff ff <0f> 0b eb fe 55
> 89 d5 57 89 c7 56 31 f6 53 8d 58 3c 83 ec 04 8b
> Feb 8 13:58:13 xmpp-alt2 kernel: EIP: [<f895011e>]
> gfs2_glmutex_unlock+0x18/0x1c [gfs2] SS:ESP 0068:f575ff64
> Feb 8 13:58:28 xmpp-alt2 kernel: <0>------------[ cut
> here ]------------
> Feb 8 13:58:28 xmpp-alt2 kernel: kernel BUG at
> fs/gfs2/glock.c:738!
> Feb 8 13:58:28 xmpp-alt2 kernel: invalid opcode: 0000 [#2]
> Feb 8 13:58:28 xmpp-alt2 kernel: Modules linked in:
> lock_nolock lock_dlm gfs2 dlm configfs
> Feb 8 13:58:28 xmpp-alt2 kernel: CPU: 0
> Feb 8 13:58:28 xmpp-alt2 kernel: EIP: 0060:[<f895011e>]
> Not tainted VLI
> Feb 8 13:58:28 xmpp-alt2 kernel: EFLAGS: 00000282
> (2.6.20-xmpp2 #1)
> Feb 8 13:58:28 xmpp-alt2 kernel: EIP is at
> gfs2_glmutex_unlock+0x18/0x1c [gfs2]
> Feb 8 13:58:28 xmpp-alt2 kernel: eax: f5085bbc ebx:
> f5085b78 ecx: f5875000 edx: f5085bbc
> Feb 8 13:58:28 xmpp-alt2 kernel: esi: f5875000 edi:
> f5085b78 ebp: 00000001 esp: f577df98
> Feb 8 13:58:28 xmpp-alt2 kernel: ds: 007b es: 007b ss:
> 0068
> Feb 8 13:58:28 xmpp-alt2 kernel: Process gfs2_scand (pid:
> 2644, ti=f577c000 task=f58d3a70 task.ti=f577c000)
> Feb 8 13:58:28 xmpp-alt2 kernel: Stack: f8951940 f8951942
> 000001e0 f5875000 f8948800 fffffffc f89519aa f5875000
> Feb 8 13:58:28 xmpp-alt2 kernel: f5875000 f894880c
> f5011dc4 c012320b 00000001 ffffffff ffffffff c012316e
> Feb 8 13:58:28 xmpp-alt2 kernel: 00000000 00000000
> 00000000 c01034df f5011dbc 00000000 00000000 00000000
> Feb 8 13:58:28 xmpp-alt2 kernel: Call Trace:
> Feb 8 13:58:28 xmpp-alt2 kernel: [<f8951940>] examine_bucket
> +0x59/0x5b [gfs2]
> Feb 8 13:58:28 xmpp-alt2 kernel: [<f8951942>] scan_glock
> +0x0/0x51 [gfs2]
> Feb 8 13:58:28 xmpp-alt2 kernel: [<f8948800>] gfs2_scand
> +0x0/0x30 [gfs2]
> Feb 8 13:58:28 xmpp-alt2 kernel: [<f89519aa>]
> gfs2_scand_internal+0x17/0x22 [gfs2]
> Feb 8 13:58:28 xmpp-alt2 kernel: [<f894880c>] gfs2_scand
> +0xc/0x30 [gfs2]
> Feb 8 13:58:28 xmpp-alt2 kernel: [<c012320b>] kthread
> +0x9d/0xce
> Feb 8 13:58:28 xmpp-alt2 kernel: [<c012316e>] kthread
> +0x0/0xce
> Feb 8 13:58:28 xmpp-alt2 kernel: [<c01034df>]
> kernel_thread_helper+0x7/0x10
> Feb 8 13:58:28 xmpp-alt2 kernel: =======================
> Feb 8 13:58:28 xmpp-alt2 kernel: Code: c3 65 a1 08 00 00 00
> 89 42 24 8b 04 24 89 42 28 89 c8 c3 0f ba 70 08 01 c7 40 24 00
> 00 00 00 c7 40 28 00 00 00 00 e8 7a fe ff ff <0f> 0b eb fe 55
> 89 d5 57 89 c7 56 31 f6 53 8d 58 3c 83 ec 04 8b
> Feb 8 13:58:28 xmpp-alt2 kernel: EIP: [<f895011e>]
> gfs2_glmutex_unlock+0x18/0x1c [gfs2] SS:ESP 0068:f577df98
>
>
>
>
>
> On 2/8/07, Steven Whitehouse <swhiteho@redhat.com> wrote:
> Hi,
>
> On Thu, 2007-02-08 at 13:15 +0100, Zbyszek ???kiewski
> wrote:
> > Hi,
> >
> > I have recently setup cluster on Debian systems and
> i got issues
> > related with kernel panic:
> > systems affected: Debian 4.0 (testing) and Debian
> 3.1 (r4)
> > gcc: 3.4 and 4.1.2
> > kernels: 2.6.19 and 2.6.20
> > cluster from latest cvs
> >
> > command invoked: mount -t gfs2 /dev/sdb1 /mnt/
> > docs that i followed :
> >
> http://sources.redhat.com/cgi-bin/cvsweb.cgi/cluster/doc/usage.txt?cvsroot=cluster
> >
> > any clue?
> >
> It looks like the mount was successful, but that the
> problem occurred
> right after mount. Just as a sanity check does the
> same thing happen if
> you use lock_nolock?
>
> There have been a number of bug fixes since 2.6.20
> which went into
> Linus' kernel yesterday, so you might want to try the
> latest upstream
> kernel, but I don't recognise this problem as being
> something we've seen
> before,
>
> Steve.
>
>
>
> --
> pozdrawiam,
> Zbyszek ???kiewski
>
>
>
> --
> pozdrawiam,
> Zbyszek ???kiewski
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20.
2007-02-08 13:26 ` Steven Whitehouse
@ 2007-02-08 14:02 ` Zbyszek Żółkiewski
2007-02-08 16:26 ` Steven Whitehouse
0 siblings, 1 reply; 11+ messages in thread
From: Zbyszek Żółkiewski @ 2007-02-08 14:02 UTC (permalink / raw)
To: cluster-devel.redhat.com
ok , i have build kernel from your git (
git://git.kernel.org/pub/scm/linux/kernel/git/steve/gfs2-2.6-nmw.git)
Linux version 2.6.20-xmpp2-ga2cf8222-dirty (root at xmpp-alt2) (gcc version
3.4.4 20050314 (prerelease) (Debian 3.4.3-13sarge1)) #1 Thu Feb 8 14:51:21
CET 2007
and there is the same problem;
kernel BUG at fs/gfs2/glock.c:704!
invalid opcode: 0000 [#2]
Modules linked in: lock_nolock lock_dlm gfs2 dlm configfs
CPU: 0
EIP: 0060:[<f8950173>] Not tainted VLI
EFLAGS: 00000282 (2.6.20-xmpp2-ga2cf8222-dirty #1)
EIP is at gfs2_glmutex_unlock+0x18/0x1c [gfs2]
and so one....
ok so waiting patiently for solution....
On 2/8/07, Steven Whitehouse <swhiteho@redhat.com> wrote:
>
> Hi,
>
> On Thu, 2007-02-08 at 14:04 +0100, Zbyszek ???kiewski wrote:
> > sorry - mail went only to Steven, now to group....
> >
> >
> > On 2/8/07, Zbyszek ???kiewski <zbyszek@toliman.pl> wrote:
> > well, thanks for answer, i have tried with nolock, and result
> > is as follow:
> > of course i made mkfs -t gfs2 -p lock_nolock -t xmpp-alt2:test
> > -j 1 /dev/sdb1 and then:
> > mount -t gfs2 /dev/sdb1 /mnt -v
> >
> > and yes - the device is mounted,
> It looks like what is happening is that a glmutex_unlock() is
> discovering that its spinlock has been dropped by glock.c:run_queue()
> which should be impossible, so something odd is happening here I think.
>
> The daemons implicated in this are there to demote unused locks on a
> periodic basis, so its presumably one of the locks used during mounting
> of the filesystem thats at fault.
>
> > (the changes to kernel you was talking about: you mean: git1
> > for 2.6.20?)
> >
> I'm not sure if its in git1 or not, I suspect it will be git2 since it
> was only yesterday that the patches went in. Linus' current git tree
> seems to be broken (both gitweb and direct via the git tools) otherwise
> I'd post a URL to the changes. In the mean time you can find them in my
> -nmw tree which will get updated just as soon as git it working again at
> kernel.org,
>
> Steve.
>
>
>
--
pozdrawiam,
Zbyszek ???kiewski
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20070208/748a6f60/attachment.htm>
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20.
2007-02-08 14:02 ` Zbyszek Żółkiewski
@ 2007-02-08 16:26 ` Steven Whitehouse
2007-02-08 17:21 ` Zbyszek Żółkiewski
0 siblings, 1 reply; 11+ messages in thread
From: Steven Whitehouse @ 2007-02-08 16:26 UTC (permalink / raw)
To: cluster-devel.redhat.com
Hi,
I cannot reproduce what you are seeing, but try the attached patch to
see if we can narrow this down,
Steve.
On Thu, 2007-02-08 at 15:02 +0100, Zbyszek ???kiewski wrote:
> ok , i have build kernel from your git (
> git://git.kernel.org/pub/scm/linux/kernel/git/steve/gfs2-2.6-nmw.git)
> Linux version 2.6.20-xmpp2-ga2cf8222-dirty (root at xmpp-alt2) (gcc
> version 3.4.4 20050314 (prerelease) (Debian 3.4.3-13sarge1)) #1 Thu
> Feb 8 14:51:21 CET 2007
>
> and there is the same problem;
> kernel BUG at fs/gfs2/glock.c:704!
> invalid opcode: 0000 [#2]
> Modules linked in: lock_nolock lock_dlm gfs2 dlm configfs
> CPU: 0
> EIP: 0060:[<f8950173>] Not tainted VLI
> EFLAGS: 00000282 (2.6.20-xmpp2-ga2cf8222-dirty #1)
> EIP is at gfs2_glmutex_unlock+0x18/0x1c [gfs2]
>
> and so one....
>
> ok so waiting patiently for solution....
>
>
>
>
> On 2/8/07, Steven Whitehouse <swhiteho@redhat.com> wrote:
> Hi,
>
> On Thu, 2007-02-08 at 14:04 +0100, Zbyszek ???kiewski wrote:
> > sorry - mail went only to Steven, now to group....
> >
> >
> > On 2/8/07, Zbyszek ???kiewski <zbyszek@toliman.pl> wrote:
> > well, thanks for answer, i have tried with nolock,
> and result
> > is as follow:
> > of course i made mkfs -t gfs2 -p lock_nolock -t
> xmpp-alt2:test
> > -j 1 /dev/sdb1 and then:
> > mount -t gfs2 /dev/sdb1 /mnt -v
> >
> > and yes - the device is mounted,
> It looks like what is happening is that a glmutex_unlock() is
> discovering that its spinlock has been dropped by
> glock.c:run_queue()
> which should be impossible, so something odd is happening here
> I think.
>
> The daemons implicated in this are there to demote unused
> locks on a
> periodic basis, so its presumably one of the locks used during
> mounting
> of the filesystem thats at fault.
>
> > (the changes to kernel you was talking about: you
> mean: git1
> > for 2.6.20?)
> >
> I'm not sure if its in git1 or not, I suspect it will be git2
> since it
> was only yesterday that the patches went in. Linus' current
> git tree
> seems to be broken (both gitweb and direct via the git tools)
> otherwise
> I'd post a URL to the changes. In the mean time you can find
> them in my
> -nmw tree which will get updated just as soon as git it
> working again at
> kernel.org,
>
> Steve.
>
>
>
>
> --
> pozdrawiam,
> Zbyszek ???kiewski
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.diff
Type: text/x-patch
Size: 1259 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20070208/ca40ccac/attachment.bin>
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20.
2007-02-08 16:26 ` Steven Whitehouse
@ 2007-02-08 17:21 ` Zbyszek Żółkiewski
2007-02-09 10:32 ` Steven Whitehouse
0 siblings, 1 reply; 11+ messages in thread
From: Zbyszek Żółkiewski @ 2007-02-08 17:21 UTC (permalink / raw)
To: cluster-devel.redhat.com
are you using the same glibc and distro?
well i have patched gfs2 and here is output:
Feb 8 18:16:28 xmpp-alt2 kernel: GFS2: fsid=xmpp-alt2:test.0: Joined
cluster. Now mounting FS...
Feb 8 18:16:28 xmpp-alt2 kernel: ------------[ cut here ]------------
Feb 8 18:16:28 xmpp-alt2 kernel: kernel BUG at fs/gfs2/glock.c:632!
Feb 8 18:16:28 xmpp-alt2 kernel: invalid opcode: 0000 [#1]
Feb 8 18:16:28 xmpp-alt2 kernel: Modules linked in: lock_nolock lock_dlm
gfs2 dlm configfs
Feb 8 18:16:28 xmpp-alt2 kernel: CPU: 0
Feb 8 18:16:28 xmpp-alt2 kernel: EIP: 0060:[<f894ff98>] Not tainted
VLI
Feb 8 18:16:28 xmpp-alt2 kernel: EFLAGS: 00000292 (2.6.20-xmpp2 #3)
Feb 8 18:16:28 xmpp-alt2 kernel: EIP is at run_queue+0x0/0x4 [gfs2]
Feb 8 18:16:28 xmpp-alt2 kernel: eax: f5f119a4 ebx: f5ef9e34 ecx:
f5f119e8 edx: f5f119e8
Feb 8 18:16:28 xmpp-alt2 kernel: esi: f5f119a4 edi: 00000000 ebp:
f5e7c000 esp: f5ef9dd4
Feb 8 18:16:28 xmpp-alt2 kernel: ds: 007b es: 007b ss: 0068
Feb 8 18:16:28 xmpp-alt2 kernel: Process mount.gfs2 (pid: 1907, ti=f5ef8000
task=f5e1b550 task.ti=f5ef8000)
Feb 8 18:16:28 xmpp-alt2 kernel: Stack: f8950d03 00000000 f5e7c000 00000000
f5ef9e34 f895101f f5ef9e34 00000001
Feb 8 18:16:28 xmpp-alt2 kernel: f5ef9df8 f5f119a4 f5eff550 f895c359
f89687a0 00000001 00000404 f5ef9e34
Feb 8 18:16:28 xmpp-alt2 kernel: 00000000 f5e7c000 f5c6ae00 00000000
f895d2bc f5ef9e7c c02d572f f5ef9e5c
Feb 8 18:16:28 xmpp-alt2 kernel: Call Trace:
Feb 8 18:16:28 xmpp-alt2 kernel: [<f8950d03>] gfs2_glock_nq+0x42/0x63
[gfs2]
Feb 8 18:16:28 xmpp-alt2 kernel: [<f895101f>] gfs2_glock_nq_num+0x4e/0x6f
[gfs2]
Feb 8 18:16:28 xmpp-alt2 kernel: [<f895c359>] init_locking+0xe1/0x2b0
[gfs2]
Feb 8 18:16:28 xmpp-alt2 kernel: [<f895d2bc>] fill_super+0xd6/0x231 [gfs2]
Feb 8 18:16:28 xmpp-alt2 kernel: [<f8951016>] gfs2_glock_nq_num+0x45/0x6f
[gfs2]
Feb 8 18:16:28 xmpp-alt2 kernel: [<c014b975>] get_sb_bdev+0xe1/0x119
Feb 8 18:16:28 xmpp-alt2 kernel: [<f895d43e>] gfs2_get_sb+0x27/0x4a [gfs2]
Feb 8 18:16:28 xmpp-alt2 kernel: [<f895d1e6>] fill_super+0x0/0x231 [gfs2]
Feb 8 18:16:28 xmpp-alt2 kernel: [<c014bb98>] vfs_kern_mount+0x4c/0x99
Feb 8 18:16:28 xmpp-alt2 kernel: [<c014bc1a>] do_kern_mount+0x35/0x50
Feb 8 18:16:28 xmpp-alt2 kernel: [<c015d4b5>] do_new_mount+0x6b/0xb7
Feb 8 18:16:28 xmpp-alt2 kernel: [<c015db41>] do_mount+0x1ca/0x1ec
Feb 8 18:16:28 xmpp-alt2 kernel: [<c0133feb>] __alloc_pages+0x54/0x2d4
Feb 8 18:16:28 xmpp-alt2 kernel: [<c015d921>] copy_mount_options+0x56/0xac
Feb 8 18:16:28 xmpp-alt2 kernel: [<c015de04>] sys_mount+0x74/0xab
Feb 8 18:16:28 xmpp-alt2 kernel: [<c01029c4>] syscall_call+0x7/0xb
Feb 8 18:16:28 xmpp-alt2 kernel: =======================
Feb 8 18:16:28 xmpp-alt2 kernel: Code: 08 8b 48 04 8b 00 89 48 04 89 01 89
5b 04 89 1b b8 07 00 00 00 0f b3 42 08 89 d8 e8 5b fd ff ff 89 d8 e8 07 79
7f c7 31 c0 5b c3 <0f> 0b eb fe 56 31 c9 31 d2 53 89 c3 83 ec 34 8d 74 24 04
89 34
Feb 8 18:16:28 xmpp-alt2 kernel: EIP: [<f894ff98>] run_queue+0x0/0x4 [gfs2]
SS:ESP 0068:f5ef9dd4
Feb 8 18:16:42 xmpp-alt2 kernel: <0>------------[ cut here ]------------
Feb 8 18:16:42 xmpp-alt2 kernel: kernel BUG at fs/gfs2/glock.c:632!
Feb 8 18:16:42 xmpp-alt2 kernel: invalid opcode: 0000 [#2]
Feb 8 18:16:42 xmpp-alt2 kernel: Modules linked in: lock_nolock lock_dlm
gfs2 dlm configfs
Feb 8 18:16:42 xmpp-alt2 kernel: CPU: 0
Feb 8 18:16:42 xmpp-alt2 kernel: EIP: 0060:[<f894ff98>] Not tainted
VLI
Feb 8 18:16:42 xmpp-alt2 kernel: EFLAGS: 00000247 (2.6.20-xmpp2 #3)
Feb 8 18:16:42 xmpp-alt2 kernel: EIP is at run_queue+0x0/0x4 [gfs2]
Feb 8 18:16:42 xmpp-alt2 kernel: eax: f5f119a4 ebx: f5f119a4 ecx:
00000001 edx: f5f119a4
Feb 8 18:16:42 xmpp-alt2 kernel: esi: f5e7c000 edi: f5f119a4 ebp:
00000001 esp: f5f33f94
Feb 8 18:16:42 xmpp-alt2 kernel: ds: 007b es: 007b ss: 0068
Feb 8 18:16:42 xmpp-alt2 kernel: Process gfs2_scand (pid: 1925, ti=f5f32000
task=f5effa70 task.ti=f5f32000)
Feb 8 18:16:42 xmpp-alt2 kernel: Stack: f895005f f8951881 f8951883 00001990
f5e7c000 f8948800 fffffffc f89518eb
Feb 8 18:16:42 xmpp-alt2 kernel: f5e7c000 f5e7c000 f894880c f5ef9dc4
c012320b 00000001 ffffffff ffffffff
Feb 8 18:16:42 xmpp-alt2 kernel: c012316e 00000000 00000000 00000000
c01034df f5ef9dbc 00000000 00000000
Feb 8 18:16:42 xmpp-alt2 kernel: Call Trace:
Feb 8 18:16:42 xmpp-alt2 kernel: [<f895005f>]
gfs2_glmutex_unlock+0x18/0x1c [gfs2]
Feb 8 18:16:42 xmpp-alt2 kernel: [<f8951881>] examine_bucket+0x59/0x5b
[gfs2]
Feb 8 18:16:42 xmpp-alt2 kernel: [<f8951883>] scan_glock+0x0/0x51 [gfs2]
Feb 8 18:16:42 xmpp-alt2 kernel: [<f8948800>] gfs2_scand+0x0/0x30 [gfs2]
Feb 8 18:16:42 xmpp-alt2 kernel: [<f89518eb>]
gfs2_scand_internal+0x17/0x22 [gfs2]
Feb 8 18:16:42 xmpp-alt2 kernel: [<f894880c>] gfs2_scand+0xc/0x30 [gfs2]
Feb 8 18:16:42 xmpp-alt2 kernel: [<c012320b>] kthread+0x9d/0xce
Feb 8 18:16:42 xmpp-alt2 kernel: [<c012316e>] kthread+0x0/0xce
Feb 8 18:16:42 xmpp-alt2 kernel: [<c01034df>]
kernel_thread_helper+0x7/0x10
Feb 8 18:16:42 xmpp-alt2 kernel: =======================
Feb 8 18:16:42 xmpp-alt2 kernel: Code: 08 8b 48 04 8b 00 89 48 04 89 01 89
5b 04 89 1b b8 07 00 00 00 0f b3 42 08 89 d8 e8 5b fd ff ff 89 d8 e8 07 79
7f c7 31 c0 5b c3 <0f> 0b eb fe 56 31 c9 31 d2 53 89 c3 83 ec 34 8d 74 24 04
89 34
Feb 8 18:16:42 xmpp-alt2 kernel: EIP: [<f894ff98>] run_queue+0x0/0x4 [gfs2]
SS:ESP 0068:f5f33f94
and?
On 2/8/07, Steven Whitehouse <swhiteho@redhat.com> wrote:
>
> Hi,
>
> I cannot reproduce what you are seeing, but try the attached patch to
> see if we can narrow this down,
>
> Steve.
>
> On Thu, 2007-02-08 at 15:02 +0100, Zbyszek ???kiewski wrote:
> > ok , i have build kernel from your git (
> > git://git.kernel.org/pub/scm/linux/kernel/git/steve/gfs2-2.6-nmw.git)
> > Linux version 2.6.20-xmpp2-ga2cf8222-dirty (root at xmpp-alt2) (gcc
> > version 3.4.4 20050314 (prerelease) (Debian 3.4.3-13sarge1)) #1 Thu
> > Feb 8 14:51:21 CET 2007
> >
> > and there is the same problem;
> > kernel BUG at fs/gfs2/glock.c:704!
> > invalid opcode: 0000 [#2]
> > Modules linked in: lock_nolock lock_dlm gfs2 dlm configfs
> > CPU: 0
> > EIP: 0060:[<f8950173>] Not tainted VLI
> > EFLAGS: 00000282 (2.6.20-xmpp2-ga2cf8222-dirty #1)
> > EIP is at gfs2_glmutex_unlock+0x18/0x1c [gfs2]
> >
> > and so one....
> >
> > ok so waiting patiently for solution....
> >
> >
> >
> >
> > On 2/8/07, Steven Whitehouse <swhiteho@redhat.com> wrote:
> > Hi,
> >
> > On Thu, 2007-02-08 at 14:04 +0100, Zbyszek ???kiewski wrote:
> > > sorry - mail went only to Steven, now to group....
> > >
> > >
> > > On 2/8/07, Zbyszek ???kiewski <zbyszek@toliman.pl> wrote:
> > > well, thanks for answer, i have tried with nolock,
> > and result
> > > is as follow:
> > > of course i made mkfs -t gfs2 -p lock_nolock -t
> > xmpp-alt2:test
> > > -j 1 /dev/sdb1 and then:
> > > mount -t gfs2 /dev/sdb1 /mnt -v
> > >
> > > and yes - the device is mounted,
> > It looks like what is happening is that a glmutex_unlock() is
> > discovering that its spinlock has been dropped by
> > glock.c:run_queue()
> > which should be impossible, so something odd is happening here
> > I think.
> >
> > The daemons implicated in this are there to demote unused
> > locks on a
> > periodic basis, so its presumably one of the locks used during
> > mounting
> > of the filesystem thats at fault.
> >
> > > (the changes to kernel you was talking about: you
> > mean: git1
> > > for 2.6.20?)
> > >
> > I'm not sure if its in git1 or not, I suspect it will be git2
> > since it
> > was only yesterday that the patches went in. Linus' current
> > git tree
> > seems to be broken (both gitweb and direct via the git tools)
> > otherwise
> > I'd post a URL to the changes. In the mean time you can find
> > them in my
> > -nmw tree which will get updated just as soon as git it
> > working again at
> > kernel.org,
> >
> > Steve.
> >
> >
> >
> >
> > --
> > pozdrawiam,
> > Zbyszek ???kiewski
>
>
--
pozdrawiam,
Zbyszek ???kiewski
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20070208/c0060ea1/attachment.htm>
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20.
2007-02-08 17:21 ` Zbyszek Żółkiewski
@ 2007-02-09 10:32 ` Steven Whitehouse
2007-02-09 10:34 ` Zbyszek Żółkiewski
0 siblings, 1 reply; 11+ messages in thread
From: Steven Whitehouse @ 2007-02-09 10:32 UTC (permalink / raw)
To: cluster-devel.redhat.com
Hi,
On Thu, 2007-02-08 at 18:21 +0100, Zbyszek ???kiewski wrote:
> are you using the same glibc and distro?
>
I don't think glibc will make a difference here. I'm using FC-6 but with
the upstream kernel.
> well i have patched gfs2 and here is output:
>
Thanks for the report, I presume that line 632 is, in your glock.c the
BUG_ON which is directly after the call to rq_promote? If so then we've
narrowed it down to that function. Having said that, even after looking
at every case of lock & unlock of gl_spin I still can't see any
unbalanced pairs of locks.
rq_promote does, in one case drop that spin lock, but it does also take
it again before exiting the function. By adding some more BUG_ON()s into
rq_promote, it might be possible to track it closer to the source.
Are you using preempt btw? That ought to be fine if you are, but its
always helpful to know when looking at locking problems,
Steve.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20.
2007-02-09 10:32 ` Steven Whitehouse
@ 2007-02-09 10:34 ` Zbyszek Żółkiewski
2007-02-09 11:41 ` Steven Whitehouse
0 siblings, 1 reply; 11+ messages in thread
From: Zbyszek Żółkiewski @ 2007-02-09 10:34 UTC (permalink / raw)
To: cluster-devel.redhat.com
well yes the line is:
static void run_queue(struct gfs2_glock *gl)
{
struct gfs2_holder *gh;
int blocked = 1;
BUG_ON(!spin_is_locked(&gl->gl_spin)); <
------------------------------ line 632
for (;;) {
.......
and i don;t use preemption (No Forced....)... i was playing with various
options but the bug still appears, no metter how i setup gfs, on what distro
(debian stable/eth)....
On 2/9/07, Steven Whitehouse <swhiteho@redhat.com> wrote:
>
> Hi,
>
> On Thu, 2007-02-08 at 18:21 +0100, Zbyszek ???kiewski wrote:
> > are you using the same glibc and distro?
> >
> I don't think glibc will make a difference here. I'm using FC-6 but with
> the upstream kernel.
>
> > well i have patched gfs2 and here is output:
> >
> Thanks for the report, I presume that line 632 is, in your glock.c the
> BUG_ON which is directly after the call to rq_promote? If so then we've
> narrowed it down to that function. Having said that, even after looking
> at every case of lock & unlock of gl_spin I still can't see any
> unbalanced pairs of locks.
>
> rq_promote does, in one case drop that spin lock, but it does also take
> it again before exiting the function. By adding some more BUG_ON()s into
> rq_promote, it might be possible to track it closer to the source.
>
> Are you using preempt btw? That ought to be fine if you are, but its
> always helpful to know when looking at locking problems,
>
> Steve.
>
>
>
--
pozdrawiam,
Zbyszek ???kiewski
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20070209/438bfa28/attachment.htm>
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20.
2007-02-09 10:34 ` Zbyszek Żółkiewski
@ 2007-02-09 11:41 ` Steven Whitehouse
2007-02-09 12:01 ` Zbyszek Żółkiewski
0 siblings, 1 reply; 11+ messages in thread
From: Steven Whitehouse @ 2007-02-09 11:41 UTC (permalink / raw)
To: cluster-devel.redhat.com
Hi,
Ok, that confuses me even more then, as thats not my expected location
of this problem. Looking back through the call chain it seems that there
is no obvious way in which this spinlock could become unlocked.
I'll have to think about this some more and get back to you I'm afraid,
Steve.
On Fri, 2007-02-09 at 11:34 +0100, Zbyszek ???kiewski wrote:
> well yes the line is:
>
> static void run_queue(struct gfs2_glock *gl)
> {
> struct gfs2_holder *gh;
> int blocked = 1;
>
> BUG_ON(!spin_is_locked(&gl->gl_spin)); <
> ------------------------------ line 632
> for (;;) {
> .......
>
> and i don;t use preemption (No Forced....)... i was playing with
> various options but the bug still appears, no metter how i setup gfs,
> on what distro (debian stable/eth)....
>
>
>
>
>
> On 2/9/07, Steven Whitehouse <swhiteho@redhat.com> wrote:
> Hi,
>
> On Thu, 2007-02-08 at 18:21 +0100, Zbyszek ???kiewski wrote:
> > are you using the same glibc and distro?
> >
> I don't think glibc will make a difference here. I'm using
> FC-6 but with
> the upstream kernel.
>
> > well i have patched gfs2 and here is output:
> >
> Thanks for the report, I presume that line 632 is, in your
> glock.c the
> BUG_ON which is directly after the call to rq_promote? If so
> then we've
> narrowed it down to that function. Having said that, even
> after looking
> at every case of lock & unlock of gl_spin I still can't see
> any
> unbalanced pairs of locks.
>
> rq_promote does, in one case drop that spin lock, but it does
> also take
> it again before exiting the function. By adding some more
> BUG_ON()s into
> rq_promote, it might be possible to track it closer to the
> source.
>
> Are you using preempt btw? That ought to be fine if you are,
> but its
> always helpful to know when looking at locking problems,
>
> Steve.
>
>
>
>
>
> --
> pozdrawiam,
> Zbyszek ???kiewski
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20.
2007-02-09 11:41 ` Steven Whitehouse
@ 2007-02-09 12:01 ` Zbyszek Żółkiewski
0 siblings, 0 replies; 11+ messages in thread
From: Zbyszek Żółkiewski @ 2007-02-09 12:01 UTC (permalink / raw)
To: cluster-devel.redhat.com
ok then, let me know if you find something...
On 2/9/07, Steven Whitehouse <swhiteho@redhat.com> wrote:
>
> Hi,
>
> Ok, that confuses me even more then, as thats not my expected location
> of this problem. Looking back through the call chain it seems that there
> is no obvious way in which this spinlock could become unlocked.
>
> I'll have to think about this some more and get back to you I'm afraid,
>
> Steve.
>
> On Fri, 2007-02-09 at 11:34 +0100, Zbyszek ???kiewski wrote:
> > well yes the line is:
> >
> > static void run_queue(struct gfs2_glock *gl)
> > {
> > struct gfs2_holder *gh;
> > int blocked = 1;
> >
> > BUG_ON(!spin_is_locked(&gl->gl_spin)); <
> > ------------------------------ line 632
> > for (;;) {
> > .......
> >
> > and i don;t use preemption (No Forced....)... i was playing with
> > various options but the bug still appears, no metter how i setup gfs,
> > on what distro (debian stable/eth)....
> >
> >
> >
> >
> >
> > On 2/9/07, Steven Whitehouse <swhiteho@redhat.com> wrote:
> > Hi,
> >
> > On Thu, 2007-02-08 at 18:21 +0100, Zbyszek ???kiewski wrote:
> > > are you using the same glibc and distro?
> > >
> > I don't think glibc will make a difference here. I'm using
> > FC-6 but with
> > the upstream kernel.
> >
> > > well i have patched gfs2 and here is output:
> > >
> > Thanks for the report, I presume that line 632 is, in your
> > glock.c the
> > BUG_ON which is directly after the call to rq_promote? If so
> > then we've
> > narrowed it down to that function. Having said that, even
> > after looking
> > at every case of lock & unlock of gl_spin I still can't see
> > any
> > unbalanced pairs of locks.
> >
> > rq_promote does, in one case drop that spin lock, but it does
> > also take
> > it again before exiting the function. By adding some more
> > BUG_ON()s into
> > rq_promote, it might be possible to track it closer to the
> > source.
> >
> > Are you using preempt btw? That ought to be fine if you are,
> > but its
> > always helpful to know when looking at locking problems,
> >
> > Steve.
> >
> >
> >
> >
> >
> > --
> > pozdrawiam,
> > Zbyszek ???kiewski
>
>
--
pozdrawiam,
Zbyszek ???kiewski
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20070209/79b7de07/attachment.htm>
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2007-02-09 12:01 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2007-02-08 12:15 [Cluster-devel] Kernel panic on mounting gfs2: kernel 2.6.19 and 2.6.20 Zbyszek Żółkiewski
2007-02-08 12:57 ` Steven Whitehouse
[not found] ` <f7ca4caa0702080502m723d4bc3r3dbdb9302a942bff@mail.gmail.com>
2007-02-08 13:04 ` Zbyszek Żółkiewski
2007-02-08 13:26 ` Steven Whitehouse
2007-02-08 14:02 ` Zbyszek Żółkiewski
2007-02-08 16:26 ` Steven Whitehouse
2007-02-08 17:21 ` Zbyszek Żółkiewski
2007-02-09 10:32 ` Steven Whitehouse
2007-02-09 10:34 ` Zbyszek Żółkiewski
2007-02-09 11:41 ` Steven Whitehouse
2007-02-09 12:01 ` Zbyszek Żółkiewski
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).