* mdadm --stop goes off and never comes back?
@ 2007-12-18 21:11 Jon Nelson
2007-12-20 1:45 ` Neil Brown
0 siblings, 1 reply; 6+ messages in thread
From: Jon Nelson @ 2007-12-18 21:11 UTC (permalink / raw)
To: linux-raid
This just happened to me.
Create raid with:
mdadm --create /dev/md2 --level=raid10 --raid-devices=3
--spare-devices=0 --layout=o2 /dev/sdb3 /dev/sdc3 /dev/sdd3
cat /proc/mdstat
md2 : active raid10 sdd3[2] sdc3[1] sdb3[0]
5855424 blocks 64K chunks 2 offset-copies [3/3] [UUU]
[==>..................] resync = 14.6% (859968/5855424)
finish=1.3min speed=61426K/sec
Some log messages:
Dec 18 15:02:28 turnip kernel: md: md2: raid array is not clean --
starting background reconstruction
Dec 18 15:02:28 turnip kernel: raid10: raid set md2 active with 3 out
of 3 devices
Dec 18 15:02:28 turnip kernel: md: resync of RAID array md2
Dec 18 15:02:28 turnip kernel: md: minimum _guaranteed_ speed: 1000
KB/sec/disk.
Dec 18 15:02:28 turnip kernel: md: using maximum available idle IO
bandwidth (but not more than 200000 KB/sec) for resync.
Dec 18 15:02:28 turnip kernel: md: using 128k window, over a total of
5855424 blocks.
Dec 18 15:03:36 turnip kernel: md: md2: resync done.
Dec 18 15:03:36 turnip kernel: md: checkpointing resync of md2.
I tried to stop the array:
mdadm --stop /dev/md2
and mdadm never came back. It's off in the kernel somewhere. :-(
kill, of course, has no effect.
The machine still runs fine, the rest of the raids (md0 and md1) work
fine (same disks).
The output (snipped, only mdadm) of 'echo t > /proc/sysrq-trigger'
Dec 18 15:09:13 turnip kernel: mdadm S 0001e5359fa38fb0 0
3943 1 (NOTLB)
Dec 18 15:09:13 turnip kernel: ffff810033e7ddc8 0000000000000086
0000000000000000 0000000000000092
Dec 18 15:09:13 turnip kernel: 0000000000000fc7 ffff810033e7dd78
ffffffff80617800 ffffffff80617800
Dec 18 15:09:13 turnip kernel: ffffffff8061d210 ffffffff80617800
ffffffff80617800 0000000000000000
Dec 18 15:09:13 turnip kernel: Call Trace:
Dec 18 15:09:13 turnip kernel: [<ffffffff803fac96>]
__mutex_lock_interruptible_slowpath+0x8b/0xca
Dec 18 15:09:13 turnip kernel: [<ffffffff802acccb>] do_open+0x222/0x2a5
Dec 18 15:09:13 turnip kernel: [<ffffffff8038705d>] md_seq_show+0x127/0x6c1
Dec 18 15:09:13 turnip kernel: [<ffffffff80275597>] vma_merge+0x141/0x1ee
Dec 18 15:09:13 turnip kernel: [<ffffffff802a2aa0>] seq_read+0x1bf/0x28b
Dec 18 15:09:13 turnip kernel: [<ffffffff8028a42d>] vfs_read+0xcb/0x153
Dec 18 15:09:13 turnip kernel: [<ffffffff8028a7c1>] sys_read+0x45/0x6e
Dec 18 15:09:13 turnip kernel: [<ffffffff80209c2e>] system_call+0x7e/0x83
What happened? Is there any debug info I can provide before I reboot?
--
Jon
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mdadm --stop goes off and never comes back?
2007-12-18 21:11 mdadm --stop goes off and never comes back? Jon Nelson
@ 2007-12-20 1:45 ` Neil Brown
2007-12-20 2:05 ` Jon Nelson
0 siblings, 1 reply; 6+ messages in thread
From: Neil Brown @ 2007-12-20 1:45 UTC (permalink / raw)
To: Jon Nelson; +Cc: linux-raid
On Tuesday December 18, jnelson-linux-raid@jamponi.net wrote:
> This just happened to me.
> Create raid with:
>
> mdadm --create /dev/md2 --level=raid10 --raid-devices=3
> --spare-devices=0 --layout=o2 /dev/sdb3 /dev/sdc3 /dev/sdd3
>
> cat /proc/mdstat
>
> md2 : active raid10 sdd3[2] sdc3[1] sdb3[0]
> 5855424 blocks 64K chunks 2 offset-copies [3/3] [UUU]
> [==>..................] resync = 14.6% (859968/5855424)
> finish=1.3min speed=61426K/sec
>
> Some log messages:
>
> Dec 18 15:02:28 turnip kernel: md: md2: raid array is not clean --
> starting background reconstruction
> Dec 18 15:02:28 turnip kernel: raid10: raid set md2 active with 3 out
> of 3 devices
> Dec 18 15:02:28 turnip kernel: md: resync of RAID array md2
> Dec 18 15:02:28 turnip kernel: md: minimum _guaranteed_ speed: 1000
> KB/sec/disk.
> Dec 18 15:02:28 turnip kernel: md: using maximum available idle IO
> bandwidth (but not more than 200000 KB/sec) for resync.
> Dec 18 15:02:28 turnip kernel: md: using 128k window, over a total of
> 5855424 blocks.
> Dec 18 15:03:36 turnip kernel: md: md2: resync done.
> Dec 18 15:03:36 turnip kernel: md: checkpointing resync of md2.
>
> I tried to stop the array:
>
> mdadm --stop /dev/md2
>
> and mdadm never came back. It's off in the kernel somewhere. :-(
>
> kill, of course, has no effect.
> The machine still runs fine, the rest of the raids (md0 and md1) work
> fine (same disks).
>
> The output (snipped, only mdadm) of 'echo t > /proc/sysrq-trigger'
>
> Dec 18 15:09:13 turnip kernel: mdadm S 0001e5359fa38fb0 0
> 3943 1 (NOTLB)
> Dec 18 15:09:13 turnip kernel: ffff810033e7ddc8 0000000000000086
> 0000000000000000 0000000000000092
> Dec 18 15:09:13 turnip kernel: 0000000000000fc7 ffff810033e7dd78
> ffffffff80617800 ffffffff80617800
> Dec 18 15:09:13 turnip kernel: ffffffff8061d210 ffffffff80617800
> ffffffff80617800 0000000000000000
> Dec 18 15:09:13 turnip kernel: Call Trace:
> Dec 18 15:09:13 turnip kernel: [<ffffffff803fac96>]
> __mutex_lock_interruptible_slowpath+0x8b/0xca
> Dec 18 15:09:13 turnip kernel: [<ffffffff802acccb>] do_open+0x222/0x2a5
> Dec 18 15:09:13 turnip kernel: [<ffffffff8038705d>] md_seq_show+0x127/0x6c1
> Dec 18 15:09:13 turnip kernel: [<ffffffff80275597>] vma_merge+0x141/0x1ee
> Dec 18 15:09:13 turnip kernel: [<ffffffff802a2aa0>] seq_read+0x1bf/0x28b
> Dec 18 15:09:13 turnip kernel: [<ffffffff8028a42d>] vfs_read+0xcb/0x153
> Dec 18 15:09:13 turnip kernel: [<ffffffff8028a7c1>] sys_read+0x45/0x6e
> Dec 18 15:09:13 turnip kernel: [<ffffffff80209c2e>] system_call+0x7e/0x83
>
>
>
> What happened? Is there any debug info I can provide before I reboot?
Don't know.... very odd.
The rest of the 'sysrq' output would possibly help.
NeilBrown
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mdadm --stop goes off and never comes back?
2007-12-20 1:45 ` Neil Brown
@ 2007-12-20 2:05 ` Jon Nelson
2007-12-20 2:07 ` Jon Nelson
0 siblings, 1 reply; 6+ messages in thread
From: Jon Nelson @ 2007-12-20 2:05 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
On 12/19/07, Neil Brown <neilb@suse.de> wrote:
> On Tuesday December 18, jnelson-linux-raid@jamponi.net wrote:
> > This just happened to me.
> > Create raid with:
> >
> > mdadm --create /dev/md2 --level=raid10 --raid-devices=3
> > --spare-devices=0 --layout=o2 /dev/sdb3 /dev/sdc3 /dev/sdd3
> >
> > cat /proc/mdstat
> >
> > md2 : active raid10 sdd3[2] sdc3[1] sdb3[0]
> > 5855424 blocks 64K chunks 2 offset-copies [3/3] [UUU]
> > [==>..................] resync = 14.6% (859968/5855424)
> > finish=1.3min speed=61426K/sec
> >
> > Some log messages:
> >
> > Dec 18 15:02:28 turnip kernel: md: md2: raid array is not clean --
> > starting background reconstruction
> > Dec 18 15:02:28 turnip kernel: raid10: raid set md2 active with 3 out
> > of 3 devices
> > Dec 18 15:02:28 turnip kernel: md: resync of RAID array md2
> > Dec 18 15:02:28 turnip kernel: md: minimum _guaranteed_ speed: 1000
> > KB/sec/disk.
> > Dec 18 15:02:28 turnip kernel: md: using maximum available idle IO
> > bandwidth (but not more than 200000 KB/sec) for resync.
> > Dec 18 15:02:28 turnip kernel: md: using 128k window, over a total of
> > 5855424 blocks.
> > Dec 18 15:03:36 turnip kernel: md: md2: resync done.
> > Dec 18 15:03:36 turnip kernel: md: checkpointing resync of md2.
> >
> > I tried to stop the array:
> >
> > mdadm --stop /dev/md2
> >
> > and mdadm never came back. It's off in the kernel somewhere. :-(
> >
> > kill, of course, has no effect.
> > The machine still runs fine, the rest of the raids (md0 and md1) work
> > fine (same disks).
> >
> > The output (snipped, only mdadm) of 'echo t > /proc/sysrq-trigger'
> >
> > Dec 18 15:09:13 turnip kernel: mdadm S 0001e5359fa38fb0 0
> > 3943 1 (NOTLB)
> > Dec 18 15:09:13 turnip kernel: ffff810033e7ddc8 0000000000000086
> > 0000000000000000 0000000000000092
> > Dec 18 15:09:13 turnip kernel: 0000000000000fc7 ffff810033e7dd78
> > ffffffff80617800 ffffffff80617800
> > Dec 18 15:09:13 turnip kernel: ffffffff8061d210 ffffffff80617800
> > ffffffff80617800 0000000000000000
> > Dec 18 15:09:13 turnip kernel: Call Trace:
> > Dec 18 15:09:13 turnip kernel: [<ffffffff803fac96>]
> > __mutex_lock_interruptible_slowpath+0x8b/0xca
> > Dec 18 15:09:13 turnip kernel: [<ffffffff802acccb>] do_open+0x222/0x2a5
> > Dec 18 15:09:13 turnip kernel: [<ffffffff8038705d>] md_seq_show+0x127/0x6c1
> > Dec 18 15:09:13 turnip kernel: [<ffffffff80275597>] vma_merge+0x141/0x1ee
> > Dec 18 15:09:13 turnip kernel: [<ffffffff802a2aa0>] seq_read+0x1bf/0x28b
> > Dec 18 15:09:13 turnip kernel: [<ffffffff8028a42d>] vfs_read+0xcb/0x153
> > Dec 18 15:09:13 turnip kernel: [<ffffffff8028a7c1>] sys_read+0x45/0x6e
> > Dec 18 15:09:13 turnip kernel: [<ffffffff80209c2e>] system_call+0x7e/0x83
> >
> >
> >
> > What happened? Is there any debug info I can provide before I reboot?
>
> Don't know.... very odd.
>
> The rest of the 'sysrq' output would possibly help.
Does this help? It's the same syscall and args, I think, as above.
Dec 18 15:09:13 turnip kernel: hald S 0001e52f4793e397 0
3040 1 (NOTLB)
Dec 18 15:09:13 turnip kernel: ffff81003aa51e38 0000000000000086
0000000000000000 ffffffff802
68ee6
Dec 18 15:09:13 turnip kernel: ffff81002a97e5c0 ffff81003aa51de8
ffffffff80617800 ffffffff806
17800
Dec 18 15:09:13 turnip kernel: ffffffff8061d210 ffffffff80617800
ffffffff80617800 ffff8100000
0bb48
Dec 18 15:09:13 turnip kernel: Call Trace:
Dec 18 15:09:13 turnip kernel: [<ffffffff80268ee6>]
get_page_from_freelist+0x3c4/0x545
Dec 18 15:09:13 turnip kernel: [<ffffffff803fac96>]
__mutex_lock_interruptible_slowpath+0x8b/
0xca
Dec 18 15:09:13 turnip kernel: [<ffffffff80387adf>] md_attr_show+0x2f/0x64
Dec 18 15:09:13 turnip kernel: [<ffffffff802cd142>] sysfs_read_file+0xb3/0x111
Dec 18 15:09:13 turnip kernel: [<ffffffff8028a42d>] vfs_read+0xcb/0x153
Dec 18 15:09:13 turnip kernel: [<ffffffff8028a7c1>] sys_read+0x45/0x6e
Dec 18 15:09:13 turnip kernel: [<ffffffff80209c2e>] system_call+0x7e/0x83
Dec 18 15:09:13 turnip kernel:
--
Jon
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mdadm --stop goes off and never comes back?
2007-12-20 2:05 ` Jon Nelson
@ 2007-12-20 2:07 ` Jon Nelson
2007-12-22 11:58 ` Neil Brown
0 siblings, 1 reply; 6+ messages in thread
From: Jon Nelson @ 2007-12-20 2:07 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
On 12/19/07, Jon Nelson <jnelson-linux-raid@jamponi.net> wrote:
> On 12/19/07, Neil Brown <neilb@suse.de> wrote:
> > On Tuesday December 18, jnelson-linux-raid@jamponi.net wrote:
> > > This just happened to me.
> > > Create raid with:
> > >
> > > mdadm --create /dev/md2 --level=raid10 --raid-devices=3
> > > --spare-devices=0 --layout=o2 /dev/sdb3 /dev/sdc3 /dev/sdd3
> > >
> > > cat /proc/mdstat
> > >
> > > md2 : active raid10 sdd3[2] sdc3[1] sdb3[0]
> > > 5855424 blocks 64K chunks 2 offset-copies [3/3] [UUU]
> > > [==>..................] resync = 14.6% (859968/5855424)
> > > finish=1.3min speed=61426K/sec
> > >
> > > Some log messages:
> > >
> > > Dec 18 15:02:28 turnip kernel: md: md2: raid array is not clean --
> > > starting background reconstruction
> > > Dec 18 15:02:28 turnip kernel: raid10: raid set md2 active with 3 out
> > > of 3 devices
> > > Dec 18 15:02:28 turnip kernel: md: resync of RAID array md2
> > > Dec 18 15:02:28 turnip kernel: md: minimum _guaranteed_ speed: 1000
> > > KB/sec/disk.
> > > Dec 18 15:02:28 turnip kernel: md: using maximum available idle IO
> > > bandwidth (but not more than 200000 KB/sec) for resync.
> > > Dec 18 15:02:28 turnip kernel: md: using 128k window, over a total of
> > > 5855424 blocks.
> > > Dec 18 15:03:36 turnip kernel: md: md2: resync done.
> > > Dec 18 15:03:36 turnip kernel: md: checkpointing resync of md2.
> > >
> > > I tried to stop the array:
> > >
> > > mdadm --stop /dev/md2
> > >
> > > and mdadm never came back. It's off in the kernel somewhere. :-(
> > >
> > > kill, of course, has no effect.
> > > The machine still runs fine, the rest of the raids (md0 and md1) work
> > > fine (same disks).
> > >
> > > The output (snipped, only mdadm) of 'echo t > /proc/sysrq-trigger'
> > >
> > > Dec 18 15:09:13 turnip kernel: mdadm S 0001e5359fa38fb0 0
> > > 3943 1 (NOTLB)
> > > Dec 18 15:09:13 turnip kernel: ffff810033e7ddc8 0000000000000086
> > > 0000000000000000 0000000000000092
> > > Dec 18 15:09:13 turnip kernel: 0000000000000fc7 ffff810033e7dd78
> > > ffffffff80617800 ffffffff80617800
> > > Dec 18 15:09:13 turnip kernel: ffffffff8061d210 ffffffff80617800
> > > ffffffff80617800 0000000000000000
> > > Dec 18 15:09:13 turnip kernel: Call Trace:
> > > Dec 18 15:09:13 turnip kernel: [<ffffffff803fac96>]
> > > __mutex_lock_interruptible_slowpath+0x8b/0xca
> > > Dec 18 15:09:13 turnip kernel: [<ffffffff802acccb>] do_open+0x222/0x2a5
> > > Dec 18 15:09:13 turnip kernel: [<ffffffff8038705d>] md_seq_show+0x127/0x6c1
> > > Dec 18 15:09:13 turnip kernel: [<ffffffff80275597>] vma_merge+0x141/0x1ee
> > > Dec 18 15:09:13 turnip kernel: [<ffffffff802a2aa0>] seq_read+0x1bf/0x28b
> > > Dec 18 15:09:13 turnip kernel: [<ffffffff8028a42d>] vfs_read+0xcb/0x153
> > > Dec 18 15:09:13 turnip kernel: [<ffffffff8028a7c1>] sys_read+0x45/0x6e
> > > Dec 18 15:09:13 turnip kernel: [<ffffffff80209c2e>] system_call+0x7e/0x83
> > >
> > >
> > >
> > > What happened? Is there any debug info I can provide before I reboot?
> >
> > Don't know.... very odd.
> >
> > The rest of the 'sysrq' output would possibly help.
>
> Does this help? It's the same syscall and args, I think, as above.
>
> Dec 18 15:09:13 turnip kernel: hald S 0001e52f4793e397 0
> 3040 1 (NOTLB)
> Dec 18 15:09:13 turnip kernel: ffff81003aa51e38 0000000000000086
> 0000000000000000 ffffffff802
> 68ee6
> Dec 18 15:09:13 turnip kernel: ffff81002a97e5c0 ffff81003aa51de8
> ffffffff80617800 ffffffff806
> 17800
> Dec 18 15:09:13 turnip kernel: ffffffff8061d210 ffffffff80617800
> ffffffff80617800 ffff8100000
> 0bb48
> Dec 18 15:09:13 turnip kernel: Call Trace:
> Dec 18 15:09:13 turnip kernel: [<ffffffff80268ee6>]
> get_page_from_freelist+0x3c4/0x545
> Dec 18 15:09:13 turnip kernel: [<ffffffff803fac96>]
> __mutex_lock_interruptible_slowpath+0x8b/
> 0xca
> Dec 18 15:09:13 turnip kernel: [<ffffffff80387adf>] md_attr_show+0x2f/0x64
> Dec 18 15:09:13 turnip kernel: [<ffffffff802cd142>] sysfs_read_file+0xb3/0x111
> Dec 18 15:09:13 turnip kernel: [<ffffffff8028a42d>] vfs_read+0xcb/0x153
> Dec 18 15:09:13 turnip kernel: [<ffffffff8028a7c1>] sys_read+0x45/0x6e
> Dec 18 15:09:13 turnip kernel: [<ffffffff80209c2e>] system_call+0x7e/0x83
> Dec 18 15:09:13 turnip kernel:
NOTE: kernel is stock openSUSE 10.3 kernel, x86_64, 2.6.22.13-0.3-default.
--
Jon
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mdadm --stop goes off and never comes back?
2007-12-20 2:07 ` Jon Nelson
@ 2007-12-22 11:58 ` Neil Brown
2007-12-22 13:01 ` Jon Nelson
0 siblings, 1 reply; 6+ messages in thread
From: Neil Brown @ 2007-12-22 11:58 UTC (permalink / raw)
To: Jon Nelson; +Cc: linux-raid
On Wednesday December 19, jnelson-linux-raid@jamponi.net wrote:
> On 12/19/07, Jon Nelson <jnelson-linux-raid@jamponi.net> wrote:
> > On 12/19/07, Neil Brown <neilb@suse.de> wrote:
> > > On Tuesday December 18, jnelson-linux-raid@jamponi.net wrote:
> > > >
> > > > I tried to stop the array:
> > > >
> > > > mdadm --stop /dev/md2
> > > >
> > > > and mdadm never came back. It's off in the kernel somewhere. :-(
Looking at your stack traces, you have the "mdadm -S" holding
an md lock and trying to get a sysfs lock as part of tearing down the
array, and 'hald' is trying to read some attribute in
/sys/block/md....
and is holding the sysfs lock and trying to get the md lock.
A classic AB-BA deadlock.
>
> NOTE: kernel is stock openSUSE 10.3 kernel, x86_64, 2.6.22.13-0.3-default.
>
It is fixed in mainline with some substantial changes to sysfs.
I don't imagine they are likely to get back ported to openSUSE, but
you could try logging a bugzilla if you like.
The 'hald' process is interruptible and killing it would release the
deadlock.
I suspect you have to be fairly unlucky to lose the race but it is
obviously quite possible.
I don't think there is anything I can do on the md side to avoid the
bug.
NeilBrown
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mdadm --stop goes off and never comes back?
2007-12-22 11:58 ` Neil Brown
@ 2007-12-22 13:01 ` Jon Nelson
0 siblings, 0 replies; 6+ messages in thread
From: Jon Nelson @ 2007-12-22 13:01 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
On 12/22/07, Neil Brown <neilb@suse.de> wrote:
> On Wednesday December 19, jnelson-linux-raid@jamponi.net wrote:
> > On 12/19/07, Jon Nelson <jnelson-linux-raid@jamponi.net> wrote:
> > > On 12/19/07, Neil Brown <neilb@suse.de> wrote:
> > > > On Tuesday December 18, jnelson-linux-raid@jamponi.net wrote:
> > > > >
> > > > > I tried to stop the array:
> > > > >
> > > > > mdadm --stop /dev/md2
> > > > >
> > > > > and mdadm never came back. It's off in the kernel somewhere. :-(
>
> Looking at your stack traces, you have the "mdadm -S" holding
> an md lock and trying to get a sysfs lock as part of tearing down the
> array, and 'hald' is trying to read some attribute in
> /sys/block/md....
> and is holding the sysfs lock and trying to get the md lock.
> A classic AB-BA deadlock.
>
> >
> > NOTE: kernel is stock openSUSE 10.3 kernel, x86_64, 2.6.22.13-0.3-default.
> >
>
> It is fixed in mainline with some substantial changes to sysfs.
> I don't imagine they are likely to get back ported to openSUSE, but
> you could try logging a bugzilla if you like.
Nah - I'm eagerly awaiting new kernels anyway as I have some network
cards that work much better (read: they work) with 2.6.24rc3+.
> The 'hald' process is interruptible and killing it would release the
> deadlock.
Cool.
> I suspect you have to be fairly unlucky to lose the race but it is
> obviously quite possible.
Sometimes we are all a little unlucky. In my case, it cost me a reboot
or, in others, nothing at all. Fortunately this was not a production
system with lots of users.
> I don't think there is anything I can do on the md side to avoid the
> bug.
In the situation I don't think that such a change would be warranted anyway.
Thanks again for looking at this. I'm a big believer in the 'canary in
a coal mine' mentality - some problems may indications of much more
serious issues, but in this case, it would appear that the issue has
already been taken care of. Have a Happy Holidays.
--
Jon
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2007-12-22 13:01 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2007-12-18 21:11 mdadm --stop goes off and never comes back? Jon Nelson
2007-12-20 1:45 ` Neil Brown
2007-12-20 2:05 ` Jon Nelson
2007-12-20 2:07 ` Jon Nelson
2007-12-22 11:58 ` Neil Brown
2007-12-22 13:01 ` Jon Nelson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox