* hung in raise_barrier() in raid1.c -- any ideas?
@ 2012-09-20 16:52 Chris Friesen
2012-09-20 17:55 ` Chris Friesen
0 siblings, 1 reply; 5+ messages in thread
From: Chris Friesen @ 2012-09-20 16:52 UTC (permalink / raw)
To: Neil Brown, linux-raid
Hi,
I've got a fairly beefy (32 cpus, 64GB ram, isci-based SAS disks, etc.)
embedded system running 2.6.27.
We're seeing issues where disk operations suddenly seem to stall. In
the most recent case we had the hung-task watchdog indicate that
md1_resync was stuck for more than 120sec in raise_barrier().
There are a bunch of "normal" tasks also stuck in wait_barrier(), so
based on that I assume we're stuck in the second call to
wait_event_lock_irq().
Has anyone seen anything like this? Could commit 73d5c38 be related?
What about 1d9d524?
Also, what's the meaning of RESYNC_DEPTH?
Thanks,
Chris
--
Chris Friesen
Software Designer
3500 Carling Avenue
Ottawa, Ontario K2H 8E9
www.genband.com
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: hung in raise_barrier() in raid1.c -- any ideas?
2012-09-20 16:52 hung in raise_barrier() in raid1.c -- any ideas? Chris Friesen
@ 2012-09-20 17:55 ` Chris Friesen
2012-09-20 21:27 ` NeilBrown
0 siblings, 1 reply; 5+ messages in thread
From: Chris Friesen @ 2012-09-20 17:55 UTC (permalink / raw)
To: Neil Brown, linux-raid
On 09/20/2012 10:52 AM, Chris Friesen wrote:
>
> Hi,
>
> I've got a fairly beefy (32 cpus, 64GB ram, isci-based SAS disks,
> etc.) embedded system running 2.6.27.
>
> We're seeing issues where disk operations suddenly seem to stall. In
> the most recent case we had the hung-task watchdog indicate that
> md1_resync was stuck for more than 120sec in raise_barrier().
>
> There are a bunch of "normal" tasks also stuck in wait_barrier(), so
> based on that I assume we're stuck in the second call to
> wait_event_lock_irq().
>
> Has anyone seen anything like this? Could commit 73d5c38 be related?
> What about 1d9d524?
Could d6b42dc be related?
Chris
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: hung in raise_barrier() in raid1.c -- any ideas?
2012-09-20 17:55 ` Chris Friesen
@ 2012-09-20 21:27 ` NeilBrown
2012-09-20 23:05 ` Chris Friesen
0 siblings, 1 reply; 5+ messages in thread
From: NeilBrown @ 2012-09-20 21:27 UTC (permalink / raw)
To: Chris Friesen; +Cc: linux-raid
[-- Attachment #1: Type: text/plain, Size: 1200 bytes --]
On Thu, 20 Sep 2012 11:55:02 -0600 Chris Friesen <chris.friesen@genband.com>
wrote:
> On 09/20/2012 10:52 AM, Chris Friesen wrote:
> >
> > Hi,
> >
> > I've got a fairly beefy (32 cpus, 64GB ram, isci-based SAS disks,
> > etc.) embedded system running 2.6.27.
> >
> > We're seeing issues where disk operations suddenly seem to stall. In
> > the most recent case we had the hung-task watchdog indicate that
> > md1_resync was stuck for more than 120sec in raise_barrier().
> >
> > There are a bunch of "normal" tasks also stuck in wait_barrier(), so
> > based on that I assume we're stuck in the second call to
> > wait_event_lock_irq().
> >
> > Has anyone seen anything like this? Could commit 73d5c38 be related?
> > What about 1d9d524?
>
> Could d6b42dc be related?
That last one seems more likely. Does the scenario fit your config.
i.e. is your RAID1 being used under LVM?
If it does, then I would say it is very likely this issue.
> Also, what's the meaning of RESYNC_DEPTH?
The maximum number of resync requests that can be concurrently active.
RESYNC_WINDOW should really be RESYNC_BLOCK_SIZE * RESYNC_DEPTH
I wonder why it isn't.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: hung in raise_barrier() in raid1.c -- any ideas?
2012-09-20 21:27 ` NeilBrown
@ 2012-09-20 23:05 ` Chris Friesen
2012-09-21 2:16 ` NeilBrown
0 siblings, 1 reply; 5+ messages in thread
From: Chris Friesen @ 2012-09-20 23:05 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
On 09/20/2012 03:27 PM, NeilBrown wrote:
> On Thu, 20 Sep 2012 11:55:02 -0600 Chris Friesen<chris.friesen@genband.com>
> wrote:
>
>> On 09/20/2012 10:52 AM, Chris Friesen wrote:
>>>
>>> Hi,
>>>
>>> I've got a fairly beefy (32 cpus, 64GB ram, isci-based SAS disks,
>>> etc.) embedded system running 2.6.27.
>>>
>>> We're seeing issues where disk operations suddenly seem to stall. In
>>> the most recent case we had the hung-task watchdog indicate that
>>> md1_resync was stuck for more than 120sec in raise_barrier().
>>>
>>> There are a bunch of "normal" tasks also stuck in wait_barrier(), so
>>> based on that I assume we're stuck in the second call to
>>> wait_event_lock_irq().
>>>
>>> Has anyone seen anything like this? Could commit 73d5c38 be related?
>>> What about 1d9d524?
>>
>> Could d6b42dc be related?
>
> That last one seems more likely. Does the scenario fit your config.
> i.e. is your RAID1 being used under LVM?
>
> If it does, then I would say it is very likely this issue.
Yes, we're using it under LVM. I've added some instrumentation to tell
if we're hitting that case. The current->bio_list handling is a bit
different in 2.6.27 but I think I figured out the equivalent to the patch.
Interesting that it took this long to fix that issue.
>> Also, what's the meaning of RESYNC_DEPTH?
>
> The maximum number of resync requests that can be concurrently active.
And each request would resync a single block?
Chris
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: hung in raise_barrier() in raid1.c -- any ideas?
2012-09-20 23:05 ` Chris Friesen
@ 2012-09-21 2:16 ` NeilBrown
0 siblings, 0 replies; 5+ messages in thread
From: NeilBrown @ 2012-09-21 2:16 UTC (permalink / raw)
To: Chris Friesen; +Cc: linux-raid
[-- Attachment #1: Type: text/plain, Size: 1744 bytes --]
On Thu, 20 Sep 2012 17:05:23 -0600 Chris Friesen <chris.friesen@genband.com>
wrote:
> On 09/20/2012 03:27 PM, NeilBrown wrote:
> > On Thu, 20 Sep 2012 11:55:02 -0600 Chris Friesen<chris.friesen@genband.com>
> > wrote:
> >
> >> On 09/20/2012 10:52 AM, Chris Friesen wrote:
> >>>
> >>> Hi,
> >>>
> >>> I've got a fairly beefy (32 cpus, 64GB ram, isci-based SAS disks,
> >>> etc.) embedded system running 2.6.27.
> >>>
> >>> We're seeing issues where disk operations suddenly seem to stall. In
> >>> the most recent case we had the hung-task watchdog indicate that
> >>> md1_resync was stuck for more than 120sec in raise_barrier().
> >>>
> >>> There are a bunch of "normal" tasks also stuck in wait_barrier(), so
> >>> based on that I assume we're stuck in the second call to
> >>> wait_event_lock_irq().
> >>>
> >>> Has anyone seen anything like this? Could commit 73d5c38 be related?
> >>> What about 1d9d524?
> >>
> >> Could d6b42dc be related?
> >
> > That last one seems more likely. Does the scenario fit your config.
> > i.e. is your RAID1 being used under LVM?
> >
> > If it does, then I would say it is very likely this issue.
>
>
> Yes, we're using it under LVM. I've added some instrumentation to tell
> if we're hitting that case. The current->bio_list handling is a bit
> different in 2.6.27 but I think I figured out the equivalent to the patch.
>
> Interesting that it took this long to fix that issue.
>
>
> >> Also, what's the meaning of RESYNC_DEPTH?
> >
> > The maximum number of resync requests that can be concurrently active.
>
> And each request would resync a single block?
Each request will resync up to RESYNC_BLOCK_SIZE bytes - i.e. up to 64K.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2012-09-21 2:16 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-09-20 16:52 hung in raise_barrier() in raid1.c -- any ideas? Chris Friesen
2012-09-20 17:55 ` Chris Friesen
2012-09-20 21:27 ` NeilBrown
2012-09-20 23:05 ` Chris Friesen
2012-09-21 2:16 ` NeilBrown
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).