From: Dave Chinner <david@fromorbit.com>
To: "Ryan C. England" <ryan.england@corvidtec.com>
Cc: Christoph Hellwig <hch@infradead.org>,
linux-mm@kvack.org, Andi Kleen <andi@firstfloor.org>,
xfs@oss.sgi.com
Subject: Re: XFS causing stack overflow
Date: Tue, 13 Dec 2011 09:47:37 +1100 [thread overview]
Message-ID: <20111212224737.GS14273@dastard> (raw)
In-Reply-To: <CAAnfqPC0Ed=PDUOowGTEZyfqHFjB3Jj2YNAaxuYqA2+wVb6tSA@mail.gmail.com>
On Mon, Dec 12, 2011 at 08:43:57AM -0500, Ryan C. England wrote:
> On Mon, Dec 12, 2011 at 4:00 AM, Dave Chinner <david@fromorbit.com> wrote:
> > On Mon, Dec 12, 2011 at 06:13:11AM +0100, Andi Kleen wrote:
> > > BTW I suppose it wouldn't be all that hard to add more stacks and
> > > switch to them too, similar to what the 32bit do_IRQ does.
> > > Perhaps XFS could just allocate its own stack per thread
> > > (or maybe only if it detects some specific configuration that
> > > is known to need much stack)
> >
> > That's possible, but rather complex, I think.
> > > It would need to be per thread if you could sleep inside them.
> >
> > Yes, we'd need to sleep, do IO, possibly operate within a
> > transaction context, etc, and a workqueue handles all these cases
> > without having to do anything special. Splitting the stack at a
> > logical point is probably better, such as this patch:
> >
> > http://oss.sgi.com/archives/xfs/2011-07/msg00443.html
>
> Is it possible to apply this patch to my current installation? We use this
> box in production and the reboots that we're experiencing are an
> inconvenience.
Not easily. The problem with a backport is that the workqueue
infrastructure changed around 2.6.36, allowing workqueues to act
like an (almost) infinite pool of worker threads and so by using a
workqueue we can have effectively unlimited numbers of concurrent
allocations in progress at once.
The workqueue implementation in 2.6.32 only allows a single work
instance per workqueue thread, and so even with per-CPU worker
threads, would only allow one allocation at a time per CPU. This
adds additional serialisation within a filesystem, between
filesystem and potentially adds new deadlock conditions as well.
So it's not exactly obvious whether it can be backported in a sane
manner or not.
> Is there is a walkthrough on how to apply this patch? If not, could your
> provide the steps necessary to apply successfully? I would greatly
> appreciate it.
It would probably need redesigning and re-implementing from scratch
because of the above reasons. It'd then need a lot of testing and
review. As a workaround, you might be better off doing what Andi
first suggested - recompiling your kernel to use 16k stacks.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
WARNING: multiple messages have this Message-ID (diff)
From: Dave Chinner <david@fromorbit.com>
To: "Ryan C. England" <ryan.england@corvidtec.com>
Cc: Andi Kleen <andi@firstfloor.org>,
Christoph Hellwig <hch@infradead.org>,
linux-mm@kvack.org, xfs@oss.sgi.com
Subject: Re: XFS causing stack overflow
Date: Tue, 13 Dec 2011 09:47:37 +1100 [thread overview]
Message-ID: <20111212224737.GS14273@dastard> (raw)
In-Reply-To: <CAAnfqPC0Ed=PDUOowGTEZyfqHFjB3Jj2YNAaxuYqA2+wVb6tSA@mail.gmail.com>
On Mon, Dec 12, 2011 at 08:43:57AM -0500, Ryan C. England wrote:
> On Mon, Dec 12, 2011 at 4:00 AM, Dave Chinner <david@fromorbit.com> wrote:
> > On Mon, Dec 12, 2011 at 06:13:11AM +0100, Andi Kleen wrote:
> > > BTW I suppose it wouldn't be all that hard to add more stacks and
> > > switch to them too, similar to what the 32bit do_IRQ does.
> > > Perhaps XFS could just allocate its own stack per thread
> > > (or maybe only if it detects some specific configuration that
> > > is known to need much stack)
> >
> > That's possible, but rather complex, I think.
> > > It would need to be per thread if you could sleep inside them.
> >
> > Yes, we'd need to sleep, do IO, possibly operate within a
> > transaction context, etc, and a workqueue handles all these cases
> > without having to do anything special. Splitting the stack at a
> > logical point is probably better, such as this patch:
> >
> > http://oss.sgi.com/archives/xfs/2011-07/msg00443.html
>
> Is it possible to apply this patch to my current installation? We use this
> box in production and the reboots that we're experiencing are an
> inconvenience.
Not easily. The problem with a backport is that the workqueue
infrastructure changed around 2.6.36, allowing workqueues to act
like an (almost) infinite pool of worker threads and so by using a
workqueue we can have effectively unlimited numbers of concurrent
allocations in progress at once.
The workqueue implementation in 2.6.32 only allows a single work
instance per workqueue thread, and so even with per-CPU worker
threads, would only allow one allocation at a time per CPU. This
adds additional serialisation within a filesystem, between
filesystem and potentially adds new deadlock conditions as well.
So it's not exactly obvious whether it can be backported in a sane
manner or not.
> Is there is a walkthrough on how to apply this patch? If not, could your
> provide the steps necessary to apply successfully? I would greatly
> appreciate it.
It would probably need redesigning and re-implementing from scratch
because of the above reasons. It'd then need a lot of testing and
review. As a workaround, you might be better off doing what Andi
first suggested - recompiling your kernel to use 16k stacks.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2011-12-12 22:47 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-12-08 18:03 XFS causing stack overflow Ryan C. England
2011-12-09 11:55 ` Christoph Hellwig
2011-12-09 11:55 ` Christoph Hellwig
2011-12-09 15:56 ` Ryan C. England
2011-12-09 15:56 ` Ryan C. England
2011-12-09 22:19 ` Dave Chinner
2011-12-09 22:19 ` Dave Chinner
[not found] ` <20111209221956.GE14273__25752.826271537$1323469420$gmane$org@dastard>
2011-12-10 19:52 ` Andi Kleen
2011-12-10 19:52 ` Andi Kleen
2011-12-10 22:13 ` Dave Chinner
2011-12-10 22:13 ` Dave Chinner
2011-12-11 0:00 ` Andi Kleen
2011-12-11 0:00 ` Andi Kleen
2011-12-11 23:05 ` Dave Chinner
2011-12-11 23:05 ` Dave Chinner
2011-12-12 2:31 ` Andi Kleen
2011-12-12 2:31 ` Andi Kleen
2011-12-12 4:36 ` Dave Chinner
2011-12-12 4:36 ` Dave Chinner
2011-12-12 5:13 ` Andi Kleen
2011-12-12 5:13 ` Andi Kleen
2011-12-12 9:00 ` Dave Chinner
2011-12-12 9:00 ` Dave Chinner
2011-12-12 13:43 ` Ryan C. England
2011-12-12 13:43 ` Ryan C. England
2011-12-12 22:47 ` Dave Chinner [this message]
2011-12-12 22:47 ` Dave Chinner
[not found] ` <20111209115513.GA19994__23079.9863501035$1323435203$gmane$org@infradead.org>
2011-12-09 19:53 ` Andi Kleen
2011-12-09 19:53 ` Andi Kleen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20111212224737.GS14273@dastard \
--to=david@fromorbit.com \
--cc=andi@firstfloor.org \
--cc=hch@infradead.org \
--cc=linux-mm@kvack.org \
--cc=ryan.england@corvidtec.com \
--cc=xfs@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.