From: Dave Chinner <david@fromorbit.com>
To: David Rientjes <rientjes@google.com>
Cc: Wu Fengguang <fengguang.wu@intel.com>, Mel Gorman <mel@csn.ul.ie>,
linux-kernel@vger.kernel.org, linux-mm@kvack.org
Subject: Re: [2.6.35-rc1, bug] mm: minute-long livelocks in memory reclaim
Date: Mon, 23 Aug 2010 22:33:39 +1000 [thread overview]
Message-ID: <20100823123339.GI31488@dastard> (raw)
In-Reply-To: <alpine.DEB.2.00.1008230219480.13384@chino.kir.corp.google.com>
On Mon, Aug 23, 2010 at 02:23:27AM -0700, David Rientjes wrote:
> On Mon, 23 Aug 2010, Wu Fengguang wrote:
>
> > > I've been testing parallel create workloads over the weekend, and
> > > I've seen this a couple of times now under 8 thread parallel creates
> > > with XFS. I'm running on an 8p VM with 4GB RAM and a fast disk
> > > subsystem. Basically I am seeing the create rate drop to zero
> > > with all 8 CPUs stuck spinning for up to 2 minutes. 'echo t >
> > > /proc/sysrq-trigger' while this is occurring gives the following
> > > trace for all the fs-mark processes:
.....
>
> You may be interested in Mel's patchset that he just proposed for -mm
> which identifies watermark variations on machines with high cpu counts
> (perhaps even eight, as in this report). The last patch actually reworks
> this hunk of the code as well.
>
> http://marc.info/?l=linux-mm&m=128255044912938
> http://marc.info/?l=linux-mm&m=128255045312950
> http://marc.info/?l=linux-mm&m=128255045012942
> http://marc.info/?l=linux-mm&m=128255045612954
>
> Dave, it would be interesting to see if this fixes your problem.
That looks promising - I'll give it a shot, though my test case is
not really what you'd call reproducable(*) so it might take a
couple of days before I can say whether the issue has gone away or
not.
Cheers,
Dave.
(*) create 100 million inodes in parallel using fsmark, collect and
watch behavioural metrics via PCP/pmchart for stuff out of the
ordinary, and dump stack traces, etc when somthing strange occurs.
--
Dave Chinner
david@fromorbit.com
WARNING: multiple messages have this Message-ID (diff)
From: Dave Chinner <david@fromorbit.com>
To: David Rientjes <rientjes@google.com>
Cc: Wu Fengguang <fengguang.wu@intel.com>, Mel Gorman <mel@csn.ul.ie>,
linux-kernel@vger.kernel.org, linux-mm@kvack.org
Subject: Re: [2.6.35-rc1, bug] mm: minute-long livelocks in memory reclaim
Date: Mon, 23 Aug 2010 22:33:39 +1000 [thread overview]
Message-ID: <20100823123339.GI31488@dastard> (raw)
In-Reply-To: <alpine.DEB.2.00.1008230219480.13384@chino.kir.corp.google.com>
On Mon, Aug 23, 2010 at 02:23:27AM -0700, David Rientjes wrote:
> On Mon, 23 Aug 2010, Wu Fengguang wrote:
>
> > > I've been testing parallel create workloads over the weekend, and
> > > I've seen this a couple of times now under 8 thread parallel creates
> > > with XFS. I'm running on an 8p VM with 4GB RAM and a fast disk
> > > subsystem. Basically I am seeing the create rate drop to zero
> > > with all 8 CPUs stuck spinning for up to 2 minutes. 'echo t >
> > > /proc/sysrq-trigger' while this is occurring gives the following
> > > trace for all the fs-mark processes:
.....
>
> You may be interested in Mel's patchset that he just proposed for -mm
> which identifies watermark variations on machines with high cpu counts
> (perhaps even eight, as in this report). The last patch actually reworks
> this hunk of the code as well.
>
> http://marc.info/?l=linux-mm&m=128255044912938
> http://marc.info/?l=linux-mm&m=128255045312950
> http://marc.info/?l=linux-mm&m=128255045012942
> http://marc.info/?l=linux-mm&m=128255045612954
>
> Dave, it would be interesting to see if this fixes your problem.
That looks promising - I'll give it a shot, though my test case is
not really what you'd call reproducable(*) so it might take a
couple of days before I can say whether the issue has gone away or
not.
Cheers,
Dave.
(*) create 100 million inodes in parallel using fsmark, collect and
watch behavioural metrics via PCP/pmchart for stuff out of the
ordinary, and dump stack traces, etc when somthing strange occurs.
--
Dave Chinner
david@fromorbit.com
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2010-08-23 12:33 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-08-22 23:48 [2.6.35-rc1, bug] mm: minute-long livelocks in memory reclaim Dave Chinner
2010-08-22 23:48 ` Dave Chinner
2010-08-23 6:58 ` Wu Fengguang
2010-08-23 6:58 ` Wu Fengguang
2010-08-23 9:23 ` David Rientjes
2010-08-23 9:23 ` David Rientjes
2010-08-23 12:33 ` Dave Chinner [this message]
2010-08-23 12:33 ` Dave Chinner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20100823123339.GI31488@dastard \
--to=david@fromorbit.com \
--cc=fengguang.wu@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mel@csn.ul.ie \
--cc=rientjes@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.