XFS thread inflation in 2.6.23rc

XFS thread inflation in 2.6.23rc

[Andi Kleen]

In 2.6.23rc I have a new kernel thread running from XFS:

30137 ?        S<     0:00 [xfs_mru_cache]


Is that one really needed? Can it be started only on demand when that MRU
feature is used? 

Thanks,

-Andi

Re: XFS thread inflation in 2.6.23rc

[David Chinner]

On Wed, Aug 08, 2007 at 12:40:21PM +0200, Andi Kleen wrote:
> 
> In 2.6.23rc I have a new kernel thread running from XFS:
> 
> 30137 ?        S<     0:00 [xfs_mru_cache]
> 
> Is that one really needed? Can it be started only on demand when that MRU
> feature is used? 

It uses a single threaded workqueue for reaping objects and the thread comes
along with that. Creating the workqueue on demand would require creating a
kernel thread inside a transaction and that's not some thing we want to do.
It can't really be put into an existing thread/workqueue because of
deadlock problems which is why it has it's own workqueue....

Besides, what's the point of having nice constructs like dedicated
workqueues if people complain when they get used to solve problems?

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS thread inflation in 2.6.23rc

[Andi Kleen]

On Wednesday 08 August 2007 14:13:59 David Chinner wrote:
> On Wed, Aug 08, 2007 at 12:40:21PM +0200, Andi Kleen wrote:
> > 
> > In 2.6.23rc I have a new kernel thread running from XFS:
> > 
> > 30137 ?        S<     0:00 [xfs_mru_cache]
> > 
> > Is that one really needed? Can it be started only on demand when that MRU
> > feature is used? 
> 
> It uses a single threaded workqueue for reaping objects and the thread comes
> along with that. Creating the workqueue on demand would require creating a
> kernel thread inside a transaction and that's not some thing we want to do.

Why not? I can't think of any possible problem except memory allocation
recursion, but even that should be handled.

> Besides, what's the point of having nice constructs like dedicated
> workqueues

It's a resource that shouldn't be overused.

Especially for such a obscure feature -- i remember reviewing your rationale
for the MRU cache and the probability of this applying to 99.9+% of users ever 
is pretty small. If you insist adding such things make them as least 
as unobtrusive as possible.

> if people complain when they get used to solve problems? 

Does XFS really need that many threads? Seems doubtful to me.

Ok part of the problem is that the workqueues are a little dumb.
e.g. it's highly doubtful per SMT thread workqueues really make any sense.
It would be probably enough to have one per socket or one per node.
But that's a separate issue from just gratuitously adding new ones.

-Andi

Re: XFS thread inflation in 2.6.23rc

[David Chinner]

On Wed, Aug 08, 2007 at 02:22:10PM +0200, Andi Kleen wrote:
> On Wednesday 08 August 2007 14:13:59 David Chinner wrote:
> > On Wed, Aug 08, 2007 at 12:40:21PM +0200, Andi Kleen wrote:
> > > 
> > > In 2.6.23rc I have a new kernel thread running from XFS:
> > > 
> > > 30137 ?        S<     0:00 [xfs_mru_cache]
> > > 
> > > Is that one really needed? Can it be started only on demand when that MRU
> > > feature is used? 
> > 
> > It uses a single threaded workqueue for reaping objects and the thread comes
> > along with that. Creating the workqueue on demand would require creating a
> > kernel thread inside a transaction and that's not some thing we want to do.
> 
> Why not? I can't think of any possible problem except memory allocation
> recursion, but even that should be handled.

Memory allocation failure + dirty transaction == filesystem shutdown.

Bad, bad, bad, bad.

> > Besides, what's the point of having nice constructs like dedicated
> > workqueues
> 
> It's a resource that shouldn't be overused.

A workqueue + thread uses, what, 10-15k of memory? That's the cost of about
10 cached inodes. It is insignificant...

> Especially for such a obscure feature -- i remember reviewing your
> rationale for the MRU cache and the probability of this applying
> to 99.9+% of users ever is pretty small. If you insist adding such
> things make them as least as unobtrusive as possible.

Sure, it was written for a small market, but it's useful for other
workloads. e.g. got a heavy tmp file load? Mark /tmp as a filestream
directory and all your temp files end up in one location on disk
and don't pollute and fragment everything else. Got a PVR that
records multiple streams at once? I wouldn't reject this out
of hand as not useful. It was created for the specific purpose
of workload isolation and that can be used in many different
ways....

> > if people complain when they get used to solve problems? 
> 
> Does XFS really need that many threads? Seems doubtful to me.

XFS is much more threaded than any other linux filesystem.
It does lots of stuff asynchronously and that's where the
threads and workqueues come in. They fit in very closely with
the nature of XFS which is to do as much stuff concurrently as
possible on as many CPUs as possible.

Hmmm. I guess you are really not going to like the patch I
have that moves the AIL pushing to a new thread to solve
some of scalability issues in the transaction subsystem......

> Ok part of the problem is that the workqueues are a little dumb.
> e.g. it's highly doubtful per SMT thread workqueues really make
> any sense.  It would be probably enough to have one per socket or
> one per node.

Agreed, but that is a workqueue implementation detail, not a reason
for preventing ppl from using workqueues.

> But that's a separate issue from just gratuitously adding new ones.

Call it what you will. Threads and workqueues are there to be used
and they cost very little for the flexibility and concurrency they
provide us.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS thread inflation in 2.6.23rc

[Andi Kleen]

On Wednesday 08 August 2007 15:14:04 David Chinner wrote:

> Memory allocation failure + dirty transaction == filesystem shutdown.

You mean if the workqueue creation would fail? 

Surely not having the MRU cache is not a catastrophe and would
allow the transaction to commit anyways? 

The other alternative would be to start it when a directory with 
the flag is first seen. That should be before any transactions.

The other thing is that it would be nasty to switch do_fork
to use GFP_NOFS, but I doubt that's really needed.
 
> > > Besides, what's the point of having nice constructs like dedicated
> > > workqueues
> > 
> > It's a resource that shouldn't be overused.
> 
> A workqueue + thread uses, what, 10-15k of memory? That's the cost of about
> 10 cached inodes. It is insignificant...

A little bloat here and a little bloat there and soon we're talking
about serious memory. 

e.g. on a dual core box in a standard configuration we're going towards
~50 kernel threads out of the box now and that's just too much IMNSHO.
Given there are other really bad wastes of memory (don't get me
started on some of the hash tables), but it's not totally insignificant.

Threads also have some CPU costs even when sleeping, e.g. there is still code
which walks all threads to do various things.

> Hmmm. I guess you are really not going to like the patch I
> have that moves the AIL pushing to a new thread to solve
> some of scalability issues in the transaction subsystem......

Per CPU or single? If the later definitely the workqueues would need to be 
fixed first.

-Andi
 

Re: XFS thread inflation in 2.6.23rc

[David Chinner]

On Wed, Aug 08, 2007 at 03:26:06PM +0200, Andi Kleen wrote:
> On Wednesday 08 August 2007 15:14:04 David Chinner wrote:
> 
> > Memory allocation failure + dirty transaction == filesystem shutdown.
> 
> You mean if the workqueue creation would fail? 

Yeah.

> Surely not having the MRU cache is not a catastrophe and would
> allow the transaction to commit anyways? 

Right now the only errors that come from filestream association
are fatal errors. i.e. they should force a shutdown if the transaction
is dirty. One of these errors is already an ENOMEM.

Now we'd have a case where we'd had a "error that is not an error"
and handle that in some way. It gets complex and that's something
I try to avoid if at all possible.

> The other alternative would be to start it when a directory with 
> the flag is first seen. That should be before any transactions.

On lookup? No thanks.

On first create? Possible, but it's still painful and it introduces
overhead into every single create operation.

> > > > Besides, what's the point of having nice constructs like dedicated
> > > > workqueues
> > > It's a resource that shouldn't be overused.
> > A workqueue + thread uses, what, 10-15k of memory? That's the cost of about
> > 10 cached inodes. It is insignificant...
> 
> A little bloat here and a little bloat there and soon we're talking
> about serious memory. 
>
> e.g. on a dual core box in a standard configuration we're going towards
> ~50 kernel threads out of the box now and that's just too much IMNSHO.

My idle desktop machine has 180 processes running on it, 60 of them
kernel threads. It doesn't concern me one bit.

Andi, you are complaining to the wrong person about thread counts. I
live in the world of excessive parallelism and multithreaded I/O. I
regularly see 4-8p boxes running hundreds to thousands of I/O
threads on a single filesystem. These are the workloads XFS is
designed for and we optimise for. A single extra thread is noise....

> > Hmmm. I guess you are really not going to like the patch I
> > have that moves the AIL pushing to a new thread to solve
> > some of scalability issues in the transaction subsystem......
> 
> Per CPU or single?

Single.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS thread inflation in 2.6.23rc

[Eric Sandeen]

David Chinner wrote:

> Andi, you are complaining to the wrong person about thread counts. I
> live in the world of excessive parallelism and multithreaded I/O. I
> regularly see 4-8p boxes running hundreds to thousands of I/O
> threads on a single filesystem. These are the workloads XFS is
> designed for and we optimise for. A single extra thread is noise....


The other thing this is doing, though, is having an impact on power
usage, I think.  Fedora is trying to identify things which wake the CPU
in an effort to reduce power usage if possible.

See for example:
http://article.gmane.org/gmane.linux.redhat.fedora.testers/50480

quoting:

"One of the biggest power problems is apps that wake up the CPU
unnecessarily. With the tickless kernel for x86 (and soon x86_64),
every time this happens it's a chance for power savings lost.

HOW YOU CAN HELP
----------------

We'd like to get as many reports of misbehaving apps as possible.

1) Install the 'powertop' package

2) Run it in a terminal window on a reasonably idle system with your
   normal combination of apps. (If you run it when you're actively
   compiling a kernel, watching a movie, or doing other CPU-intensive
   things, the results aren't as useful.)

3) Note the results..."

and here are the results w/ xfs mounted:

< Detailed C-state information is only available on Mobile CPUs (laptops) >
P-states (frequencies)
  2.21 Ghz     0.0%
  2.00 Ghz     0.0%
  1.80 Ghz   100.0%
Wakeups-from-idle per second : 31.7
no ACPI power usage estimate available
Top causes for wakeups:
  37.0% ( 20.0)             mount : xfs_mru_cache_create

...

looks like having this running (note: i'm not using filestreams on this
mounted filesystem) may be impacting the power usage of the CPU?  Or
maybe SGI doesn' care about that, if the goal is 100% utilization of
these machines on altixes etc... but still - something to consider I think.

Thanks,
-Eric

Re: XFS thread inflation in 2.6.23rc

[David Chinner]

On Fri, Aug 10, 2007 at 02:34:10PM -0500, Eric Sandeen wrote:
> David Chinner wrote:
> 
> > Andi, you are complaining to the wrong person about thread counts. I
> > live in the world of excessive parallelism and multithreaded I/O. I
> > regularly see 4-8p boxes running hundreds to thousands of I/O
> > threads on a single filesystem. These are the workloads XFS is
> > designed for and we optimise for. A single extra thread is noise....
> 
> 
> The other thing this is doing, though, is having an impact on power
> usage, I think.  Fedora is trying to identify things which wake the CPU
> in an effort to reduce power usage if possible.

Sure, but an idle thread doesn't use any power, and the workqueue
is now only used when there are streams active.

> and here are the results w/ xfs mounted:
> 
> < Detailed C-state information is only available on Mobile CPUs (laptops) >
> P-states (frequencies)
>   2.21 Ghz     0.0%
>   2.00 Ghz     0.0%
>   1.80 Ghz   100.0%
> Wakeups-from-idle per second : 31.7
> no ACPI power usage estimate available
> Top causes for wakeups:
>   37.0% ( 20.0)             mount : xfs_mru_cache_create

http://oss.sgi.com/archives/xfs/2007-08/msg00009.html

"On-demand reaping of the MRU cache

Instead of running the mru cache reaper all the time based on a
timeout, we should only run it when the cache has active objects.
This allows CPUs to sleep when there is no activity rather than
be woken repeatedly just to check if there is anything to do."

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS thread inflation in 2.6.23rc

[Eric Sandeen]

David Chinner wrote:

> http://oss.sgi.com/archives/xfs/2007-08/msg00009.html
> 
> "On-demand reaping of the MRU cache
> 
> Instead of running the mru cache reaper all the time based on a
> timeout, we should only run it when the cache has active objects.
> This allows CPUs to sleep when there is no activity rather than
> be woken repeatedly just to check if there is anything to do."

Ah, cool.  Missed that, sorry.  Will re-run powertop with that just to
keep you on your toes.  ;-)

-Eric