Re: [RFC] [PATCH] A clean approach to writeout throttling

[Andrew Morton <akpm@linux-foundation.org>]

On Thu, 6 Dec 2007 01:48:52 -0800 Daniel Phillips <phillips@phunq.net> wrote:

> On Wednesday 05 December 2007 23:31, Andrew Morton wrote:
> > > > Rather than asking the stack "how much memory will this request consume"
> > > > you could instead ask "how much memory are you currently using".
> > > > 
> > > > ie: on entry to the stack, do 
> > > > 
> > > > 	current->account_block_allocations = 1;
> > > > 	make_request(...);
> > > > 	rq->used_memory += current->pages_used_for_block_allocations;
> > > > 
> > > > and in the page allocator do
> > > > 
> > > > 	if (!in_interrupt() && current->account_block_allocations)
> > > > 		current->pages_used_for_block_allocations++;
> > > > 
> > > > and then somehow handle deallocation too ;)
> > > 
> > > Ah, and how do you ensure that you do not deadlock while making this
> > > inquiry?
> > 
> > It isn't an inquiry - it's a plain old submit_bio() and it runs to
> > completion in the usual fashion.
> > 
> > Thing is, we wouldn't have called it at all if this queue was already over 
> > its allocation limit.  IOW, we know that it's below its allocation limit,
> > so we know it won't deadlock.  Given, of course, reasonably pessimistc
> > error margins.
> 
> OK, I see what you are suggesting.  Yes, one could set the inflight limit
> very low and the reserve very high, and run a bio through the stack (what
> I meant by "inquiry") to discover the actual usage, then shrink the reserve
> accordingly.  By also running a real bio through the stack we can discover
> something about the latency.  So we would then know roughly how high
> the inflight limit should be set and how much the memalloc reserve
> should be increased to handle that particular driver instance.
> 
> The big fly in this ointment is that we cannot possibly know that our bio
> followed the worst case resource consumption path, whereas it is fairly
> easy (hopefully) for a programmer to determine this statically.

nonono...

Consider an example.

- We a-priori decide to limit a particular stack's peak memory usage to
  1MB

- We empirically discover that the maximum amount of memory which is
  allocated by that stack on behalf of a single BIO is 16kb.  (ie: that's
  the most it has ever used for a single BIO).

- Now, we refuse to feed any more BIOs into the stack when its
  instantaneous memory usage exceeds (1MB - 16kb).

Of course, the _average_ memory-per-BIO is much less than 16kb.  So there
are a lot of BIOs in flight - probably hundreds, but a minimum of 63.

There is a teeny so-small-it-doesn't-matter chance that the stack will
exceed the 1MB limit.  If it happens to be at its (1MB-16kb) limit and all
the memory in the machine is AWOL and then someone throws a
never-seen-before twirly BIO at it.  Not worth worrying about, surely.