Re: IO controller discussion (Was: Re: [PATCH 01/10] Documentation)

[Andrea Righi <righi.andrea@gmail.com>]

On Fri, Apr 17, 2009 at 11:09:51AM -0700, Nauman Rafique wrote:
> > Thinking more about it. Memory controller can probably enforce the higher
> > limit but it would not easily translate into a fixed upper async write
> > rate. Till the process hits the page cache limit or is slowed down by
> > dirty page writeout, it can get a very high async write BW.
> >
> > So memory controller page cache limit will help but it would not direclty
> > translate into what max bw limit patches are doing.
> >
> > Even if we do max bw control at IO scheduler level, async writes are
> > problematic again. IO controller will not be able to throttle the process
> > until it sees actuall write request. In big memory systems, writeout might
> > not happen for some time and till then it will see a high throughput.
> >
> > So doing async write throttling at higher layer and not at IO scheduler
> > layer gives us the opprotunity to produce more accurate results.
> 
> Wouldn't 'doing control on writes at a higher layer' have the same
> problems as the ones we talk about in dm-ioband? What if the cgroup
> being throttled for dirtying pages has a high weight assigned to it at
> the IO scheduler level? What if there are threads of different classes
> within that cgroup, and we would want to let RT task dirty the pages
> before BE tasks? I am not sure all these questions make sense, but
> just wanted to raise issues that might pop up.

To a large degree, this seems to be related to provide "fair throttling"
at higher level. I mean, throttle equally the tasks belongin to a cgroup
that exceeded the limits. With equally I mean proportionally to the IO
traffic previously generated _and_ the IO priority.

Otherwise a low priority task doing a lot of IO can consumes all the
available cgroup BW and other high priority tasks in the same cgroup may
be blocked when they try to write to disk, even if they try to write a
small amount of bytes.

> 
> If the whole system is designed with cgroups in mind, then throttling
> at IO scheduler layer should lead to backlog, that could be seen at
> higher level. For example, if a cgroup is not getting service at IO
> scheduler level, it should run out of request descriptors, and thus
> the thread writing back dirty pages should notice it (if its pdflush,
> blocking it is probably not the best idea). And that should mean the
> cgroup should hit the dirty threshold, and disallow the task to dirty
> further pages. There is a possibility though that getting all this
> right might be an overkill and we can get away with a simpler
> solution. One possibility seems to be that we provide some feedback
> from IO scheduling layer to higher layers, that cgroup is hitting its
> write bandwith limit, and should not be allowed to dirty any more
> pages.
> 

IMHO accounting the IO activity in the IO scheduler and blocking the
offending application at the higher level is a good solution.

Throttle dirty page ratio could be a nice feature, but probably it's
enough to provide a max amount of dirty pages per cgroup and force the
tasks to directly writeback those pages when the cgroup exceeded the
dirty limit. In this way the dirty page ratio will be automatically
throttled by the underlying IO controller.

-Andrea