Re: [Lsf] IO less throttling and cgroup aware writeback (Was: Re: Preliminary Agenda and Activities for LSF)

Vivek Goyal
2011-04-07T17:56:18Z

On Thu, Apr 07, 2011 at 09:50:39AM +1000, Dave Chinner wrote:
> On Wed, Apr 06, 2011 at 11:37:15AM -0400, Vivek Goyal wrote:
> > On Wed, Apr 06, 2011 at 08:56:40AM +1000, Dave Chinner wrote:
> > > On Tue, Apr 05, 2011 at 09:13:59AM -0400, Vivek Goyal wrote:
> > > It also means you will have handle the case of a cgroup over a
> > > throttle limit and no inodes on it's dirty list. It's not a case of
> > > "probably can live with" the resultant mess, the mess will occur and
> > > so handling it needs to be designed in from the start.
> >
> > This behavior can happen due to shared page accounting. One possible
> > way to mitigate this problme is to traverse through LRU list of pages
> > of memcg and find an inode to do the writebak.
>
> Page LRU ordered writeback is something we need to avoid. It causes
> havok with IO and allocation patterns. Also, how expensive is such a
> walk? If it's a common operation, then it's a non-starter for the
> generic writeback code.
>

Agreed that LRU ordered writeback needs to be avoided as it is going to be
expensive. That's why the notion of inode on per memcg list and do per inode
writeback. This seems to be only backup plan in case when there is no
inode to do IO due to shared inode accounting issues.

Do you have ideas on better way to handle it? The other proposal of
maintaining a list memcg_mapping, which tracks which inode this cgroup
has dirtied has been deemed complex and been kind of rejected at least
for the first step.

> BTW, how is "shared page accounting" different to the shared dirty
> inode case we've been discussing?

IIUC, there are two problems.

- Issues because of shared page accounting
- Issues because of shared inode accouting.

So in shared page accounting, if two process do IO to same page, IO gets
charged to cgroup who first touched the page. So if a cgroup is writting
on lots of shared pages, it will be charged to the other cgroup who
brought the page in memory to begin with and will drive its dirty ratio
up. So this seems to be case of weaker isolation in case of shared pages,
and we got to live with it.

Similarly if inode is shared, inode gets put on the list of memcg who dirtied
it first. So now if two cgroups are dirtying pages on inode, then pages should
be charged to respective cgroup but inode will be only on one memcg and once
writeback is performed it might happen that cgroup is over its background
limit but there are no inodes to do writeback.

>
> > After yesterday's discussion it looked like people agreed that to
> > begin with keep it simple and maintain the notion of one inode on
> > one memcg list. So instead of inode being on global bdi dirty list
> > it will be on per memecg per bdi dirty list.
>
> Good to hear.
>
> > > how metadata IO is going to be handled by
> > > IO controllers,
> >
> > So IO controller provides two mechanisms.
> >
> > - IO throttling(bytes_per_second, io_per_second interface)
> > - Proportional weight disk sharing
> >
> > In case of proportional weight disk sharing, we don't run into issues of
> > priority inversion and metadata handing should not be a concern.
>
> Though metadata IO will affect how much bandwidth/iops is available
> for applications to use.

I think meta data IO will be accounted to the process submitting the meta
data IO. (IO tracking stuff will be used only for page cache pages during
page dirtying time). So yes, the process doing meta data IO will be
charged for it.

I think I am missing something here and not understanding your concern
exactly here.

>
> > For throttling case, apart from metadata, I found that with simple
> > throttling of data I ran into issues with journalling with ext4 mounuted
> > in ordered mode. So it was suggested that WRITE IO throttling should
> > not be done at device level instead try to do it in higher layers,
> > possibly balance_dirty_pages() and throttle process early.
>
> The problem with doing it at the page cache entry level is that
> cache hits then get throttled. It's not really a an IO controller at
> that point, and the impact on application performance could be huge
> (i.e. MB/s instead of GB/s).

Agreed that throttling cache hits is not a good idea. Can we determine
if page being asked for is in cache or not and charge for IO accordingly.

>
> > So yes, I agree that little more documentation and more clarity on this
> > would be good. All this cgroup aware writeback primarily is being done
> > for CFQ's proportional disk sharing at the moment.
> >
> > > what kswapd is going to do writeback when the pages
> > > it's trying to writeback during a critical low memory event belong
> > > to a cgroup that is throttled at the IO level, etc.
> >
> > Throttling will move up so kswapd will not be throttled. Even today,
> > kswapd is part of root group and we do not suggest throttling root group.
>
> So once again you have the problem of writeback from kswapd (which
> is ugly to begin with) affecting all the groups. Given kswapd likes
> to issue what is effectively random IO, this coul dhave devastating
> effect on everything else....

Implementing throttling at higher layer has the problem of IO spikes
at the end level devices when flusher or kswapd decide to do bunch of
IO. I really don't have a good answer for that. Doing throttling at
device level runs into issues with journalling. So I guess issues of
IO spikes is lesser concern as compared to issue of choking filesystem.

Following two things might help though a bit with IO spikes.

- Keep per cgroup background dirty ratio low so that flusher tries to
flush out pages sooner than later.

- All the IO coming from flusher/kswapd will be going in root group
from throttling perspective. We can try to throttle it again to
some reasonable value to reduce the impact of IO spikes.

Ideas to handle this better?

>
> > For the case of proportional disk sharing, we will probably account
> > IO to respective cgroups (pages submitted by kswapd) and that should
> > not flush to disk fairly fast and should not block for long time as it is
> > work consering mechanism.
>
> Well, it depends. I can still see how, with proportional IO, kswapd
> would get slowed cleaning dirty pages on one memcg when there are
> clean pages in another memcg that it could reclaim without doing any
> IO. i.e. it has potential to slow down memory reclaim significantly.
> (Note, I'm assuming proportional IO doesn't mean "no throttling" it
> just means there is a much lower delay on IO issue.)

Proportional IO can delay submitting an IO only if there is IO happening
in other groups. So IO can still be throttled and limits are decided
by fair share of a group. But if other groups are not doing IO and not
using their fair share, then the group doing IO gets bigger share.

So yes, if heavy IO is happening at disk while kswapd is also trying
to reclaim memory, then IO submitted by kswapd can be delayed and
this can slow down reclaim. (Does kswapd has to block after submitting
IO from a memcg. Can't it just move onto next memcg and either free
pages if not dirty, or also submit IO from next memcg?)

Thanks
Vivek