Support for global block IO throttling?

Support for global block IO throttling?

[Juerg Haefliger]

Hi,

Are there any plans to support global (per cgroup) IO throttling in
the near future? I'm in need of this and would gladly work on it if
nobody else is and if people think this is a useful feature. What I
envision is that one can specify the limits of individual devices in a
group but also specify the overall aggregate limit of the whole group.
I just started looking at the throttling code and haven't quite
figured it out yet so it might take me a bit to get up to speed.

Thanks
...Juerg

Re: Support for global block IO throttling?

[Vivek Goyal]

On Wed, Mar 23, 2011 at 02:48:18PM +0100, Juerg Haefliger wrote:
> Hi,
> 
> Are there any plans to support global (per cgroup) IO throttling in
> the near future? I'm in need of this and would gladly work on it if
> nobody else is and if people think this is a useful feature. What I
> envision is that one can specify the limits of individual devices in a
> group but also specify the overall aggregate limit of the whole group.
> I just started looking at the throttling code and haven't quite
> figured it out yet so it might take me a bit to get up to speed.

Hi Juerg,

What's the use case? Why do you need global block IO throttling?

Thanks
Vivek

Re: Support for global block IO throttling?

[Juerg Haefliger]

Hi Vivek,

>> Are there any plans to support global (per cgroup) IO throttling in
>> the near future? I'm in need of this and would gladly work on it if
>> nobody else is and if people think this is a useful feature. What I
>> envision is that one can specify the limits of individual devices in a
>> group but also specify the overall aggregate limit of the whole group.
>> I just started looking at the throttling code and haven't quite
>> figured it out yet so it might take me a bit to get up to speed.
>
> Hi Juerg,
>
> What's the use case? Why do you need global block IO throttling?

Resource management for guest VMs. I want to make sure a VM cannot
exceed a certain IO limit. If the VM does IOs to only one device, it
gets the full specified bandwidth to that single device but if the VM
talks to multiple devices, it only gets a fraction of the global
bandwidth per device, not exceeded the global limit.

...Juerg


> Thanks
> Vivek
>

Re: Support for global block IO throttling?

[Vivek Goyal]

On Wed, Mar 23, 2011 at 03:39:28PM +0100, Juerg Haefliger wrote:
> Hi Vivek,
> 
> >> Are there any plans to support global (per cgroup) IO throttling in
> >> the near future? I'm in need of this and would gladly work on it if
> >> nobody else is and if people think this is a useful feature. What I
> >> envision is that one can specify the limits of individual devices in a
> >> group but also specify the overall aggregate limit of the whole group.
> >> I just started looking at the throttling code and haven't quite
> >> figured it out yet so it might take me a bit to get up to speed.
> >
> > Hi Juerg,
> >
> > What's the use case? Why do you need global block IO throttling?
> 
> Resource management for guest VMs. I want to make sure a VM cannot
> exceed a certain IO limit. If the VM does IOs to only one device, it
> gets the full specified bandwidth to that single device but if the VM
> talks to multiple devices, it only gets a fraction of the global
> bandwidth per device, not exceeded the global limit.

Can't we specify per device limit for VM if VM is doing IO to multiple
devices?

I know it is not exact replacement of global limits as one has to know
the topology and figure out to what all devices IO is happening. This
might get even more difficult if VM's disk is a file on filesystem on
host and not a device directly eported to guest.

Somebody also had mentioned that if we have mutiple layers of qemu
snapshots and if snapshots are sitting on different logical devices
then it becomes hard to figure out where all VM traffic is going
and how to even divide global limits in per device limits 

Also current IO controller does not work for NFS. So if a global limit
is imposed in higher layers it will solve that issue too.

Also if you implement it above page cache then it should automatically
handle the case of async writes too. The downside is that if limits
are low, buffered writers will be throttled very frequently.

So in general, I think global limits is a useful concept. Do you have
a implementation ready?

Thanks
Vivek

Re: Support for global block IO throttling?

[Juerg Haefliger]

>> Hi Vivek,
>>
>> >> Are there any plans to support global (per cgroup) IO throttling in
>> >> the near future? I'm in need of this and would gladly work on it if
>> >> nobody else is and if people think this is a useful feature. What I
>> >> envision is that one can specify the limits of individual devices in a
>> >> group but also specify the overall aggregate limit of the whole group.
>> >> I just started looking at the throttling code and haven't quite
>> >> figured it out yet so it might take me a bit to get up to speed.
>> >
>> > Hi Juerg,
>> >
>> > What's the use case? Why do you need global block IO throttling?
>>
>> Resource management for guest VMs. I want to make sure a VM cannot
>> exceed a certain IO limit. If the VM does IOs to only one device, it
>> gets the full specified bandwidth to that single device but if the VM
>> talks to multiple devices, it only gets a fraction of the global
>> bandwidth per device, not exceeded the global limit.
>
> Can't we specify per device limit for VM if VM is doing IO to multiple
> devices?

Yes we can but it's not exact. As I mentioned, the way I want to
specify it is so that the VMs alloted bandwidth is shared across all
the drives presented to that VM.

> I know it is not exact replacement of global limits as one has to know
> the topology and figure out to what all devices IO is happening. This
> might get even more difficult if VM's disk is a file on filesystem on
> host and not a device directly eported to guest.
>
> Somebody also had mentioned that if we have mutiple layers of qemu
> snapshots and if snapshots are sitting on different logical devices
> then it becomes hard to figure out where all VM traffic is going
> and how to even divide global limits in per device limits
>
> Also current IO controller does not work for NFS. So if a global limit
> is imposed in higher layers it will solve that issue too.
>
> Also if you implement it above page cache then it should automatically
> handle the case of async writes too. The downside is that if limits
> are low, buffered writers will be throttled very frequently.
>
> So in general, I think global limits is a useful concept. Do you have
> a implementation ready?

Err... No... I was hoping the community can give me some pointers and
infos to get me started. Just like the details you mentioned above
which are very useful.

Thanks
...Juerg


> Thanks
> Vivek
>

Re: Support for global block IO throttling?

[Juerg Haefliger]

Hi Vivek,


> Also current IO controller does not work for NFS. So if a global limit
> is imposed in higher layers it will solve that issue too.
>
> Also if you implement it above page cache then it should automatically
> handle the case of async writes too. The downside is that if limits
> are low, buffered writers will be throttled very frequently.

I'm not exactly sure where in the stack you see this global bandwidth
controller. What do you mean by 'above the page cache'? I interpret
this such that access to the page cache would be bandwidth limited as
well? That doesn't sound right to me.

Thanks
...Juerg

Re: Support for global block IO throttling?

[Vivek Goyal]

On Fri, Apr 01, 2011 at 10:05:41AM +0200, Juerg Haefliger wrote:
> Hi Vivek,
> 
> 
> > Also current IO controller does not work for NFS. So if a global limit
> > is imposed in higher layers it will solve that issue too.
> >
> > Also if you implement it above page cache then it should automatically
> > handle the case of async writes too. The downside is that if limits
> > are low, buffered writers will be throttled very frequently.
> 
> I'm not exactly sure where in the stack you see this global bandwidth
> controller. What do you mean by 'above the page cache'? I interpret
> this such that access to the page cache would be bandwidth limited as
> well? That doesn't sound right to me.

You are right. For reads it will not make sense to implement it above
page cache as reads coming out of page cache need not be throttled.

So it will probably boil down to that implement buffered write throttling
when entring the page cache and read throttling somewhere below.
generic_make_request()? But that will not take care of network file system
case I think. 

May be we can enhance blk_throtl_bio() itself to also check for global
rules and subject bio to first global rules and put in a global
queue. This probably will work for your virtual machine cases but
it does not take care of buffered writes and NFS IO. May be that is
still a good first step towards global IO limit implementation.

Thanks
Vivek