Re: [RFC 0/3] block: proportional based blk-throttling

[Shaohua Li <shli@fb.com>]

On Fri, Jan 22, 2016 at 01:08:44PM -0500, Tejun Heo wrote:
> Hello, Shaohua.
> 
> On Fri, Jan 22, 2016 at 09:57:10AM -0800, Shaohua Li wrote:
> > > Let's say per-cgroup buffer budget B is calculated as, say, 100ms
> > > worth of IO cost (or bandwidth or iops) available to the cgroup.  In
> > > practice, this may have to be adjusted down depending on the number of
> > > cgroups performing active IOs.  For a given cgroup, B can be
> > > distributed among the CPUs that are actively issuing IOs in that
> > > cgroup.  It will degenerate to round robin of small budget if there
> > > are too many active for the budget available but for most cases this
> > > will cut down most of cross-CPU traffic.
> > 
> > The cgroup could be a single thread. It uses cpu0's per-cpu budget B-1,
> > move to cpu1 and use another B - 1, and so on
> 
> Sure, just ensure that the total cached is bound by B and expire if
> not used over a certain amount of time.  The thing is as long as we
> can go through percpu cache most of the time, it's all fine.  We can
> spend a lot of processing budget for corner cases.
> 
> > >  cost = F + R * size
> > 
> > F could be IOPS. and the real cost becomes R. How do you get R? We can't
> > simply use R(4k) = 1, R(8k) = 2 .... I tried the idea several years ago:
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lwn.net_Articles_474164_&d=CwIBAg&c=5VD0RTtNlTh3ycd41b3MUw&r=X13hAPkxmvBro1Ug8vcKHw&m=4X56EQmXhfF82BH-eQkQL08afWwbrOErtEVkn5xKsWA&s=_IkvDWMM7AXgh840OrQKndkJpBVcKrGhgLnHkA_aYNg&e= 
> > The idea is the same. But the reality is we can't get R. I don't want to
> > have a random math working for one SSD but not for another.
> 
> Yeah, it'll have to be adaptive.  We can't use fixed values; however,
> note that using bandwidth means that we assume F == 0 and R == 1,
> which wouldn't be appropriate for most devices.

It's true bandwidth means R == 1. But it has a kind of adaptive. The
cgroup bandwidth == share * disk_bandwidth. disk_bandwidth is adaptive.
It might not work well if cgroups have completely different IO pattern
though. 

Thanks,
Shaohua