Re: [ckrm-tech] [RFC 3/5] sched: Add CPU rate hard caps

[Shailabh Nagar <nagar@watson.ibm.com>]

Kirill Korotaev wrote:

>>>>- disk I/O bandwidth:
>>>>we started to use CFQv2, but it is quite poor in this regard. First, it 
>>>>doesn't prioritizes writes and async disk operations :( And even for 
>>>>sync reads we found some problems we work on now...
>>>>        
>>>>
>
>  
>
>>CKRM (on e-series) had an implementation based on a modified CFQ
>>scheduler. Shailabh is currently working on porting that controller to
>>f-series.
>>    
>>
>can you explain what was changed by CKRM there? Did you made it to 
>control ASYNC read/writes? I don't think so...
>  
>
In e-series, CFQ was modified to
- maintain request queues per ckrm-class (now resource group) rather 
than per-tgid
- explicitly maintain I/O bandwidth of each request queue (in terms of 
I/O issued by the I/O scheduler)
- select the "next request queue to service" based on its I/O 
bandwidth...if a queue exceeds its allocation (as calculated
from the CKRM guarantee values), the queue gets skipped.

So this did not use the CFQ priority scheme as such and only implemented 
the "limit" part.

The current plan is to exploit the CFQ prio levels and rely on CFQ doing 
a good enough job in maintaining an adequate
bandwidth differential between those prio levels.
Again, each queue would maintain a count of its consumed bandwidth as 
well as target bandwidth. While picking the next request
from the queue, if its observed that the queue is above its "guarantee", 
its priority will get reduced (it'll still supply a request) while
a queue that is below its share will get bumped up....Control will be 
much more gradual but the basic idea is to leverage CFQ's priority
handling than supplant it (since we get anticipation in the form of 
time-slicing for free).

One concern is whether the time-slicing of CFQ plays well with queues 
that aren't organized by tgid...I'm still looking into that.

>Do you have any plots on what is concurrent bandwidth is depending on 
>weights? Because, our measurements show that CFQ is not ideal and 
>behaves poorly when prio 0,5,6,7 are used :/ Only 1,2,3,4 are really 
>linear-scalable...
>  
>
Interesting. Whats the time-scale over which you expect I/O bandwidth 
rates to get enforced ?

Perhaps the iosched discussion should use  a different thread....

--Shailabh