blkcg throttle settings page cache writes

blkcg throttle settings page cache writes

[Nikolay Borisov]

Hello,

I'm observing blkcg throttle limits to not apply on writes which go
through the page cache. E.g. I have a container whose write iops limits
are set to 300 I then run a fio job with direct=0 (the default) and can
see that writes/s (via iostat) are around 6-7k. Tracing confirmed thae
AIO that fio is submitting is being handled by a kworker thread, which
is in the root cgroup and as such has no lio limits. Direct writes/reads
work as expected and are throttled.

After Paolo Valente's patch
(https://patchwork.kernel.org/patch/9249247/) I was left with the
impression that bios submitted would just carry the information of the
submitted throughout the storage stack. Apparently this is not the case.
I also went through an old thread I started on dm-devel dealing with
this exact problem (even before Paolo's patch), where Vivel Goyal hinted
at this particular problem :
https://www.redhat.com/archives/dm-devel/2016-March/msg00012.html

So Tejun, my question is what is the problem for the blkcg information
to be transferred alongside the bio (which I believed to be happening
with the blkcg_association calls) so that even buffered writes are
throttled via the cgroup policies? Or are the numbers I'm seeing in
iostat actually the writes to the page-cache and not the underlying
block device?

Regards,
Nikolay

Re: blkcg throttle settings page cache writes

[Tejun Heo]

Hello,

On Wed, Sep 14, 2016 at 11:25:49AM +0300, Nikolay Borisov wrote:
> Hello,
> So Tejun, my question is what is the problem for the blkcg information
> to be transferred alongside the bio (which I believed to be happening
> with the blkcg_association calls) so that even buffered writes are
> throttled via the cgroup policies? Or are the numbers I'm seeing in
> iostat actually the writes to the page-cache and not the underlying
> block device?

You gotta use cgroup v2 for that.  Controlling buffered writes
requires coordination between memory and io controllers which isn't
possible with cgroup v1.

Thanks.

-- 
tejun

Re: blkcg throttle settings page cache writes

[Nikolay Borisov]

On 09/16/2016 06:57 PM, Tejun Heo wrote:
> Hello,
> 
> On Wed, Sep 14, 2016 at 11:25:49AM +0300, Nikolay Borisov wrote:
>> Hello,
>> So Tejun, my question is what is the problem for the blkcg information
>> to be transferred alongside the bio (which I believed to be happening
>> with the blkcg_association calls) so that even buffered writes are
>> throttled via the cgroup policies? Or are the numbers I'm seeing in
>> iostat actually the writes to the page-cache and not the underlying
>> block device?
> 
> You gotta use cgroup v2 for that.  Controlling buffered writes
> requires coordination between memory and io controllers which isn't
> possible with cgroup v1.

Thanks for the information Tejun, that makes sense. However, in my
testing I observed that when a throttle limit is set on IOPS it's not
being enforced for buffered writes, but when it's set on bps it is
actually working as expected. Is this expected or are there conflating
factors which I'm not taking into account and as such what I'm seeing -
iostat reports perfect utilisation of a particular device according to
the bps throttle limit.


> 
> Thanks.
> 

Re: blkcg throttle settings page cache writes

[Tejun Heo]

Hello,

On Tue, Sep 27, 2016 at 02:28:25PM +0300, Nikolay Borisov wrote:
> Thanks for the information Tejun, that makes sense. However, in my
> testing I observed that when a throttle limit is set on IOPS it's not
> being enforced for buffered writes, but when it's set on bps it is
> actually working as expected. Is this expected or are there conflating
> factors which I'm not taking into account and as such what I'm seeing -
> iostat reports perfect utilisation of a particular device according to
> the bps throttle limit.

If you're saying that you're seeing buffered writes controlled on
cgroup v1 in non-root cgroups, there's something very wrong with the
test setup.  The code simply isn't there.

Thanks.

-- 
tejun