question about the usage of avg_lat in io latency controller

question about the usage of avg_lat in io latency controller

[xiaohui li]

hi Dennis Zhou and other who are familiar with io latency controller :

i have a question about the io latency controller :
how to configure the min latency value of a blk cgroup ?
using the avg_lat value may be not right.

from the Documentation/admin-guide/cgroup-v2.rst, i know we can do
configuring work in this way:
-------------------
Use the avg_lat value as a basis for
your real setting, setting at 10-15% higher than the value in io.stat
-------------------

but when i have found  the avg_lat value is the total sum of running
average of io latency in the past time, and it can't reflect the
average time cost of per single IO request.
but whether one thread can be throttled depends on the compare result
of stat.mean and iolat->min_lat_nsec.
and stat.mean can reflect the average time cost of per single IO request.

so from above analysis, if do the configuring min io latency value
work of a blk group, use the avg_lat may not be appropriate, because
it is the total sum of running average.
why not make use of the stat.mean to do the configuring min io latency
value work ?

one experiment on my device:
cat io.stat
8:0 rbytes=586723328 wbytes=99248033792 rios=143243 wios=331782
dbytes=0 dios=0 use_delay=12 delay_nsec=0 depth=1 avg_lat=11526
win=800

so the avg_lat value 11526(ns) is so big, it can't be the average time
cost of per single IO request on our device.
so it can not be used for do the configuring min io latency value work.

Maybe I'm not familiar with the io latency controller code.
If any mistake exist in my above analysis, welcome to add your suggestions.

Re: question about the usage of avg_lat in io latency controller

[Dennis Zhou]

Hello,

On Tue, May 26, 2020 at 07:27:34PM +0800, xiaohui li wrote:
> hi Dennis Zhou and other who are familiar with io latency controller :
> 
> i have a question about the io latency controller :
> how to configure the min latency value of a blk cgroup ?
> using the avg_lat value may be not right.
> 
> from the Documentation/admin-guide/cgroup-v2.rst, i know we can do
> configuring work in this way:
> -------------------
> Use the avg_lat value as a basis for
> your real setting, setting at 10-15% higher than the value in io.stat
> -------------------
> 
> but when i have found  the avg_lat value is the total sum of running
> average of io latency in the past time, and it can't reflect the
> average time cost of per single IO request.
> but whether one thread can be throttled depends on the compare result
> of stat.mean and iolat->min_lat_nsec.
> and stat.mean can reflect the average time cost of per single IO request.
> 
> so from above analysis, if do the configuring min io latency value
> work of a blk group, use the avg_lat may not be appropriate, because
> it is the total sum of running average.
> why not make use of the stat.mean to do the configuring min io latency
> value work ?
> 

The logic isn't perfect. The idea is that you only care about this
metric when the device under some constant load. For spinning disks,
it's calculated by contributing the average latency of an interval
cur_win_nsec's long to an exponential moving average. See
iolat_update_total_lat_avg().

> one experiment on my device:
> cat io.stat
> 8:0 rbytes=586723328 wbytes=99248033792 rios=143243 wios=331782
> dbytes=0 dios=0 use_delay=12 delay_nsec=0 depth=1 avg_lat=11526
> win=800
> 
> so the avg_lat value 11526(ns) is so big, it can't be the average time
> cost of per single IO request on our device.
> so it can not be used for do the configuring min io latency value work.
> 

Yeah so I can't remember if this change preceeded the other change, but
timing is now calculated based on issue time which comes the time marked
before it enters the block layer. This means it includes any delays that
have been incurred. But from the output, you can see that you're at
depth=1 and use_delay=12 meaning it is in play iirc.

To get the value you're looking for, we'd have to plumb through
rq->io_start_time_ns. There is an inherent mismatch here as
blk-iolatency works at the bio layer and not the request layer.

However, blk-iolatency is meant to protect a single cgroup from the
system. In this case, it seems like given your workload, workload + the
workload on the system is saturating the disk. Meaningfully, someone
else is throttling you as their iolatency is being violated. I'd
consider switching to iocost for a proportional io controller.

> Maybe I'm not familiar with the io latency controller code.
> If any mistake exist in my above analysis, welcome to add your suggestions.

Thanks,
Dennis