[RFC] distinguish foreground and background IOs in block throttle

[RFC] distinguish foreground and background IOs in block throttle

[xuejiufei]

Hi all,

Cgroup writeback is supported since v4.2. I found there exists a
problem in the following case.

A cgroup may send both buffer and direct/sync IOs. The foreground
thread will be stalled when periodic writeback IOs is flushed because
the service queue already has a plenty of writeback IOs, then
foreground IOs should be enqueued with its FIFO policy.

I wonder if we can distinguish foreground and background IOs in block
throttle to fix the above problem.

Any suggestion are always appreciated.


Thanks,
Jiufei

Re: [RFC] distinguish foreground and background IOs in block throttle

[Paolo Valente]

[-- Attachment #1: Type: text/plain, Size: 1319 bytes --]



> Il giorno 25 dic 2017, alle ore 03:44, xuejiufei <xuejiufei@gmail.com> ha scritto:
> 
> Hi all,
> 
> Cgroup writeback is supported since v4.2. I found there exists a
> problem in the following case.
> 
> A cgroup may send both buffer and direct/sync IOs. The foreground
> thread will be stalled when periodic writeback IOs is flushed because
> the service queue already has a plenty of writeback IOs, then
> foreground IOs should be enqueued with its FIFO policy.
> 
> I wonder if we can distinguish foreground and background IOs in block
> throttle to fix the above problem.
> 
> Any suggestion are always appreciated.
> 

Hi,
to address similar issues, I have just sent a patch [1] for the BFQ
I/O scheduler.  If you want to give it a try, it might solve, or at
least mitigate, your problem (the patch does not involve groups,
though, at least for the moment).

There are still pending patches related to the low_latency mode of
BFQ, so I suggest you to try with low_latency disabled, i.e., as root:
echo bfq > /sys/block/<your-device>/queue/scheduler
echo 0 > /sys/block/<your-device>/queue/iosched/low_latency 

For your possible convenience, I have attached the patch, gzipped, to
this email too.

Thanks,
Paolo

[1] https://www.spinics.net/lists/kernel/msg2684463.html


[-- Attachment #2: 0001-block-bfq-limit-tags-for-writes-and-async-I-O.patch.gz --]
[-- Type: application/x-gzip, Size: 2413 bytes --]

[-- Attachment #3: Type: text/plain, Size: 25 bytes --]



> 
> Thanks,
> Jiufei

Re: [RFC] distinguish foreground and background IOs in block throttle

[Joseph Qi]

Hi Paolo,

On 17/12/27 20:36, Paolo Valente wrote:
> 
> 
>> Il giorno 25 dic 2017, alle ore 03:44, xuejiufei <xuejiufei-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> ha scritto:
>>
>> Hi all,
>>
>> Cgroup writeback is supported since v4.2. I found there exists a
>> problem in the following case.
>>
>> A cgroup may send both buffer and direct/sync IOs. The foreground
>> thread will be stalled when periodic writeback IOs is flushed because
>> the service queue already has a plenty of writeback IOs, then
>> foreground IOs should be enqueued with its FIFO policy.
>>
>> I wonder if we can distinguish foreground and background IOs in block
>> throttle to fix the above problem.
>>
>> Any suggestion are always appreciated.
>>
> 
> Hi,
> to address similar issues, I have just sent a patch [1] for the BFQ
> I/O scheduler.  If you want to give it a try, it might solve, or at
> least mitigate, your problem (the patch does not involve groups,
> though, at least for the moment).
> 
> There are still pending patches related to the low_latency mode of
> BFQ, so I suggest you to try with low_latency disabled, i.e., as root:
> echo bfq > /sys/block/<your-device>/queue/scheduler
> echo 0 > /sys/block/<your-device>/queue/iosched/low_latency 
> 
> For your possible convenience, I have attached the patch, gzipped, to
> this email too.
> > Thanks,
> Paolo
> 
> [1] https://www.spinics.net/lists/kernel/msg2684463.html
> 
I don't get why the issue Jiufei described has relations with scheduler.

IMO, the core reason is current we only have read/write queues in block
throttle. That means all sync/async writes will go into the same queue.
Once writeback IOs flush out and throttle happens, sync write will also
have to be queued up. Since there are more async writes ahead of sync
writes in the queue, and the current policy is dispatching 6 reads and 2
writes during each round, sync writes will get significantly delay,
which we don't expect.

So like read/write queue design, we may add another queue in block
throttle to distinguish sync/async writes, and then we can dispatch more
sync writes than async writes.

Thanks,
Joseph

Re: [RFC] distinguish foreground and background IOs in block throttle

[Joseph Qi]

Hi Paolo,

On 17/12/27 20:36, Paolo Valente wrote:
> 
> 
>> Il giorno 25 dic 2017, alle ore 03:44, xuejiufei <xuejiufei@gmail.com> ha scritto:
>>
>> Hi all,
>>
>> Cgroup writeback is supported since v4.2. I found there exists a
>> problem in the following case.
>>
>> A cgroup may send both buffer and direct/sync IOs. The foreground
>> thread will be stalled when periodic writeback IOs is flushed because
>> the service queue already has a plenty of writeback IOs, then
>> foreground IOs should be enqueued with its FIFO policy.
>>
>> I wonder if we can distinguish foreground and background IOs in block
>> throttle to fix the above problem.
>>
>> Any suggestion are always appreciated.
>>
> 
> Hi,
> to address similar issues, I have just sent a patch [1] for the BFQ
> I/O scheduler.  If you want to give it a try, it might solve, or at
> least mitigate, your problem (the patch does not involve groups,
> though, at least for the moment).
> 
> There are still pending patches related to the low_latency mode of
> BFQ, so I suggest you to try with low_latency disabled, i.e., as root:
> echo bfq > /sys/block/<your-device>/queue/scheduler
> echo 0 > /sys/block/<your-device>/queue/iosched/low_latency 
> 
> For your possible convenience, I have attached the patch, gzipped, to
> this email too.
> > Thanks,
> Paolo
> 
> [1] https://www.spinics.net/lists/kernel/msg2684463.html
> 
I don't get why the issue Jiufei described has relations with scheduler.

IMO, the core reason is current we only have read/write queues in block
throttle. That means all sync/async writes will go into the same queue.
Once writeback IOs flush out and throttle happens, sync write will also
have to be queued up. Since there are more async writes ahead of sync
writes in the queue, and the current policy is dispatching 6 reads and 2
writes during each round, sync writes will get significantly delay,
which we don't expect.

So like read/write queue design, we may add another queue in block
throttle to distinguish sync/async writes, and then we can dispatch more
sync writes than async writes.

Thanks,
Joseph

Re: [RFC] distinguish foreground and background IOs in block throttle

[Paolo Valente]

> Il giorno 28 dic 2017, alle ore 02:51, Joseph Qi <jiangqi903@gmail.com> ha scritto:
> 
> Hi Paolo,
> 
> On 17/12/27 20:36, Paolo Valente wrote:
>> 
>> 
>>> Il giorno 25 dic 2017, alle ore 03:44, xuejiufei <xuejiufei@gmail.com> ha scritto:
>>> 
>>> Hi all,
>>> 
>>> Cgroup writeback is supported since v4.2. I found there exists a
>>> problem in the following case.
>>> 
>>> A cgroup may send both buffer and direct/sync IOs. The foreground
>>> thread will be stalled when periodic writeback IOs is flushed because
>>> the service queue already has a plenty of writeback IOs, then
>>> foreground IOs should be enqueued with its FIFO policy.
>>> 
>>> I wonder if we can distinguish foreground and background IOs in block
>>> throttle to fix the above problem.
>>> 
>>> Any suggestion are always appreciated.
>>> 
>> 
>> Hi,
>> to address similar issues, I have just sent a patch [1] for the BFQ
>> I/O scheduler.  If you want to give it a try, it might solve, or at
>> least mitigate, your problem (the patch does not involve groups,
>> though, at least for the moment).
>> 
>> There are still pending patches related to the low_latency mode of
>> BFQ, so I suggest you to try with low_latency disabled, i.e., as root:
>> echo bfq > /sys/block/<your-device>/queue/scheduler
>> echo 0 > /sys/block/<your-device>/queue/iosched/low_latency 
>> 
>> For your possible convenience, I have attached the patch, gzipped, to
>> this email too.
>>> Thanks,
>> Paolo
>> 
>> [1] https://www.spinics.net/lists/kernel/msg2684463.html
>> 
> I don't get why the issue Jiufei described has relations with scheduler.
> 
> IMO, the core reason is current we only have read/write queues in block
> throttle. That means all sync/async writes will go into the same queue.
> Once writeback IOs flush out and throttle happens, sync write will also
> have to be queued up. Since there are more async writes ahead of sync
> writes in the queue, and the current policy is dispatching 6 reads and 2
> writes during each round, sync writes will get significantly delay,
> which we don't expect.
> 
> So like read/write queue design, we may add another queue in block
> throttle to distinguish sync/async writes, and then we can dispatch more
> sync writes than async writes.
> 

Hi Joseph,
your description of the problem would be complete *if* sync write
requests were issued as expected.  But the problem I have seen, and
tried to solve with my patch, is that async requests easily eat all
request tags (I'm referring to blk-mq of course).  So sync I/O has a
hard time finding free tags for its requests, and is then throttled
upstream.  Differentiating throttling downstream may not solve this
problem, because async requests will accumulate upstream as before.
The problem may even get worse, because downstream throttling may
increase the duration of upstream congestion.

My patch tries to solve this upstream problem.  BFQ then does the
rest, downstream, for me.  Things may change in your scenario, though:
you may not have this upstream problem at all, or you may have both
the upstream and the downstream problems.

Thanks,
Paolo

> Thanks,
> Joseph

Re: [RFC] distinguish foreground and background IOs in block throttle

[Paolo Valente]

> Il giorno 28 dic 2017, alle ore 02:51, Joseph Qi =
<jiangqi903@gmail.com> ha scritto:
>=20
> Hi Paolo,
>=20
> On 17/12/27 20:36, Paolo Valente wrote:
>>=20
>>=20
>>> Il giorno 25 dic 2017, alle ore 03:44, xuejiufei =
<xuejiufei@gmail.com> ha scritto:
>>>=20
>>> Hi all,
>>>=20
>>> Cgroup writeback is supported since v4.2. I found there exists a
>>> problem in the following case.
>>>=20
>>> A cgroup may send both buffer and direct/sync IOs. The foreground
>>> thread will be stalled when periodic writeback IOs is flushed =
because
>>> the service queue already has a plenty of writeback IOs, then
>>> foreground IOs should be enqueued with its FIFO policy.
>>>=20
>>> I wonder if we can distinguish foreground and background IOs in =
block
>>> throttle to fix the above problem.
>>>=20
>>> Any suggestion are always appreciated.
>>>=20
>>=20
>> Hi,
>> to address similar issues, I have just sent a patch [1] for the BFQ
>> I/O scheduler.  If you want to give it a try, it might solve, or at
>> least mitigate, your problem (the patch does not involve groups,
>> though, at least for the moment).
>>=20
>> There are still pending patches related to the low_latency mode of
>> BFQ, so I suggest you to try with low_latency disabled, i.e., as =
root:
>> echo bfq > /sys/block/<your-device>/queue/scheduler
>> echo 0 > /sys/block/<your-device>/queue/iosched/low_latency=20
>>=20
>> For your possible convenience, I have attached the patch, gzipped, to
>> this email too.
>>> Thanks,
>> Paolo
>>=20
>> [1] https://www.spinics.net/lists/kernel/msg2684463.html
>>=20
> I don't get why the issue Jiufei described has relations with =
scheduler.
>=20
> IMO, the core reason is current we only have read/write queues in =
block
> throttle. That means all sync/async writes will go into the same =
queue.
> Once writeback IOs flush out and throttle happens, sync write will =
also
> have to be queued up. Since there are more async writes ahead of sync
> writes in the queue, and the current policy is dispatching 6 reads and =
2
> writes during each round, sync writes will get significantly delay,
> which we don't expect.
>=20
> So like read/write queue design, we may add another queue in block
> throttle to distinguish sync/async writes, and then we can dispatch =
more
> sync writes than async writes.
>=20

Hi Joseph,
your description of the problem would be complete *if* sync write
requests were issued as expected.  But the problem I have seen, and
tried to solve with my patch, is that async requests easily eat all
request tags (I'm referring to blk-mq of course).  So sync I/O has a
hard time finding free tags for its requests, and is then throttled
upstream.  Differentiating throttling downstream may not solve this
problem, because async requests will accumulate upstream as before.
The problem may even get worse, because downstream throttling may
increase the duration of upstream congestion.

My patch tries to solve this upstream problem.  BFQ then does the
rest, downstream, for me.  Things may change in your scenario, though:
you may not have this upstream problem at all, or you may have both
the upstream and the downstream problems.

Thanks,
Paolo

> Thanks,
> Joseph