CFQ or BFQ scheduler and XFS

CFQ or BFQ scheduler and XFS

[Gionatan Danti]

Hi list,
I have a question about CFQ scheduler and the old warning one can find 
on the faq page: "As of kernel 3.2.12, the default i/o scheduler, CFQ, 
will defeat much of the parallelization in XFS".

Can I ask for more information about the bad interaction between CFQ and 
XFS, and especially why it does defeat filesystem parallelization? Is 
this warning still valid? What about the newer BFQ?

Note: I always used deadline or noop with XFS, but I am facing a disk 
with random read starvation when NCQ is enabled and a mixed sequential & 
random load happens. So far I saw that the only scheduler (somewhat) 
immune to the issue is CFQ, probably because it does not mix IO from 
multiple processes (it issue IO from one process at time, if I 
understand it correctly).

Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: CFQ or BFQ scheduler and XFS

[Dave Chinner]

On Sun, Jan 23, 2022 at 10:27:32PM +0100, Gionatan Danti wrote:
> Hi list,
> I have a question about CFQ scheduler and the old warning one can find on
> the faq page: "As of kernel 3.2.12, the default i/o scheduler, CFQ, will
> defeat much of the parallelization in XFS".
> 
> Can I ask for more information about the bad interaction between CFQ and
> XFS, and especially why it does defeat filesystem parallelization? Is this
> warning still valid? What about the newer BFQ?

CFQ doesn't understand that IO from different threads/tasks are
related and so it cannot account for/control multi-threaded IO
workloads.  Given that XFS's journal IO is inherently a
multi-threaded IO workload, CFQ IO accounting and throttling simply
does not work properly with XFS or any other filesystem that
decouples IO from the user task context that requires the IO to be
done on it's behalf.

> Note: I always used deadline or noop with XFS, but I am facing a disk with
> random read starvation when NCQ is enabled and a mixed sequential & random
> load happens. So far I saw that the only scheduler (somewhat) immune to the
> issue is CFQ, probably because it does not mix IO from multiple processes
> (it issue IO from one process at time, if I understand it correctly).

And, as per above, that's exactly why it doesn't work well with XFS...

Read starvation during sequential writes sounds more like a problem
with the block layer writeback throttle or something you need to
use an IO controller (blk-cgroups) to address, not try to fix with a
IO scheduler...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: CFQ or BFQ scheduler and XFS

[Gionatan Danti]

Il 2022-01-23 23:52 Dave Chinner ha scritto:
> CFQ doesn't understand that IO from different threads/tasks are
> related and so it cannot account for/control multi-threaded IO
> workloads.  Given that XFS's journal IO is inherently a
> multi-threaded IO workload, CFQ IO accounting and throttling simply
> does not work properly with XFS or any other filesystem that
> decouples IO from the user task context that requires the IO to be
> done on it's behalf.

Hi Dave,
ah, so it forces all threads under the same time slice? I thought it was 
thread aware...
Is it the same even for BFQ?

> And, as per above, that's exactly why it doesn't work well with XFS...
> 
> Read starvation during sequential writes sounds more like a problem
> with the block layer writeback throttle or something you need to
> use an IO controller (blk-cgroups) to address, not try to fix with a
> IO scheduler...

To tell the truth, I simply disabled NCQ on the affected drive. I don't 
think anything is wrong with the block layer simply because I can 
constantly replicate the random read starvation by running, at the same 
time, the following fio commands against the raw block device:

fio --name=test --filename=/dev/sda --direct=1 --rw=read --runtime=30    
  #seq read
fio --name=test --filename=/dev/sda --direct=1 --rw=randread 
--runtime=30 #rnd read

The first fio sucks almost any resources, leaving the randread process 
with ~2 IOPS with frequent drops to 0.
Using CFQ seems to avoid total starvation, but I then remembered it has 
some scalability and performance issues.
Disabling NCQ solves the issue, at the cost of not using any hardware 
queue (obviously).

Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: CFQ or BFQ scheduler and XFS

[Dave Chinner]

On Mon, Jan 24, 2022 at 08:55:38AM +0100, Gionatan Danti wrote:
> Il 2022-01-23 23:52 Dave Chinner ha scritto:
> > CFQ doesn't understand that IO from different threads/tasks are
> > related and so it cannot account for/control multi-threaded IO
> > workloads.  Given that XFS's journal IO is inherently a
> > multi-threaded IO workload, CFQ IO accounting and throttling simply
> > does not work properly with XFS or any other filesystem that
> > decouples IO from the user task context that requires the IO to be
> > done on it's behalf.
> 
> Hi Dave,
> ah, so it forces all threads under the same time slice? I thought it was
> thread aware...

No, the opposite. e.g. thread A does IO and then runs fsync(), which
needs to force the log, that triggers CIL push work which runs in a
different thread B.  Thread A blocks waiting for the CIL push,
thread B runs immediately.  Thread B then issues log IO, but the IO
from thread B is queued by CFQ because thread A's timeslice hasn't
expired. Hence log force is delayed until thread A's timeslice
expires, even though it is being done on behalf of thread A and
thread A is blocked until the IO from Thread B is scheduled and
completed.

> Is it the same even for BFQ?

No idea - I use noop for everything these days because IO schedulers
often cause more problems than they solve on SSDs, sparse virtual
machine images, thinly provisioned storage, etc....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: CFQ or BFQ scheduler and XFS

[Gionatan Danti]

Il 2022-01-24 22:13 Dave Chinner ha scritto:
> No, the opposite. e.g. thread A does IO and then runs fsync(), which
> needs to force the log, that triggers CIL push work which runs in a
> different thread B.  Thread A blocks waiting for the CIL push,
> thread B runs immediately.  Thread B then issues log IO, but the IO
> from thread B is queued by CFQ because thread A's timeslice hasn't
> expired. Hence log force is delayed until thread A's timeslice
> expires, even though it is being done on behalf of thread A and
> thread A is blocked until the IO from Thread B is scheduled and
> completed.

Hi Dave,
ok, I can see now why it is so bad (especially when multiple threads are 
involved).

> 
> No idea - I use noop for everything these days because IO schedulers
> often cause more problems than they solve on SSDs, sparse virtual
> machine images, thinly provisioned storage, etc....
> 
> Cheers,
> 
> Dave.

Right...

Thank you so much for the clear explanation.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8