All of lore.kernel.org
 help / color / mirror / Atom feed
From: Jens Axboe <axboe@kernel.dk>
To: Stefan Hajnoczi <stefanha@redhat.com>
Cc: John Maguire <jmaguire2013@gmail.com>,
	f.ebner@proxmox.com, fam@euphon.net, hreitz@redhat.com,
	kwolf@redhat.com, qemu-block@nongnu.org, qemu-devel@nongnu.org,
	t.lamprecht@proxmox.com
Subject: Re: Excessive IO PSI for iothread when using io_uring since QEMU 10.2
Date: Wed, 18 Nov 2026 12:51:21 -0700	[thread overview]
Message-ID: <962d44d3-ca7b-4cc8-b0f7-77fb8d63cf41@kernel.dk> (raw)
In-Reply-To: <20261118185617.GA670091@shajnocz.csb>

On 11/18/26 11:56 AM, Stefan Hajnoczi wrote:
> On Wed, Nov 18, 2026 at 08:58:11AM -0700, Jens Axboe wrote:
>> On 11/15/26 4:05 PM, John Maguire wrote:
>>> On Tue, Jun 02, 2026, Stefan Hajnoczi wrote:
>>>> Pinging Jens: io_uring accounts POLL_ADD SQEs as blocking I/O activity
>>>> whereas select(2)/poll(2)/epoll_wait(2) do not. Would it make sense to
>>>> follow the same accounting as the syscalls for this operation since that
>>>> is probably expected?
>>>
>>> Hello,
>>>
>>> I run Incus on two hosts and see misleading iowait and IO pressure on
>>> both since Incus 7.6, which puts virtio disks on a dedicated iothread by
>>> default. Incus users now hit this without opting into iothreads.
>>>
>>> Host 1: Arch Linux, stock QEMU 11.1.3, kernel 6.18.62, 16 CPUs, five
>>> VMs. /proc/pressure/io shows about 80% "some" and 75% "full", with
>>> 20-30% iowait. The disks are lightly loaded (under 15% util), there are
>>> no D-state tasks, the pressure is confined to the cgroup holding the
>>> QEMU processes, and each QEMU process has one thread in io_cqring_wait.
>>> It started the day I upgraded from Incus 7.5.1 to 7.6; QEMU 11.1.x had
>>> been in use on this host since late September without it.
>>>
>>> Host 2: Debian 13, kernel 6.12.116, 56 CPUs, four VMs, Zabbly's Incus
>>> packages. These bundle QEMU 11.1.3 with the pve-qemu patch
>>> "fdmon-io_uring: avoid idle event loop being accounted as IO wait".
>>> Pressure is about 68% "some" and 65% "full" with disks under 1% util,
>>> and again one thread per QEMU process in io_cqring_wait. This kernel
>>> does not advertise IORING_FEAT_NO_IOWAIT (added in 6.15), so the patch
>>> has no effect there.
>>>
>>> So the downstream patch only helps on 6.15+ kernels, and distros
>>> shipping unpatched QEMU get these numbers by default.
>>>
>>> Is there a preferred way forward? Fiona offered to submit the Proxmox
>>> workaround to qemu-devel, and Stefan asked whether io_uring should
>>> instead account POLL_ADD the way poll(2) does. Either would fix this
>>> for users on distro packages.
>>
>> We can just queue up the patches for stable so they bubble back to
>> older versions, I think that's the cleanest approach and gets us
>> a unified interface across supported kernel versions.
> 
> Fiona: On the QEMU side, please submit the QEMU IORING_FEAT_NO_IOWAIT
> patch upstream and let's merge it on qemu.git/master and
> qemu.git/stable. Given that the kernel feature is available, QEMU might
> as well use it.
> 
> Jens: I still think io_uring operations should perform accounting inside
> the kernel the same way as the equivalent syscalls do (e.g. poll(2) does
> not count as iowait).

And slow everything down...

> Pushing the responsibility of deciding what counts as iowait to
> userspace via IORING_FEAT_NO_IOWAIT means the decision is duplicated in
> every application that uses io_uring. That only makes sense to me if
> accounting should work differently on a per-application basis and
> applications therefore need full control. I'm not aware of applications
> actually wanting to do that? Or is iowait fundamentally incompatible
> with io_uring so that applications are the only ones who can decide?

iowait is fundamentally incompatible with sane accounting in the first
place, that is the problem. It really tells you nothing, and was
deprecated since the days of uniprocessor systems. The problem here is
not how iowait is done in the kernel, it's that people are using this
metric in the first place. Which is why you just want to NOT do that,
and do sane accounting on your disk drives, if you care about that kind
of thing.

Which is why I refuse to cater to any kind of iowait accounting metrics,
it's just adding overhead to support a metric that is largely
nonsensical in the first place. It doesn't answer any questions that an
application/tier owner would want to know. Those questions might be
things like "am I I/O bound?" or "is the device I'm using being slow?".
I wish people would realize that, rather than continue to place value in
iowait as a metric for ANYTHING, and stop trying to work around it.

-- 
Jens Axboe


  reply	other threads:[~2026-11-18 19:52 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-15 23:05 Excessive IO PSI for iothread when using io_uring since QEMU 10.2 John Maguire
2026-11-18 14:06 ` Fernando Casas Schösow
2026-11-18 15:58 ` Jens Axboe
2026-11-18 18:56   ` Stefan Hajnoczi
2026-11-18 19:51     ` Jens Axboe [this message]
2026-11-19 12:46       ` Stefan Hajnoczi
2026-11-19 19:00   ` Jens Axboe
  -- strict thread matches above, loose matches on Subject: below --
2026-04-24 10:25 Fiona Ebner
2026-04-27 19:13 ` Stefan Hajnoczi
2026-04-28 12:10   ` Fiona Ebner
2026-04-28 13:31     ` Fiona Ebner
2026-04-28 16:19     ` Stefan Hajnoczi
2026-04-29  8:00       ` Fiona Ebner
2026-04-29 12:20         ` Stefan Hajnoczi
2026-06-01 17:20         ` Stefan Hajnoczi
2026-06-02  8:41           ` Fiona Ebner
2026-06-02 12:08             ` Stefan Hajnoczi

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=962d44d3-ca7b-4cc8-b0f7-77fb8d63cf41@kernel.dk \
    --to=axboe@kernel.dk \
    --cc=f.ebner@proxmox.com \
    --cc=fam@euphon.net \
    --cc=hreitz@redhat.com \
    --cc=jmaguire2013@gmail.com \
    --cc=kwolf@redhat.com \
    --cc=qemu-block@nongnu.org \
    --cc=qemu-devel@nongnu.org \
    --cc=stefanha@redhat.com \
    --cc=t.lamprecht@proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.