Re: [PATCH RFC v1 0/3] aio-poll: improve aio-polling efficiency

[Stefan Hajnoczi <stefanha@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 14999 bytes --]

On Mon, Mar 23, 2026 at 09:08:13AM -0500, JAEHOON KIM wrote:
> On 3/9/2026 3:46 PM, JAEHOON KIM wrote:
> > On 2/26/2026 12:03 AM, JAEHOON KIM wrote:
> > > On 2/20/2026 1:00 PM, JAEHOON KIM wrote:
> > > > On 2/19/2026 4:27 PM, Stefan Hajnoczi wrote:
> > > > > Hi Jaehoon,
> > > > > Following the call earlier this week I ran a single fio job to get a
> > > > > clearer picture of:
> > > > > 1. The QEMU 10.0.0 regression that prompted you to optimize AioContext
> > > > >     polling.
> > > > > 2. How the poll-weight parameter affects IOPS.
> > > > > 
> > > > > run      rw        bs   numjobs iothreads iops   diff
> > > > > v9.2.0   randread  8k   1       1         174944 3.6%
> > > > > v10.0.0  randread  8k   1       1         174285 3.2%
> > > > > baseline randread  8k   1       1         168908 0.0%
> > > > > w2       randread  8k   1       1         163718 -3.1%
> > > > > w3       randread  8k   1       1         165805 -1.8%
> > > > > w4       randread  8k   1       1         167388 -0.9%
> > > > > 
> > > > > This time I only ran randread bs=8k iodepth=8 numjobs=1 with a single
> > > > > IOThread.
> > > > > 
> > > > > Observations:
> > > > > 
> > > > > - There might be an IOPS regression between v10.0.0 and the baseline
> > > > >    (9ad7f544c696) that your patches apply on top of. This is different
> > > > >    from the CPU utilization regression that you found in v9.2.0 ->
> > > > >    v10.0.0. I will bisect it.
> > > > > 
> > > > > - poll-weight=3 and 4 improve IOPS to a level that is acceptable. CPU
> > > > >    utilization looks like this:
> > > > > 
> > > > > run         %usr     %nice      %sys   %iowait    %steal
> > > > > %irq     %soft    %guest    %gnice     %idle
> > > > > baseline   49.37      0.00     31.10      0.00      0.00
> > > > > 11.61      0.04      0.00      0.00      7.89
> > > > > w2         46.24      0.00     32.61      0.00      0.00
> > > > > 11.84      0.10      0.00      0.00      9.21
> > > > > w3         48.04      0.00     32.17      0.00      0.00
> > > > > 11.98      0.08      0.00      0.00      7.73
> > > > > w4         48.56      0.00     31.23      0.00      0.00
> > > > > 11.48      0.03      0.00      0.00      8.69
> > > > > 
> > > > > poll-weight=2 is the winner at CPU utilization. I'm not sure if
> > > > > poll-weight=3 will produce an acceptable CPU utilization
> > > > > improvement for
> > > > > you. Do you have data or want to re-run to measure poll-weight=3?
> > > > > 
> > > > > Stefan
> > > > 
> > > > Thank you very much for sharing the detailed measurement results.
> > > > I truly appreciate the effort.
> > > > 
> > > > Regarding w=3, I will discuss with our performance team to see
> > > > if the CPU consumption levels are acceptable within our internal
> > > > test environment. I will get back to you with more definitive data
> > > > as soon as possible.
> > > > 
> > > > Thanks again for your thorough analysis.
> > > > 
> > > > Regards,
> > > > Jaehoon.
> > > > 
> > > Hello Stefan,
> > > 
> > > Thank you for your patience. I would like to share the observed
> > > changes in throughput
> > > and CPU consumption in our performance test environment as follows.
> > > 
> > > We observed that when using W=3, performance returns to a level
> > > comparable to
> > > QEMU v9.1, while W=2 results in slightly lower CPU consumption.
> > > 
> > > Our initial preference is to use W=2 by default.
> > > However, we fully understand your concerns regarding the potential
> > > performance drop.
> > > Should the patch be accepted, we would provide guidance on using the
> > > weighted value as
> > > a configurable option.
> > > 
> > > For reference, we consider values between -2 and 2 as noise in our
> > > analysis.
> > > I look forward to your feedback.
> > > 
> > > The table below shows a comparison between:
> > > *  - Host:* RHEL10.1-GA+qemu-10.0.0-14.el10_1, *Guest:* RHEL 9.6 GA vs.
> > > *  - Host:* RHEL10.1-GA+qemu-10.0.0-14.el10_1 (w=2, w=3), *Guest:*
> > > RHEL 9.6 GA
> > >     for FIO FCP and FICON with 1 iothread and 8 iothreads.
> > >     The values shown are the averages for numjobs 1, 4, and 8.
> > > 
> > > FIO FCP -1 iothread
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > |                     | Seq. Read  | Seq. Read  | Seq. Write | Seq.
> > > Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
> > > |                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3
> > > (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > | Throughput avg      |   -3.00    |   -2.33    |    0.00    |
> > >  -0.33    |   -3.00    |   -3.33    |    1.33    |   -0.33    |
> > > | CPU consumption avg |   -5.67    |   -4.33    |   -6.33    |
> > >  -5.33    |   -7.33    |   -5.33    |  -10.33    |   -8.67    |
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > 
> > > FIO FCP -8 iothread
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > |                     | Seq. Read  | Seq. Read  | Seq. Write | Seq.
> > > Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
> > > |                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3
> > > (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > | Throughput avg      |   -5.00    |   -4.00    |   -3.67    |
> > >  -3.33    |   -4.67    |   -4.00    |   -0.33    |   -0.67    |
> > > | CPU consumption avg |  -13.00    |  -10.67    |  -16.00    |
> > > -14.33    |  -14.00    |   -9.33    |  -13.67    |  -11.00    |
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > 
> > > 
> > > FIO FICON -1 iothread
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > |                     | Seq. Read  | Seq. Read  | Seq. Write | Seq.
> > > Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
> > > |                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3
> > > (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > | Throughput avg      |   -0.67    |   -0.67    |   -6.33    |
> > >  -6.67    |   -0.67    |    0.00    |    1.33    |    1.33    |
> > > | CPU consumption avg |   -7.67    |   -7.33    |  -13.67    |
> > > -13.00    |   -9.00    |   -8.33    |   -5.00    |   -4.33    |
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > 
> > > FIO FICON -8 iothread
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > |                     | Seq. Read  | Seq. Read  | Seq. Write | Seq.
> > > Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
> > > |                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3
> > > (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > | Throughput avg      |   -3.00    |   -2.67    |   -7.33    |
> > >  -7.00    |   -0.67    |   -1.00    |    0.67    |    0.67    |
> > > | CPU consumption avg |  -16.33    |  -14.33    |  -25.33    |
> > > -27.00    |   -8.67    |   -7.00    |   -6.67    |   -5.00    |
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > 
> > > 
> > > The table below shows a comparison between:
> > > * - Host:* RHEL 10.0 GA + qemu-9.1.0-15.el10, *Guest:* RHEL 9.6 GA vs.
> > > * - Host:* RHEL 10.1 GA + qemu-10.0.0-14.el10_1 (w=2, w=3), *Guest:*
> > > RHEL 9.6 GA.
> > > 
> > > FIO FCP -1 iothread
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > |                     | Seq. Read  | Seq. Read  | Seq. Write | Seq.
> > > Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
> > > |                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3
> > > (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > | Throughput avg      |   -0.67    |    0.00    |   -0.33    |
> > >  -1.00    |   -1.00    |   -0.67    |    3.33    |    2.00    |
> > > | CPU consumption avg |    0.67    |    2.00    |    1.67    | 3.00 
> > >   |   -3.33    |   -0.33    |   -2.33    |    0.00    |
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > 
> > > FIO FCP -8 iothread
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > |                     | Seq. Read  | Seq. Read  | Seq. Write | Seq.
> > > Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
> > > |                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3
> > > (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > | Throughput avg      |   -2.00    |   -1.33    |   -1.00    | 0.00 
> > >   |   -0.33    |    1.00    |    1.00    |    0.67    |
> > > | CPU consumption avg |   -3.00    |   -1.00    |   -2.00    | 0.00 
> > >   |  -10.33    |   -5.67    |   -6.67    |   -3.33    |
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > 
> > > 
> > > FIO FICON -1 iothread
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > |                     | Seq. Read  | Seq. Read  | Seq. Write | Seq.
> > > Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
> > > |                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3
> > > (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > | Throughput avg      |   -1.67    |   -1.67    |   -0.33    | 0.00 
> > >   |   -1.33    |   -1.00    |   -1.67    |   -2.33    |
> > > | CPU consumption avg |   -0.33    |    1.00    |    1.00    | 2.00 
> > >   |   -2.33    |   -1.33    |   -1.33    |   -0.33    |
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > 
> > > FIO FICON -8 iothread
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > |                     | Seq. Read  | Seq. Read  | Seq. Write | Seq.
> > > Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
> > > |                     |   w2 (%)   |   w3 (%)   |   w2 (%)   |  w3
> > > (%)   |   w2 (%)   |   w3 (%)   |   w2 (%)   |   w3 (%)   |
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > | Throughput avg      |   -1.00    |   -0.67    |   -0.33    | 0.33 
> > >   |    0.67    |    0.33    |    0.33    |    0.33    |
> > > | CPU consumption avg |   -1.33    |    1.00    |    4.33    | 2.33 
> > >   |   -2.00    |    0.00    |   -3.00    |   -1.33    |
> > > +---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
> > > 
> > > 
> > > 
> > > Regards,
> > > Jaehoon.
> > > 
> > > 
> > Hello Stefan,
> > 
> > I’m following up to see if you’ve had a chance to review the performance
> > results I shared.
> > As I mentioned, we are completely fine with using w=3 as the default
> > value.
> > It effectively restores performance to QEMU v9.1 levels, and we will
> > provide guidance
> > for users who need further CPU savings to use w=2.
> > 
> > I’ve updated the patch to set the default value to 3 and include the
> > feedback so far.
> > Please let me know if you have any comments or if I should post the new
> > patch.
> > 
> > Regards,
> > Jaehoon
> > 
> > 
> Hello,
> 
> I'd like to follow up on this thread. I have posted a v2 incorporating the
> feedback so far.
> I'd really appreciate your thoughts when you have a chance.
> 
> Thanks again for your time.

Hi Jaehoon,
Sorry that I haven't replied yet. I will review your emails and the v2
patches tomorrow.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]