[PATCH v4 0/8] Reinvent BQL-free PIO/MMIO

[Igor Mammedov <imammedo@redhat.com>]

v4:
  * improoved doc comment for  memory_region_enable_lockless_io()
  * add additional helper to set interrupt to pair with new cpu_test_interrupt()
    and use it tree wide to set interrutps
  * merge 'cpu_test_interrupt()/cpu_set_interrupt()' with patches
    that use them (v3 6-7,9/10).
  * pick up acks
v3:
  * hpet: replace explicit atomics with use seqlock API  (PeterX)
  * introduce cpu_test_interrupt() (Paolo)
    and use it tree wide for checking interrupts
  * don't take BQL for setting exit_request, use qatomic_set() instead. (Paolo)
  * after above change, relace conditional BQL with unconditional
    to simlify things a bit (Paolo)
  * drop not needed barriers (Paolo)
  * minor tcg:cpu_handle_interrupt() cleanup

v2:
  * Make both read and write pathes BQL-less (Gerd)
  * Refactor HPET to handle lock-less access correctly
    when stopping/starting counter in parallel. (Peter Maydell)
  * Publish kvm-unit-tests HPET bench/torture test [1] to verify
    HPET lock-less handling

When booting WS2025 with following CLI
 1)   -M q35,hpet=off -cpu host -enable-kvm -smp 240,sockets=4
the guest boots very slow and is sluggish after boot
or it's stuck on boot at spinning circle (most of the time).

pref shows that VM is experiencing heavy BQL contention on IO path
which happens to be ACPI PM timer read access. A variation with
HPET enabled moves contention to HPET timer read access.
And it only gets worse with increasing number of VCPUs.

Series prevents large VM vCPUs contending on BQL due to PM|HPET timer
access and lets Windows to move on with boot process.

Testing lock-less IO with HPET micro benchmark [2] shows approx 80%
better performance than the current BLQ locked path.
[chart https://ibb.co/MJY9999 shows much better scaling of lock-less
IO compared to BQL one.]

In my tests, with CLI WS2025 guest wasn't able to boot within 30min
on both hosts
  * 32 core 2NUMA nodes
  * 448 cores 8NUMA nodes
With ACPI PM timer in BQL-free read mode, guest boots within approx:
 * 2min
 * 1min
respectively.

With HPET enabled boot time shrinks ~2x
 * 4m13 -> 2m21
 * 2m19 -> 1m15
respectively.

2) "[kvm-unit-tests PATCH v4 0/5] x86: add HPET counter tests"
    https://lore.kernel.org/kvm/20250725095429.1691734-1-imammedo@redhat.com/T/#t
PS:
Using hv-time=on cpu option helps a lot (when it works) and
lets [1] guest boot fine in ~1-2min. Series doesn't make
a significant impact in this case.

PS2:
Tested series with a bunch of different guests:
 RHEL-[6..10]x64, WS2012R2, WS2016, WS2022, WS2025

PS3:
 dropped mention of https://bugzilla.redhat.com/show_bug.cgi?id=1322713
 as it's not reproducible with current software stack or even with
 the same qemu/seabios as reported (kernel versions mentioned in
 the report were interim ones and no longer available,
 so I've used nearest released at the time for testing) 

git tree: https://gitlab.com/imammedo/qemu lockless_io_v4

CC: Paolo Bonzini <pbonzini@redhat.com>
CC: Peter Xu <peterx@redhat.com>
CC: "Michael S. Tsirkin" <mst@redhat.com>
CC: mtosatti@redhat.com 

Igor Mammedov (8):
  memory: reintroduce BQL-free fine-grained PIO/MMIO
  acpi: mark PMTIMER as unlocked
  hpet: switch to fain-grained device locking
  hpet: move out main counter read into a separate block
  hpet: make main counter read lock-less
  add cpu_test_interrupt()/cpu_set_interrupt() helpers and use them tree
    wide
  kvm: i386: irqchip: take BQL only if there is an interrupt
  tcg: move interrupt caching and single step masking closer to user

 include/hw/core/cpu.h               | 25 ++++++++++++++++
 include/system/memory.h             | 12 ++++++++
 accel/tcg/cpu-exec.c                | 25 +++++++---------
 accel/tcg/tcg-accel-ops.c           |  2 +-
 accel/tcg/user-exec.c               |  2 +-
 hw/acpi/core.c                      |  1 +
 hw/intc/s390_flic.c                 |  2 +-
 hw/openrisc/cputimer.c              |  2 +-
 hw/timer/hpet.c                     | 38 +++++++++++++++++++-----
 system/cpus.c                       |  2 +-
 system/memory.c                     | 15 ++++++++++
 system/physmem.c                    |  2 +-
 target/alpha/cpu.c                  |  8 ++---
 target/arm/cpu.c                    | 20 ++++++-------
 target/arm/helper.c                 | 18 +++++------
 target/arm/hvf/hvf.c                |  6 ++--
 target/avr/cpu.c                    |  2 +-
 target/hppa/cpu.c                   |  2 +-
 target/i386/hvf/hvf.c               |  4 +--
 target/i386/hvf/x86hvf.c            | 21 +++++++------
 target/i386/kvm/kvm.c               | 46 ++++++++++++++---------------
 target/i386/nvmm/nvmm-all.c         | 24 +++++++--------
 target/i386/tcg/system/seg_helper.c |  2 +-
 target/i386/tcg/system/svm_helper.c |  2 +-
 target/i386/whpx/whpx-all.c         | 34 ++++++++++-----------
 target/loongarch/cpu.c              |  2 +-
 target/m68k/cpu.c                   |  2 +-
 target/microblaze/cpu.c             |  2 +-
 target/mips/cpu.c                   |  6 ++--
 target/mips/kvm.c                   |  2 +-
 target/openrisc/cpu.c               |  3 +-
 target/ppc/cpu_init.c               |  2 +-
 target/ppc/kvm.c                    |  2 +-
 target/rx/cpu.c                     |  3 +-
 target/rx/helper.c                  |  2 +-
 target/s390x/cpu-system.c           |  2 +-
 target/sh4/cpu.c                    |  2 +-
 target/sh4/helper.c                 |  2 +-
 target/sparc/cpu.c                  |  2 +-
 target/sparc/int64_helper.c         |  4 +--
 40 files changed, 211 insertions(+), 144 deletions(-)

-- 
2.47.1