[PATCH 00/15] KVM: arm64: Improvements to GICv3 LPI injection

Oliver Upton
2024-01-24T20:49:22Z

The unfortunate reality is there are increasingly large systems that are
shipping today without support for GICv4 vLPI injection. Serialization
in KVM's LPI routing/injection code has been a significant bottleneck
for VMs on these machines when under a high load of LPIs (e.g. a
multi-queue NIC).

Even though the long-term solution is quite clearly **direct
injection**, we really ought to do something about the LPI scaling
issues within KVM.

This series aims to improve the performance of LPI routing/injection in
KVM by moving readers of LPI configuration data away from the
lpi_list_lock in favor or using RCU.

Patches 1-5 change out the representation of LPIs in KVM from a
linked-list to an xarray. While not strictly necessary for making the
locking improvements, this seems to be an opportune time to switch to a
data structure that can actually be indexed.

Patches 6-10 transition vgic_get_lpi() and vgic_put_lpi() away from
taking the lpi_list_lock in favor of using RCU for protection. Note that
this requires some rework to the way references are taken on LPIs and
how reclaim works to be RCU safe.

Lastly, patches 11-15 rework the LRU policy on the LPI translation cache
to not require moving elements in the linked-list and take advantage of
this to make it an rculist readable outside of the lpi_list_lock.

All of this was tested on top of v6.8-rc1. Apologies if any of the
changelogs are a bit too light, I'm happy to rework those further in
subsequent revisions.

I would've liked to have benchmark data showing the improvement on top
of upstream with this series, but I'm currently having issues with our
internal infrastructure and upstream kernels. However, this series has
been found to have a near 2x performance improvement to redis-memtier [*]
benchmarks on our kernel tree.

[*] https://github.com/RedisLabs/memtier_benchmark

Oliver Upton (15):
KVM: arm64: vgic: Store LPIs in an xarray
KVM: arm64: vgic: Use xarray to find LPI in vgic_get_lpi()
KVM: arm64: vgic-v3: Iterate the xarray to find pending LPIs
KVM: arm64: vgic-its: Walk the LPI xarray in vgic_copy_lpi_list()
KVM: arm64: vgic: Get rid of the LPI linked-list
KVM: arm64: vgic: Use atomics to count LPIs
KVM: arm64: vgic: Free LPI vgic_irq structs in an RCU-safe manner
KVM: arm64: vgic: Rely on RCU protection in vgic_get_lpi()
KVM: arm64: vgic: Ensure the irq refcount is nonzero when taking a ref
KVM: arm64: vgic: Don't acquire the lpi_list_lock in vgic_put_irq()
KVM: arm64: vgic-its: Lazily allocate LPI translation cache
KVM: arm64: vgic-its: Pick cache victim based on usage count
KVM: arm64: vgic-its: Protect cached vgic_irq pointers with RCU
KVM: arm64: vgic-its: Treat the LPI translation cache as an rculist
KVM: arm64: vgic-its: Rely on RCU to protect translation cache reads

arch/arm64/kvm/vgic/vgic-debug.c | 2 +-
arch/arm64/kvm/vgic/vgic-init.c | 7 +-
arch/arm64/kvm/vgic/vgic-its.c | 190 ++++++++++++++++++-------------
arch/arm64/kvm/vgic/vgic-v3.c | 3 +-
arch/arm64/kvm/vgic/vgic.c | 56 +++------
arch/arm64/kvm/vgic/vgic.h | 12 +-
include/kvm/arm_vgic.h | 9 +-
7 files changed, 146 insertions(+), 133 deletions(-)

base-commit: 6613476e225e090cc9aad49be7fa504e290dd33d
--
2.43.0.429.g432eaa2c6b-goog