Re: [RFC PATCH v3 0/6] Add support for AMD IOMMU GAPPI

[Sairaj Kodilkar <sarunkod@amd.com>]

Hi everyone

Gentle ping...

On 7/13/2026 4:20 PM, Sairaj Kodilkar wrote:
> Introduction
> ============
> On newer generations of AMD processors, IOMMU AVIC/x2AVIC guest-mode interrupt
> remapping can use Guest APIC Physical Processor Interrupt (GAPPI) as an
> alternative host-notification path when device interrupts target a vCPU that
> is not running (IRTE[IsRun] = 0).
> 
> With AVIC enabled, the IOMMU posts device interrupts into the guest virtual
> APIC backing page. When the vCPU is not running, KVM must additionally be
> notified so it can schedule the vCPU.
> 
> The legacy notification path is the GA log (GALOG): the IOMMU appends vCPU
> tags to a shared GA log buffer and raises a single GA log interrupt.  KVM
> registers a notifier and scans the buffer to decide which vCPUs to wake.
> Under heavy interrupt load this adds latency and can overflow the buffer
> because all wakeups funnel through one interrupt and one shared log.
> 
> Guest APIC Physical Processor Interrupt (GAPPI), defined in section 2.2.5.4
> of the AMD I/O Virtualization Technology (IOMMU) Specification [1], is an
> alternative.  With GAPPI enabled, the IOMMU still updates the guest vAPIC
> backing page IRR, but may deliver a physical APIC interrupt directly to
> IRTE[Destination], using IRTE[GATag][7:0] as the vector.  This distributes
> host wakeup notifications across CPUs instead of centralizing them in a
> log buffer.
> 
> This series programs guest-mode IRTEs accordingly: IRTE[Destination] carries
> the target host physical APIC ID, IRTE[GATag] is set to
> POSTED_INTR_WAKEUP_VECTOR, and IRTE[GAPPIDis] / IRTE[GALogIntr] are set
> based on whether KVM requests host wakeup.  GAPPI is selected at boot via
> the amd_iommu=gappi kernel parameter on capable hardware, otherwise the
> existing GA log path is unchanged.
> 
> 
> SVM/AMD IOMMU interface changes
> ===============================
> The first four patches refactor the SVM/AMD IOMMU interface ahead of GAPPI.
> 
> The cpu field is renamed to apicid because it carries the host physical
> APIC ID for IRTE[Destination], not a Linux CPU number.
> 
> The ga_log_intr boolean is renamed to wakeup_intr (and the synthetic
> AVIC_PHYSICAL_ID_ENTRY_GA_LOG_INTR shadow bit to
> AVIC_PHYSICAL_ID_ENTRY_WAKEUP_INTR). wakeup_intr describes KVM's intent
> (request host wakeup while the vCPU is not running), not a specific hardware
> mechanism.
> 
> A separate is_running boolean is added to IOMMU interface because GAPPI
> requires a valid apicid in IRTE[Destination] even when the vCPU is not running. 
> The prior encoding (apicid >= 0 means running, apicid == -1 means not running)
> no longer works once apicid carries the GAPPI destination while IRTE[IsRun] is
> clear.  The IOMMU driver keys IRTE[IsRun] and destination programming off this
> explicit boolean instead of inferring running state from apicid.
> 
> 
> KVM GAPPI wakeup scheme
> =======================
> SVM follows the Intel posted-interrupt wakeup model already used by VMX.
> Each pCPU maintains a list of blocked vCPUs that may be woken by a GAPPI
> delivery to that CPU.  When a vCPU blocks while waiting for a device
> interrupt, SVM enqueues it on the wakeup list of the pCPU on which it was
> previously running (gappi_cpu) and passes that pCPU's physical APIC ID to
> the IOMMU to program IRTE[Destination].  The rationale is that the vCPU is
> likely to run again on the same pCPU, which is common when vCPUs are pinned;
> targeting GAPPI notifications there reduces unnecessary VMEXITs from GAPPI
> deliveries on other CPUs.  When the vCPU is scheduled in again, it is
> removed from the list and IRTE[Destination] is updated to the current pCPU.
> 
> SVM registers avic_gappi_wakeup_handler() via
> kvm_set_posted_intr_wakeup_handler().  On POSTED_INTR_WAKEUP_VECTOR delivery,
> the handler walks the local per-CPU list and wakes vCPUs with a pending
> LAPIC IRR.  The IOMMU has already posted the interrupt into the guest
> vAPIC; waking the vCPU lets it observe the pending interrupt and run.
> 
> List maintenance is hooked into the existing AVIC vCPU and IRQ affinity
> paths: vCPU load/put through avic_update_iommu_vcpu_affinity(), the first
> IRQ affined to a non-running vCPU through avic_pi_update_irte() when ir_list
> was empty at put time, removal when the last IRTE is detached, and cleanup
> on vCPU destroy.  All GAPPI-specific logic is gated on amd_iommu_gappi.
> 
> 
> Changes since v2
> ================
> https://lore.kernel.org/linux-iommu/20260708091408.12106-1-sarunkod@amd.com/
> 
> Patch[1-6]
>   - Expand commit messages to explain GAPPI, the interface changes, and the
>     per-CPU wakeup list scheme [Sean].
> 
> Patch[1-3]
>   - Split the monolithic SVM/IOMMU API refactor into four preparatory
>     patches [Sean]
>   - Rename posted_intr to wakeup_intr to reflect host wakeup intent, not
>     guest interrupt posting [Sean]
>   - Pass vCPU running status with a extra parameter (is_running) instead of
>     flags.
> 
> Patch[4,5]
>   - Move ga_tag=POSTED_INTR_WAKEUP_VECTOR setting from IOMMU to SVM layer.
> 
> 
> Changes since V1:
> ================
> https://lore.kernel.org/all/20260626105906.14577-1-sarunkod@amd.com/
> 
> Patch4
>     - Disable interrupts while holding wakeup list lock inside [sashiko]
>       avic_add_vcpu_to_gappi_wakeup_list and avic_remove_vcpu_from_gappi_wakeup_list
>     - Unregister posted_intr_wakeup_handler during module unload [sashiko]
> 
> Patch5
>     - Disable GAPPI feature during kexec and suspend path [sashiko]
> 
> 
> ------
> [1] https://docs.amd.com/v/u/en-US/48882_3.11_IOMMU_PUB
> 
> Sairaj Kodilkar (6):
>   iommu/amd: KVM: SVM: Rename cpu to apicid in IOMMU interface
>   iommu/amd: KVM: SVM: Rename ga_log_intr to wakeup_intr in IOMMU
>     interface
>   iommu/amd: KVM: SVM: Add explicit vCPU running state to IOMMU
>     interface
>   iommu/amd: Program guest-mode IRTEs for GAPPI wakeup when IRTE[IsRun]
>     = 0
>   KVM: SVM: Add support for AMD IOMMU Guest APIC Physical Processor
>     Interrupt (GAPPI)
>   iommu/amd: Provide kernel command line option to enable GAPPI
> 
>  .../admin-guide/kernel-parameters.txt         |   3 +-
>  arch/x86/include/asm/irq_remapping.h          |   5 +-
>  arch/x86/include/asm/svm.h                    |   9 +-
>  arch/x86/kvm/svm/avic.c                       | 173 ++++++++++++++----
>  arch/x86/kvm/svm/svm.c                        |   2 +
>  arch/x86/kvm/svm/svm.h                        |   5 +
>  drivers/iommu/amd/amd_iommu.h                 |   1 +
>  drivers/iommu/amd/amd_iommu_types.h           |   6 +-
>  drivers/iommu/amd/init.c                      |  31 +++-
>  drivers/iommu/amd/iommu.c                     |  59 +++---
>  include/linux/amd-iommu.h                     |  13 +-
>  11 files changed, 237 insertions(+), 70 deletions(-)
> 
> 
> base-commit: 8cd9520d35a6c38db6567e97dd93b1f11f185dc6