[GIT PULL] KVM: x86: TDX changes for 6.19

[Sean Christopherson <seanjc@google.com>]

Please pull a large overhaul of lock-related TDX code (particularly in the
S-EPT and mirror SPTE code), along with a few fixes and cleanups.

*Huge* kudos to Rick, Yan, Binbin, Ira, and Kai (hopefully I didn't forget
anyone) for their meticulous reviews, testing and debug, clever testcases,
and help determining exactly what scenarios KVM needs to deal with in terms
of avoiding lock contention in the TDX Module.

P.S. There are few one-off TDX changes in the "vmx" pull request.  I don't
     expect to have a dedicated TDX pull request for most releases, I created
     one this time around because of the scope of the overhaul.

The following changes since commit 6146a0f1dfae5d37442a9ddcba012add260bceb0:

  Linux 6.18-rc4 (2025-11-02 11:28:02 -0800)

are available in the Git repository at:

  https://github.com/kvm-x86/linux.git tags/kvm-x86-tdx-6.19

for you to fetch changes up to 398180f93cf3c7bb0ee3f512b139ad01843f3ddf:

  KVM: TDX: Use struct_size to simplify tdx_get_capabilities() (2025-11-13 08:30:07 -0800)

----------------------------------------------------------------
KVM TDX changes for 6.19:

 - Overhaul the TDX code to address systemic races where KVM (acting on behalf
   of userspace) could inadvertantly trigger lock contention in the TDX-Module,
   which KVM was either working around in weird, ugly ways, or was simply
   oblivious to (as proven by Yan tripping several KVM_BUG_ON()s with clever
   selftests).

 - Fix a bug where KVM could corrupt a vCPU's cpu_list when freeing a vCPU if
   creating said vCPU failed partway through.

 - Fix a few sparse warnings (bad annotation, 0 != NULL).

 - Use struct_size() to simplify copying capabilities to userspace.

----------------------------------------------------------------
Dave Hansen (2):
      KVM: TDX: Remove __user annotation from kernel pointer
      KVM: TDX: Fix sparse warnings from using 0 for NULL

Rick Edgecombe (1):
      KVM: TDX: Take MMU lock around tdh_vp_init()

Sean Christopherson (27):
      KVM: Make support for kvm_arch_vcpu_async_ioctl() mandatory
      KVM: Rename kvm_arch_vcpu_async_ioctl() to kvm_arch_vcpu_unlocked_ioctl()
      KVM: TDX: Drop PROVE_MMU=y sanity check on to-be-populated mappings
      KVM: x86/mmu: Add dedicated API to map guest_memfd pfn into TDP MMU
      KVM: x86/mmu: WARN if KVM attempts to map into an invalid TDP MMU root
      Revert "KVM: x86/tdp_mmu: Add a helper function to walk down the TDP MMU"
      KVM: x86/mmu: Rename kvm_tdp_map_page() to kvm_tdp_page_prefault()
      KVM: TDX: Return -EIO, not -EINVAL, on a KVM_BUG_ON() condition
      KVM: TDX: Fold tdx_sept_drop_private_spte() into tdx_sept_remove_private_spte()
      KVM: x86/mmu: Drop the return code from kvm_x86_ops.remove_external_spte()
      KVM: TDX: WARN if mirror SPTE doesn't have full RWX when creating S-EPT mapping
      KVM: TDX: Avoid a double-KVM_BUG_ON() in tdx_sept_zap_private_spte()
      KVM: TDX: Use atomic64_dec_return() instead of a poor equivalent
      KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
      KVM: TDX: ADD pages to the TD image while populating mirror EPT entries
      KVM: TDX: Fold tdx_sept_zap_private_spte() into tdx_sept_remove_private_spte()
      KVM: TDX: Combine KVM_BUG_ON + pr_tdx_error() into TDX_BUG_ON()
      KVM: TDX: Derive error argument names from the local variable names
      KVM: TDX: Assert that mmu_lock is held for write when removing S-EPT entries
      KVM: TDX: Add macro to retry SEAMCALLs when forcing vCPUs out of guest
      KVM: TDX: Add tdx_get_cmd() helper to get and validate sub-ioctl command
      KVM: TDX: Convert INIT_MEM_REGION and INIT_VCPU to "unlocked" vCPU ioctl
      KVM: TDX: Use guard() to acquire kvm->lock in tdx_vm_ioctl()
      KVM: TDX: Don't copy "cmd" back to userspace for KVM_TDX_CAPABILITIES
      KVM: TDX: Guard VM state transitions with "all" the locks
      KVM: TDX: Bug the VM if extending the initial measurement fails
      KVM: TDX: Use struct_size to simplify tdx_get_capabilities()

Thorsten Blum (1):
      KVM: TDX: Check size of user's kvm_tdx_capabilities array before allocating

Yan Zhao (2):
      KVM: TDX: Drop superfluous page pinning in S-EPT management
      KVM: TDX: Fix list_add corruption during vcpu_load()

 arch/arm64/kvm/arm.c               |   6 +
 arch/loongarch/kvm/Kconfig         |   1 -
 arch/loongarch/kvm/vcpu.c          |   4 +-
 arch/mips/kvm/Kconfig              |   1 -
 arch/mips/kvm/mips.c               |   4 +-
 arch/powerpc/kvm/Kconfig           |   1 -
 arch/powerpc/kvm/powerpc.c         |   4 +-
 arch/riscv/kvm/Kconfig             |   1 -
 arch/riscv/kvm/vcpu.c              |   4 +-
 arch/s390/kvm/Kconfig              |   1 -
 arch/s390/kvm/kvm-s390.c           |   4 +-
 arch/x86/include/asm/kvm-x86-ops.h |   1 +
 arch/x86/include/asm/kvm_host.h    |   7 +-
 arch/x86/kvm/mmu.h                 |   3 +-
 arch/x86/kvm/mmu/mmu.c             |  87 ++++-
 arch/x86/kvm/mmu/tdp_mmu.c         |  50 +--
 arch/x86/kvm/vmx/main.c            |   9 +
 arch/x86/kvm/vmx/tdx.c             | 712 ++++++++++++++++++-------------------
 arch/x86/kvm/vmx/tdx.h             |   8 +-
 arch/x86/kvm/vmx/x86_ops.h         |   1 +
 arch/x86/kvm/x86.c                 |  13 +
 include/linux/kvm_host.h           |  14 +-
 virt/kvm/Kconfig                   |   3 -
 virt/kvm/kvm_main.c                |   6 +-
 24 files changed, 496 insertions(+), 449 deletions(-)