[PATCH v6 00/21] KVM: s390: vsie: Add VSIE SIGP ECALL and SRSI Interpretation

[Christoph Schlameuss <schlameuss@linux.ibm.com>]

In the IBM Z machine generation z17 the s390x architecture does have a
VSIE Interpretation Extension Facility to improve guest-3
guest performance.

To exploit the machine support the guest-1 KVM needs to create and
maintain shadow structures pointing to the original state descriptions
and system control areas of currently running guest-3 configurations.
These pointers are followed by the machines firmware and modifications
of the original SCA for guest-3 (located in guest-2) are monitored and
handled by firmware. This results in fewer VSIE exits.

The SIGP orders EXTERNAL CALL (2) and SENSE RUNNING STATUS (21) can be
handled if requested by the setting the ECA_SIGPI respective ECB_SRSI
state description bits.

The series includes some changes to the non ssca path as well.
- original SCA pinning is lazily kept after exiting vsie
- addr_to_page radix_tree usage is replaced by xarray

**Performance Expectations**
SIGP sense running instructions perform about −97% faster without the
intercept. External call is harder to measure but about the same
(faster) on the sending side.
Entry/exit are within +-4% variation against the prior state.
Only tested on shared systems so far, so the numbers are to be taken
with caution.

**Limitations**
Due to no ESCA support in old g2 kernel and qemu the patches do
introduce a limitation for setups with a unaware g2 when g1 KVM has
ssca=1 the g2 kernel will report KVM_CAP_MAX_VCPUS=64.

---
Do not expect any answers in the next two weeks.

---
Changes in v6:
- Fix valid sashiko findings
- Change some get_vsie_* functions to have a normal rc and out parameter
  instead of returning ERR_PTR
- Re-insert xa_erase hunk lost on rebasing
- Fix compile problems in intermediate patches
- Link to v5: https://patch.msgid.link/20260826-vsie-sigpi-v5-0-cf471edb46b2@linux.ibm.com

Changes in v5:
- Rebased to current master already containing dependency
- Dropped "Guard against invalid CPU address" patch, replaced with
  masking icpua usages - upper bits of icpua do not need to be checked
- Switched patch order 05 <-> 06 as un/pin_scb() depends on
  un/pin_guest_pages()
- Fix data race on flags fields (reported by sashiko)
- Remove addr_to_page usage on ssca path - vsie_pages are held in
  vsie_sca
- Change pin_guest_pages() to rc and out-parameter for clarity
- Inject intercept into current cpu on bad secondary cpu flags
- Harmonized method names (reset_vsie_page, init_vsie_page)
- Extracted _shadow_sca() to make _shadow_sca_cpu() to make easier to
  read
- Fixed non ssca path loosing scao on reentry
- Link to v4: https://patch.msgid.link/20260812-vsie-sigpi-v4-0-f19e49666315@linux.ibm.com

Changes in v4:
- Sorry for the spam!
- Same code as v2 / v3, rebased to v7.2-rc7 so that sashiko can
  definitely apply it and provide feedback
- Removed dependency on change-id: 20260716-vsie-cleanup-202607-8690ba7ac3ea:v1
- Link to v3: https://patch.msgid.link/20260811-vsie-sigpi-v3-0-45aa1ab07c7c@linux.ibm.com

Changes in v3:
- Same code as v2, just rebased to kvms390/next so that sashiko can
  apply it and provide feedback
- Link to v2: https://patch.msgid.link/20260810-vsie-sigpi-v2-0-e8d59a2f2f70@linux.ibm.com

Changes in v2:

Changes requested by Janosch (Thanks!)
- First 3 commits split out to its own vsie-cleanup series
- Re-split main code into commits a bit differently and hopefully easier
- Renamed sclp bit to has_vsie_interp_extf
- Renamed module parameter to ssca
- Renamed arch struct variable to ssca
- Add head and tail struct groups in struct vsie_sca
- Make writes to ssca atomic
- Add patch introducing SCA_ALIGNMENT_SHIFT
- Force vsie_sca->ssca offset to 0 instead of complex assertion
- Do not error out on scb_addr 0 (architecture compliant)
- move VSIE_PAGE_PINNED const
- decrease sca_o_nr_pages variable size

- Fixed Sashiko findings
  - allow 4 page esca if properly aligned
  - put sca on error in get_vsie_sca()
  - force reshadow on _shadow_sca() fail
  - unpin when destroy vsie_page
  - fix locking in sca change cases
  - fix writeback of SCAO in unpin_blocks() if g3 does not use 64bscao
  - fix err goto from shadow_sca()
  - set validity on shadow sca from pin_blocks()
  - remove vsie_page mention from comment
  - fix get_sca_entry_addr() for sca pages 2-5
  - fixed commit message, fix double read conditions for configuration values
  - use xarray insert instead of store and change alloc flags to *_ACCOUNT
  - fix double read conditions for max_sc* configuration values
  - empty xarrays on destroy
  - correct stat counter increments
- Link to v1: https://lore.kernel.org/r/20260709-vsie-sigpi-v1-0-ea9f12066408@linux.ibm.com

Changes from RFC:
- fix locking and vsie_page reuse to run stable
- pull some methods out of the main patch to simplify the review
- rework kvm_s390_handle_vsie() to allocate vsie_sca first to store
  vsie_page references directly
- move sie_uses_esca() to patch 1 as it is used there as well - thanks
  Eric
- change pointers in {read,write}_scao() to generic 'unsigned long' to
  not imply gpa or hpa as it can in fact be either depending on the
  situation - thanks Claudio
- simplify code in sca_mcn_equals() using memcmp() - thanks Janosch
- copy mcn into vsie_sca using memcpy() instead of loop
- only shadow sca if the g3 wants and can to use sigpif - thanks Janosch
- change vsie_sca->ssca from pointer to embedded struct to as vsie_sca
  is only needed when we actually also want the ssca
- fix kvm_s390_handle_vsie() cleanup order
- rename sca variables to vsie_sca
- cleanup error paths
- adjust some comments

To: kvm@vger.kernel.org
To: linux-s390@vger.kernel.org
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Claudio Imbrenda <imbrenda@linux.ibm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Eric Farman <farman@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Janosch Frank <frankja@linux.ibm.com>
Cc: Nico Boehr <nrb@linux.ibm.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Sean Christopherson <seanjc@google.com>

---
Christoph Schlameuss (21):
      KVM: s390: vsie: Add SCAO read and write helpers
      KVM: s390: vsie: Move SCAO validation into a function
      KVM: s390: vsie: Add vsie_interp_extf detection
      KVM: s390: vsie: Add ssca_block and ssca_entry structs
      KVM: s390: vsie: Move pin/unpin guest page
      KVM: s390: vsie: Move pin/unpin_scb methods
      KVM: s390: vsie: Move release/acquire gmap shadow
      KVM: s390: vsie: Create helpers to alloc and free vsie_pages
      KVM: s390: vsie: Replace radix_tree with xarray addr_to_page
      KVM: s390: vsie: Refactor kvm_s390_vsie_destroy and extract reusable methods
      KVM: s390: vsie: Add helper reset_vsie_page() and unshadow_intercept()
      KVM: s390: vsie: Add helper unshadow_intercept()
      KVM: s390: vsie: Lazily keep original scb pinned after vsie exit
      KVM: s390: vsie: Add helper to pin and unpin multiple guest pages
      KVM: s390: vsie: Add struct vsie_sca with pin and unpin methods
      KVM: s390: vsie: Shadow VSIE SCA in guest-1
      KVM: s390: vsie: Allow guest-3 cpu add and remove with ssca
      KVM: s390: vsie: Add VSIE max shadow configuration
      KVM: s390: vsie: Add VSIE shadow stat counters
      KVM: s390: vsie: Create minimal scb shadows for not running g3 blocks
      KVM: s390: vsie: Enable use of VSIE SSCA

 arch/s390/include/asm/kvm_host_s390.h          |   28 +-
 arch/s390/include/asm/kvm_host_s390_types.h    |   28 +-
 arch/s390/include/asm/sclp.h                   |    1 +
 arch/s390/kvm/s390/s390.c                      |   15 +-
 arch/s390/kvm/s390/vsie.c                      | 1320 +++++++++++++++++++-----
 drivers/s390/char/sclp_early.c                 |    1 +
 tools/testing/selftests/kvm/include/s390/sie.h |    2 +-
 7 files changed, 1118 insertions(+), 277 deletions(-)
---
base-commit: 73e3f0710014fe6d4ed98cfc02292f6121db7558
change-id: 20251113-vsie-sigpi-3de01243efd6

Best regards,
--  
Christoph Schlameuss <schlameuss@linux.ibm.com>