[PATCH v5 00/17] Enable VAS and NX-GZIP support on PowerVM

Haren Myneni
2021-06-13T10:53:58Z

Virtual Accelerator Switchboard (VAS) allows kernel subsystems
and user space processes to directly access the Nest Accelerator
(NX) engines which provides HW compression. The true user mode
VAS/NX support on PowerNV is already included in Linux. Whereas
PowerVM support is available from P10 onwards.

This patch series enables VAS / NX-GZIP on PowerVM which allows
the user space to do copy/paste with the same existing interface
that is available on PowerNV.

VAS Enablement:
- Get all VAS capabilities using H_QUERY_VAS_CAPABILITIES that are
available in the hypervisor. These capabilities tells OS which
type of features (credit types such as Default and Quality of
Service (QoS)). Also gives specific capabilities for each credit
type: Maximum window credits, Maximum LPAR credits, Target credits
in that parition (varies from max LPAR credits based DLPAR
operation), whether supports user mode COPY/PASTE and etc.
- Register LPAR VAS operations such as open window. get paste
address and close window with the current VAS user space API.
- Open window operation - Use H_ALLOCATE_VAS_WINDOW HCALL to open
window and H_MODIFY_VAS_WINDOW HCALL to setup the window with LPAR
PID and etc.
- mmap to paste address returned in H_ALLOCATE_VAS_WINDOW HCALL
- To close window, H_DEALLOCATE_VAS_WINDOW HCALL is used to close in
the hypervisor.

NX Enablement:
- Get NX capabilities from the the hypervisor which provides Maximum
buffer length in a single GZIP request, recommended minimum
compression / decompression lengths.
- Register to VAS to enable user space VAS API

Main feature differences with PowerNV implementation:
- Each VAS window will be configured with a number of credits which
means that many requests can be issues simultaniously on that
window. On PowerNV, 1K credits are configured per window.
Whereas on PowerVM, the hypervisor allows 1 credit per window
at present.
- The hypervisor introduced 2 different types of credits: Default -
Uses normal priority FIFO and Quality of Service (QoS) - Uses high
priority FIFO. On PowerVM, VAS/NX HW resources are shared across
LPARs. The total number of credits available on a system depends
on cores configured. We may see more credits are assigned across
the system than the NX HW resources can handle. So to avoid NX HW
contention, the hypervisor introduced QoS credits which can be
configured by system administration with HMC API. Then the total
number of available default credits on LPAR varies based on QoS
credits configured.
- On PowerNV, windows are allocated on a specific VAS instance
and the user space can select VAS instance with the open window
ioctl. Since VAS instances can be shared across partitions on
PowerVM, the hypervisor manages window allocations on different
VAS instances. So H_ALLOCATE_VAS_WINDOW allows to select by domain
indentifiers (H_HOME_NODE_ASSOCIATIVITY values by cpu). By default
the hypervisor selects VAS instance closer to CPU resources that the
parition uses. So vas_id in ioctl interface is ignored on PowerVM
except vas_id=-1 which is used to allocate window based on CPU that
the process is executing. This option is needed for process affinity
to NUMA node.

The existing applications that linked with libnxz should work as
long as the job request length is restricted to
req_max_processed_len.

Tested the following patches on P10 successfully with test cases
given: https://github.com/libnxz/power-gzip

Note: The hypervisor supports user mode NX from p10 onwards. Linux
supports user mode VAS/NX on P10 only with radix page tables.

Patch 1: Fix to release reference to tgid during window close
Patches 2- 6: Move the code that is needed for both PowerNV and
PowerVM to powerpc book3s platform directory
Patch 7: Modify vas-window struct to support both platforms
and the related changes.
Patch 8: Define HCALL and the related VAS/NXGZIP specific
structs.
Patch 9: Define QoS credit flag in window open ioctl
Patch 10: Implement Allocate, Modify and Deallocate HCALLs
Patch 11: Retrieve VAS capabilities from the hypervisor
Patch 12; Implement window operations and integrate with API
Patch 13: Setup IRQ and NX fault handling
Patch 14 - 15: Make the code common to add NX-GZIP enablement
Patch 16: Get NX capabilities from the hypervisor
patch 17; Add sysfs interface to expose NX capabilities

Changes in v2:
- Rebase on 5.12-rc6
- Moved VAS Kconfig changes to arch/powerpc/platform as suggested
by Christophe Leroy
- build fix with allyesconfig (reported by kernel test build)

Changes in v3:
- Rebase on 5.12-rc7
- Moved vas-api.c and VAS Kconfig changes to
arch/powerpc/platform/book3s as Michael Ellerman suggested

Changes in v4:
- Rebase on 5.13-rc2
- Changes based on review comments from Nicholas Piggin
- Add seperate patch to define user window operations
- Drop "sysfs interface to export VAS capabilities" patch
This interface is mainly needed for DLPAR operations
and this patch will be included when DLPAR/LPM support
is added.
- Other cleanup changes

Changes in v5:
- Rebase on 5.13-rc5
- Changes based on review comments from Nicholas Piggin
and Michael Ellerman:
- Add new patch to fix tgid reference release. This fix
should be also included in stable 5.8+
- Define platform specific VAS window structs
- Define helper functions to take and release pid, tgid
and mm references
- Other cleanup suggestions

Haren Myneni (17):
powerpc/powernv/vas: Release reference to tgid during window close
powerpc/vas: Move VAS API to book3s common platform
powerpc/powernv/vas: Rename register/unregister functions
powerpc/vas: Add platform specific user window operations
powerpc/vas: Create take/drop pid and mm reference functions
powerpc/vas: Move update_csb/dump_crb to common book3s platform
powerpc/vas: Define and use common vas_window struct
powerpc/pseries/vas: Define VAS/NXGZIP hcalls and structs
powerpc/vas: Define QoS credit flag to allocate window
powerpc/pseries/vas: Add hcall wrappers for VAS handling
powerpc/pseries/vas: Implement getting capabilities from hypervisor
powerpc/pseries/vas: Integrate API with open/close windows
powerpc/pseries/vas: Setup IRQ and fault handling
crypto/nx: Rename nx-842-pseries file name to nx-common-pseries
crypto/nx: Register and unregister VAS interface on PowerVM
crypto/nx: Get NX capabilities for GZIP coprocessor type
crypto/nx: Add sysfs interface to export NX capabilities

arch/powerpc/include/asm/hvcall.h | 7 +
arch/powerpc/include/asm/vas.h | 109 ++-
arch/powerpc/include/uapi/asm/vas-api.h | 6 +-
arch/powerpc/platforms/Kconfig | 1 +
arch/powerpc/platforms/Makefile | 1 +
arch/powerpc/platforms/book3s/Kconfig | 15 +
arch/powerpc/platforms/book3s/Makefile | 2 +
arch/powerpc/platforms/book3s/vas-api.c | 473 +++++++++++++
arch/powerpc/platforms/powernv/Kconfig | 14 -
arch/powerpc/platforms/powernv/Makefile | 2 +-
arch/powerpc/platforms/powernv/vas-api.c | 278 --------
arch/powerpc/platforms/powernv/vas-debug.c | 27 +-
arch/powerpc/platforms/powernv/vas-fault.c | 173 +----
arch/powerpc/platforms/powernv/vas-trace.h | 4 +-
arch/powerpc/platforms/powernv/vas-window.c | 264 ++++----
arch/powerpc/platforms/powernv/vas.h | 48 +-
arch/powerpc/platforms/pseries/Makefile | 1 +
arch/powerpc/platforms/pseries/vas.c | 636 ++++++++++++++++++
arch/powerpc/platforms/pseries/vas.h | 125 ++++
drivers/crypto/nx/Kconfig | 1 +
drivers/crypto/nx/Makefile | 2 +-
drivers/crypto/nx/nx-common-powernv.c | 6 +-
.../{nx-842-pseries.c => nx-common-pseries.c} | 138 ++++
23 files changed, 1717 insertions(+), 616 deletions(-)
create mode 100644 arch/powerpc/platforms/book3s/Kconfig
create mode 100644 arch/powerpc/platforms/book3s/Makefile
create mode 100644 arch/powerpc/platforms/book3s/vas-api.c
delete mode 100644 arch/powerpc/platforms/powernv/vas-api.c
create mode 100644 arch/powerpc/platforms/pseries/vas.c
create mode 100644 arch/powerpc/platforms/pseries/vas.h
rename drivers/crypto/nx/{nx-842-pseries.c => nx-common-pseries.c} (90%)

--
2.18.2