* [PATCH v4 0/4] Error recovery for zPCI passthrough devices
@ 2026-08-31 18:31 Farhan Ali
2026-08-31 18:31 ` [PATCH v4 1/4] linux-headers: Update Linux header to 7.3-rc1 Farhan Ali
` (4 more replies)
0 siblings, 5 replies; 21+ messages in thread
From: Farhan Ali @ 2026-08-31 18:31 UTC (permalink / raw)
To: qemu-devel, qemu-s390x; +Cc: mjrosato, alifm, farman, cohuck, alex, clg, armbru
Hi,
This patch series introduces support for error recovery for passthrough PCI
devices on System Z (s390x). This is the user space component for the Linux
kernel patches [1]. The kernel patches were merged for 7.3 and it had some
significant changes in VFIO API since the last QEMU series was posted. But the
design for QEMU component is still the same.
For QEMU on eventfd notification for PCI error from vfio-pci driver we call the
vfio error handler. We can use a per device error handler callback to override
the default vfio error handler. For s390x specific error handler, we retrieve
the architecture specific PCI error information and inject the information into
the guest. Once the guest receives the error information, the guest drivers will
drive the error recovery. Typically recovery involves a device reset which
translate to CLP disable/enable cycle for the device.
I would appreciate some feedback on this patch series.
Thanks
Farhan
[1] https://lore.kernel.org/all/20260818164326.387bb27b@shazbot.org/
ChangeLog
---------
v3 https://lore.kernel.org/qemu-devel/20250925174852.1302-1-alifm@linux.ibm.com/
v3 -> v4
- Include linux headers for 7.3-rc1.
- Rework VFIO API changes based on the kernel API (patch 3).
- Address Markus's comments from v3 (patch 2).
v2 https://lore.kernel.org/qemu-devel/20250825212434.2255-1-alifm@linux.ibm.com/
v2 -> v3
- Update arch_err_handler to err_handler and include Error ** in
function definition. (patch 2)
- Introduce helper function to hide the internal indirection of device_feature()
(patch 3)
- Update function definitions to include Error ** (patch 4)
v1 https://lore.kernel.org/qemu-devel/20250813174152.1238-1-alifm@linux.ibm.com/
v1 -> v2
- Use VFIO_DEVICE_FEATURE ioctl to get device error information.
(Based on Alex's feedback on kernel series)
Farhan Ali (4):
linux-headers: Update Linux header to 7.3-rc1
vfio/pci: Add an error handler callback
s390x/pci: Add PCI error handling for vfio pci devices
s390x/pci: Reset a device in error state
hw/s390x/s390-pci-bus.c | 16 ++
hw/s390x/s390-pci-vfio-stubs.c | 10 +
hw/s390x/s390-pci-vfio.c | 110 +++++++++++
hw/vfio/pci.c | 27 ++-
hw/vfio/pci.h | 1 +
include/hw/s390x/s390-pci-bus.h | 2 +
include/hw/s390x/s390-pci-vfio.h | 2 +
include/standard-headers/drm/drm_fourcc.h | 209 ++++++++++++++++++--
include/standard-headers/linux/ethtool.h | 6 +
include/standard-headers/linux/fuse.h | 63 +++++-
include/standard-headers/linux/virtio_ids.h | 1 +
linux-headers/asm-arm64/kvm.h | 1 +
linux-headers/asm-riscv/kvm.h | 13 ++
linux-headers/linux/iommufd.h | 45 ++++-
linux-headers/linux/kvm.h | 1 +
linux-headers/linux/userfaultfd.h | 54 ++++-
linux-headers/linux/vduse.h | 28 +++
linux-headers/linux/vfio.h | 21 ++
linux-headers/linux/vfio_zdev.h | 2 +
19 files changed, 579 insertions(+), 33 deletions(-)
--
2.43.0
^ permalink raw reply [flat|nested] 21+ messages in thread* [PATCH v4 1/4] linux-headers: Update Linux header to 7.3-rc1 2026-08-31 18:31 [PATCH v4 0/4] Error recovery for zPCI passthrough devices Farhan Ali @ 2026-08-31 18:31 ` Farhan Ali 2026-08-31 18:31 ` [PATCH v4 2/4] vfio/pci: Add an error handler callback Farhan Ali ` (3 subsequent siblings) 4 siblings, 0 replies; 21+ messages in thread From: Farhan Ali @ 2026-08-31 18:31 UTC (permalink / raw) To: qemu-devel, qemu-s390x; +Cc: mjrosato, alifm, farman, cohuck, alex, clg, armbru Update the headers to include VFIO and zPCI VFIO changes. Signed-off-by: Farhan Ali <alifm@linux.ibm.com> --- include/standard-headers/drm/drm_fourcc.h | 209 ++++++++++++++++++-- include/standard-headers/linux/ethtool.h | 6 + include/standard-headers/linux/fuse.h | 63 +++++- include/standard-headers/linux/virtio_ids.h | 1 + linux-headers/asm-arm64/kvm.h | 1 + linux-headers/asm-riscv/kvm.h | 13 ++ linux-headers/linux/iommufd.h | 45 ++++- linux-headers/linux/kvm.h | 1 + linux-headers/linux/userfaultfd.h | 54 ++++- linux-headers/linux/vduse.h | 28 +++ linux-headers/linux/vfio.h | 21 ++ linux-headers/linux/vfio_zdev.h | 2 + 12 files changed, 417 insertions(+), 27 deletions(-) diff --git a/include/standard-headers/drm/drm_fourcc.h b/include/standard-headers/drm/drm_fourcc.h index 2ba0b5c925..ca4f4e3d8c 100644 --- a/include/standard-headers/drm/drm_fourcc.h +++ b/include/standard-headers/drm/drm_fourcc.h @@ -1645,36 +1645,69 @@ drm_fourcc_canonicalize_nvidia_format_mod(uint64_t modifier) * For multi-plane formats the above surfaces get merged into one plane for * each format plane, based on the required alignment only. * - * Bits Parameter Notes - * ----- ------------------------ --------------------------------------------- + * Bits Parameter Notes + * ------- ------------------------ --------------------------------------------- + * + * DRM format modifier fields on AMD GPUs: + * 7:0 TILE_VERSION Values are AMD_FMT_MOD_TILE_VER_* + * 12:8 TILE Values are AMD_FMT_MOD_TILE_<version>_* + * 13 DCC Delta Color Compression, supported on GFX8 and newer + * 55:14 (chip specific) See below for details, depends on GFX block version + * 63:56 Vendor Value is DRM_FORMAT_MOD_VENDOR_AMD + * + * Chip specific fields on Gfx9 and newer: + * 14 DCC_RETILE + * 15 DCC_PIPE_ALIGN + * 16 DCC_INDEPENDENT_64B + * 17 DCC_INDEPENDENT_128B + * 19:18 DCC_MAX_COMPRESSED_BLOCK Values are AMD_FMT_MOD_DCC_BLOCK_* + * 20 DCC_CONSTANT_ENCODE + * 23:21 PIPE_XOR_BITS Only for some chips + * 26:24 BANK_XOR_BITS Only for some chips + * 29:27 PACKERS Only for some chips + * 32:30 RB Only for some chips + * 35:33 PIPE Only for some chips + * 55:36 - Reserved for future use, must be zero + * + * Chip specific fields on Gfx6-8: + * 16:14 MICROTILE Micro tile format + * 21:17 PIPE_CONFIG Number of pipes and how pipes are interleaved + * 24:22 TILE_SPLIT Tile split size + * 26:25 BANK_WIDTH Number of tiles in the X direction in the same bank + * 28:27 BANK_HEIGHT Number of tiles in the Y direction in the same bank + * 30:29 MACRO_TILE_ASPECT Macro tile aspect ratio + * 32:31 NUM_BANKS Number of banks + * 55:33 - Reserved for future use, must be zero * - * 7:0 TILE_VERSION Values are AMD_FMT_MOD_TILE_VER_* - * 12:8 TILE Values are AMD_FMT_MOD_TILE_<version>_* - * 13 DCC - * 14 DCC_RETILE - * 15 DCC_PIPE_ALIGN - * 16 DCC_INDEPENDENT_64B - * 17 DCC_INDEPENDENT_128B - * 19:18 DCC_MAX_COMPRESSED_BLOCK Values are AMD_FMT_MOD_DCC_BLOCK_* - * 20 DCC_CONSTANT_ENCODE - * 23:21 PIPE_XOR_BITS Only for some chips - * 26:24 BANK_XOR_BITS Only for some chips - * 29:27 PACKERS Only for some chips - * 32:30 RB Only for some chips - * 35:33 PIPE Only for some chips - * 55:36 - Reserved for future use, must be zero */ #define AMD_FMT_MOD fourcc_mod_code(AMD, 0) #define IS_AMD_FMT_MOD(val) (((val) >> 56) == DRM_FORMAT_MOD_VENDOR_AMD) -/* Reserve 0 for GFX8 and older */ +#define AMD_FMT_MOD_TILE_VER_GFX6 0 #define AMD_FMT_MOD_TILE_VER_GFX9 1 #define AMD_FMT_MOD_TILE_VER_GFX10 2 #define AMD_FMT_MOD_TILE_VER_GFX10_RBPLUS 3 #define AMD_FMT_MOD_TILE_VER_GFX11 4 #define AMD_FMT_MOD_TILE_VER_GFX12 5 +/* + * Gfx6-8 tiling modes. + * A complete reference implementation is found in addrlib in the Mesa code base. + * + * - Microtiled modes (1D): + * Pixel data is organized into micro tiles of 8x8 pixels. + * + * - Macrotiled modes (2D): + * Micro tiles are further organized into macro tiles. + * These are optimized for even load distribution among memory channels. + * + * Note that only THIN1 modes are exposed here. + * THICK and XTHICK are for 3D images and not relevant to DRM format modifiers. + */ +#define AMD_FMT_MOD_TILE_GFX6_1D_TILED_THIN1 0x2 +#define AMD_FMT_MOD_TILE_GFX6_2D_TILED_THIN1 0x4 + /* * 64K_S is the same for GFX9/GFX10/GFX10_RBPLUS and hence has GFX9 as canonical * version. @@ -1773,6 +1806,146 @@ drm_fourcc_canonicalize_nvidia_format_mod(uint64_t modifier) #define AMD_FMT_MOD_PIPE_SHIFT 33 #define AMD_FMT_MOD_PIPE_MASK 0x7 +/* + * MICRO_TILE_MODE, 3 bits. Determines the micro tile format. + * Only relevant to Gfx6-8. + * + * DISPLAY - Displayable tiling + * THIN - Non-displayable tiling, a.k.a thin micro tiling + * DEPTH, THICK - not exposed, not relevant to DRM format modifier use cases + * ROTATED - not exposed, not implemented in Linux or Mesa + */ +#define AMD_FMT_MOD_MICROTILE_SHIFT 14ULL +#define AMD_FMT_MOD_MICROTILE_MASK 0x7 + +#define AMD_FMT_MOD_MICROTILE_DISPLAY 0x0 +#define AMD_FMT_MOD_MICROTILE_THIN 0x1 + +/* + * PIPE_CONFIG, 5 bits. Number of pipes and how pipes are interleaved on the surface, + * which means the shader engine tile size and packer tile size. + * Typically matches the number of memory channels, or number of RBs. + * Only relevant to Gfx6-8 macro tiled modes. + * + * P<n>_<a>x<b>_<c>x<d> + * where: + * <n> - number of pipes + * <a>x<b> - shader engine tile size + * <c>x<d> - packer tile size + */ +#define AMD_FMT_MOD_PIPE_CONFIG_SHIFT 17ULL +#define AMD_FMT_MOD_PIPE_CONFIG_MASK 0x1f + +#define AMD_FMT_MOD_PIPE_CONFIG_P2 0x0 +#define AMD_FMT_MOD_PIPE_CONFIG_P4_8x16 0x4 +#define AMD_FMT_MOD_PIPE_CONFIG_P4_16x16 0x5 +#define AMD_FMT_MOD_PIPE_CONFIG_P4_16x32 0x6 +#define AMD_FMT_MOD_PIPE_CONFIG_P4_32x32 0x7 +#define AMD_FMT_MOD_PIPE_CONFIG_P8_16x16_8x16 0x8 +#define AMD_FMT_MOD_PIPE_CONFIG_P8_16x32_8x16 0x9 +#define AMD_FMT_MOD_PIPE_CONFIG_P8_32x32_8x16 0xa +#define AMD_FMT_MOD_PIPE_CONFIG_P8_16x32_16x16 0xb +#define AMD_FMT_MOD_PIPE_CONFIG_P8_32x32_16x16 0xc +#define AMD_FMT_MOD_PIPE_CONFIG_P8_32x32_16x32 0xd +#define AMD_FMT_MOD_PIPE_CONFIG_P8_32x64_32x32 0xe +#define AMD_FMT_MOD_PIPE_CONFIG_P16_32x32_8x16 0x10 +#define AMD_FMT_MOD_PIPE_CONFIG_P16_32x32_16x16 0x11 + +/* + * TILE_SPLIT, 3 bits. + * Only relevant to Gfx6-8 macro tiled modes. + * + * On GFX6 (or with depth tiling modes on GFX7 and newer), + * the GFX block uses the GB_TILE_MODE.TILE_SPLIT field directly. + * + * On GFX7 and newer with non-depth tiling modes, the GFX block uses a + * split factor which is stored in the GB_TILE_MODE.SAMPLE_SPLIT field. + * SAMPLE_SPLIT may be: 0 - 1 byte; 1 - 2 bytes; 2 - 4 bytes; 3 - 8 bytes. + * The actual tile size and tile split bytes are calculated as follows: + * + * bpp = ... <- bits per pixel in the current image + * thickness = ... <- depends on array mode; may be: 1, 4, 8 + * num_samples = ... <- number of samples in the current image + * tile_size_pixels = 8 * 8 + * tile_bytes_1x = thickness * tile_size_pixels * bpp / 8 + * sample_split_factor = 1 << SAMPLE_SPLIT + * tile_split_bytes = clamp(tile_bytes_1x * sample_split_factor, 256, dram_row_size_bytes) + * tile_bytes = clamp(tile_bytes_1x * num_samples, 64, tile_split_bytes) + * + * In both cases, the display block (DCE) has no SAMPLE_SPLIT + * and just needs the tile split bytes in the GRPH_CONTROL.GRPH_TILE_SPLIT field. + * To maximize compatibility between GFX6-7, we don't include the SAMPLE_SPLIT + * in the format modifiers. + * + * The actual tile split in bytes is: 64 << field value + * Possible values of this field: + * + * 0 - Tile split is 64 bytes + * 1 - Tile split is 128 bytes + * 2 - Tile split is 256 bytes + * 3 - Tile split is 512 bytes + * 4 - Tile split is 1 KiB + * 5 - Tile split is 2 KiB + * 6 - Tile split is 4 KiB + */ +#define AMD_FMT_MOD_TILE_SPLIT_SHIFT 22ULL +#define AMD_FMT_MOD_TILE_SPLIT_MASK 0x7 + +/* + * BANK_WIDTH, 2 bits. Number of tiles in the X direction in the same bank. + * Only relevant to Gfx6-8 macro tiled modes. + * The actual bank width is: 1 << field value + * Possible values: + * + * 0 - bank width is 1 + * 1 - bank width is 2 + * 2 - bank width is 4 + * 3 - bank width is 8 + */ +#define AMD_FMT_MOD_BANK_WIDTH_SHIFT 25ULL +#define AMD_FMT_MOD_BANK_WIDTH_MASK 0x3 + +/* + * BANK_HEIGHT, 2 bits. Number of tiles in the Y direction in the same bank. + * Only relevant to Gfx6-8 macro tiled modes. + * The actual bank height is: 1 << field value + * Possible values: + * + * 0 - bank height is 1 + * 1 - bank height is 2 + * 2 - bank height is 4 + * 3 - bank height is 8 + */ +#define AMD_FMT_MOD_BANK_HEIGHT_SHIFT 27ULL +#define AMD_FMT_MOD_BANK_HEIGHT_MASK 0x3 + +/* + * MACRO_TILE_ASPECT, 2 bits. Macro tile aspect ratio. + * Only relevant to Gfx6-8 macro tiled modes. + * Possible values: + * + * 0 - aspect ratio is 1:1 + * 1 - aspect ratio is 4:1 + * 2 - aspect ratio is 16:1 + * 3 - aspect ratio is 64:1 + */ +#define AMD_FMT_MOD_MACRO_TILE_ASPECT_SHIFT 29ULL +#define AMD_FMT_MOD_MACRO_TILE_ASPECT_MASK 0x3 + +/* + * NUM_BANKS, 2 bits. Number of banks. + * Only relevant to Gfx6-8 macro tiled modes. + * The actual number of banks is: 2 << field value + * Possible values: + * + * 0 - number of banks is 2 + * 1 - number of banks is 4 + * 2 - number of banks is 8 + * 3 - number of banks is 16 + */ +#define AMD_FMT_MOD_NUM_BANKS_SHIFT 31ULL +#define AMD_FMT_MOD_NUM_BANKS_MASK 0x3 + #define AMD_FMT_MOD_SET(field, value) \ ((uint64_t)(value) << AMD_FMT_MOD_##field##_SHIFT) #define AMD_FMT_MOD_GET(field, value) \ diff --git a/include/standard-headers/linux/ethtool.h b/include/standard-headers/linux/ethtool.h index 55f608c3a7..622d634d04 100644 --- a/include/standard-headers/linux/ethtool.h +++ b/include/standard-headers/linux/ethtool.h @@ -2015,7 +2015,13 @@ enum ethtool_link_mode_bit_indices { ETHTOOL_LINK_MODE_100000baseLR4_ER4_Full_BIT = 39, ETHTOOL_LINK_MODE_50000baseSR2_Full_BIT = 40, ETHTOOL_LINK_MODE_1000baseX_Full_BIT = 41, + + /* Despite the "baseCR" in 10000baseCR, this is not an IEEE 802.3 baseCR + * It represents SFF-8431 Appendix-E SFP+ Direct Attach (10G-SFI-DA). + * The name is kept as-is for uAPI backward compatibility. + */ ETHTOOL_LINK_MODE_10000baseCR_Full_BIT = 42, + ETHTOOL_LINK_MODE_10000baseSR_Full_BIT = 43, ETHTOOL_LINK_MODE_10000baseLR_Full_BIT = 44, ETHTOOL_LINK_MODE_10000baseLRM_Full_BIT = 45, diff --git a/include/standard-headers/linux/fuse.h b/include/standard-headers/linux/fuse.h index abf3a78858..cee8342d4e 100644 --- a/include/standard-headers/linux/fuse.h +++ b/include/standard-headers/linux/fuse.h @@ -240,6 +240,14 @@ * - add FUSE_COPY_FILE_RANGE_64 * - add struct fuse_copy_file_range_out * - add FUSE_NOTIFY_PRUNE + * + * 7.46 + * - add FUSE_IO_URING_CMD_ADD_QUEUE + * - add FUSE_HAS_IO_URING_BUFPOOL + * - add fuse_uring_cmd_req bufpool struct + * - add bufpool offset field to fuse_uring_ent_in_out struct + * - add FUSE_URING_ZERO_COPY, FUSE_URING_ENT_ZERO_COPY, and + * FOPEN_IO_URING_ZERO_COPY flag */ #ifndef _LINUX_FUSE_H @@ -271,7 +279,7 @@ #define FUSE_KERNEL_VERSION 7 /** Minor version number of this interface */ -#define FUSE_KERNEL_MINOR_VERSION 45 +#define FUSE_KERNEL_MINOR_VERSION 46 /** The node ID of the root inode */ #define FUSE_ROOT_ID 1 @@ -379,6 +387,12 @@ struct fuse_file_lock { * FOPEN_NOFLUSH: don't flush data cache on close (unless FUSE_WRITEBACK_CACHE) * FOPEN_PARALLEL_DIRECT_WRITES: Allow concurrent direct writes on the same inode * FOPEN_PASSTHROUGH: passthrough read/write io for this open file + * FOPEN_IO_URING_ZERO_COPY: use io-uring zero-copy for reads/writes on this + * open file. Honored only when the serving io-uring + * queue was set up for zero-copy + * (FUSE_URING_ZERO_COPY) and the request carries page + * payload. Otherwise reads/writes fall back to + * copying. */ #define FOPEN_DIRECT_IO (1 << 0) #define FOPEN_KEEP_CACHE (1 << 1) @@ -388,6 +402,7 @@ struct fuse_file_lock { #define FOPEN_NOFLUSH (1 << 5) #define FOPEN_PARALLEL_DIRECT_WRITES (1 << 6) #define FOPEN_PASSTHROUGH (1 << 7) +#define FOPEN_IO_URING_ZERO_COPY (1 << 8) /** * INIT request/reply flags @@ -444,6 +459,7 @@ struct fuse_file_lock { * FUSE_OVER_IO_URING: Indicate that client supports io-uring * FUSE_REQUEST_TIMEOUT: kernel supports timing out requests. * init_out.request_timeout contains the timeout (in secs) + * FUSE_HAS_IO_URING_BUFPOOL: kernel supports io-uring buffer pools */ #define FUSE_ASYNC_READ (1 << 0) #define FUSE_POSIX_LOCKS (1 << 1) @@ -491,6 +507,7 @@ struct fuse_file_lock { #define FUSE_ALLOW_IDMAP (1ULL << 40) #define FUSE_OVER_IO_URING (1ULL << 41) #define FUSE_REQUEST_TIMEOUT (1ULL << 42) +#define FUSE_HAS_IO_URING_BUFPOOL (1ULL << 43) /** * CUSE INIT request/reply flags @@ -1247,6 +1264,13 @@ struct fuse_supp_groups { #define FUSE_URING_IN_OUT_HEADER_SZ 128 #define FUSE_URING_OP_IN_OUT_SZ 128 +/** + * fuse_uring_ent_in_out flags + * + * FUSE_URING_ENT_ZERO_COPY: Set if the ent's payload is zero-copied + */ +#define FUSE_URING_ENT_ZERO_COPY (1 << 0) + /* Used as part of the fuse_uring_req_header */ struct fuse_uring_ent_in_out { uint64_t flags; @@ -1259,7 +1283,9 @@ struct fuse_uring_ent_in_out { /* size of user payload buffer */ uint32_t payload_sz; - uint32_t padding; + + /* Offset into the bufpool, if bufpools are used */ + uint32_t offset; uint64_t reserved; }; @@ -1288,8 +1314,22 @@ enum fuse_uring_cmd { /* commit fuse request result and fetch next request */ FUSE_IO_URING_CMD_COMMIT_AND_FETCH = 2, + + /* add a queue */ + FUSE_IO_URING_CMD_ADD_QUEUE = 3, + + /* add a bufpool to a queue */ + FUSE_IO_URING_CMD_ADD_BUFPOOL = 4, }; +/* + * fuse_uring_cmd_req flags for FUSE_IO_URING_CMD_ADD_QUEUE + * + * FUSE_URING_ZERO_COPY is only supported for queues with bufpools on privileged + * servers + */ +#define FUSE_URING_ZERO_COPY (1 << 0) + /** * In the 80B command area of the SQE. */ @@ -1302,6 +1342,25 @@ struct fuse_uring_cmd_req { /* queue the command is for (queue index) */ uint16_t qid; uint8_t padding[6]; + + union { + struct { + /* base address of bufpool */ + uint64_t uaddr; + uint32_t len; + uint32_t reserved; + } bufpool; + + /* + * Index of this entry's slot in the server's io_uring + * registered buffer table, where the kernel registers the + * request's pages for zero-copy. Set for + * FUSE_IO_URING_CMD_REGISTER cmds only, and only on queues + * created with FUSE_URING_ZERO_COPY. On a non-zero-copy queue + * this must be 0 + */ + uint16_t ent_zero_copy_buf_index; + }; }; #endif /* _LINUX_FUSE_H */ diff --git a/include/standard-headers/linux/virtio_ids.h b/include/standard-headers/linux/virtio_ids.h index 6c12db16fa..f9056af0c6 100644 --- a/include/standard-headers/linux/virtio_ids.h +++ b/include/standard-headers/linux/virtio_ids.h @@ -69,6 +69,7 @@ #define VIRTIO_ID_BT 40 /* virtio bluetooth */ #define VIRTIO_ID_GPIO 41 /* virtio gpio */ #define VIRTIO_ID_SPI 45 /* virtio spi */ +#define VIRTIO_ID_MEDIA 48 /* virtio media */ /* * Virtio Transitional IDs diff --git a/linux-headers/asm-arm64/kvm.h b/linux-headers/asm-arm64/kvm.h index 6aefe79738..780cc7f729 100644 --- a/linux-headers/asm-arm64/kvm.h +++ b/linux-headers/asm-arm64/kvm.h @@ -106,6 +106,7 @@ struct kvm_regs { #define KVM_ARM_VCPU_PTRAUTH_GENERIC 6 /* VCPU uses generic authentication */ #define KVM_ARM_VCPU_HAS_EL2 7 /* Support nested virtualization */ #define KVM_ARM_VCPU_HAS_EL2_E2H0 8 /* Limit NV support to E2H RES0 */ +#define KVM_ARM_VCPU_PMU_V3_STRICT 9 /* No default PMU creation */ struct kvm_vcpu_init { __u32 target; diff --git a/linux-headers/asm-riscv/kvm.h b/linux-headers/asm-riscv/kvm.h index 504e733053..20d9959ca4 100644 --- a/linux-headers/asm-riscv/kvm.h +++ b/linux-headers/asm-riscv/kvm.h @@ -102,6 +102,11 @@ struct kvm_riscv_smstateen_csr { unsigned long sstateen0; }; +/* Zicfiss CSR for KVM_GET_ONE_REG and KVM_SET_ONE_REG */ +struct kvm_riscv_zicfiss_csr { + unsigned long ssp; +}; + /* TIMER registers for KVM_GET_ONE_REG and KVM_SET_ONE_REG */ struct kvm_riscv_timer { __u64 frequency; @@ -199,6 +204,8 @@ enum KVM_RISCV_ISA_EXT_ID { KVM_RISCV_ISA_EXT_ZCLSD, KVM_RISCV_ISA_EXT_ZILSD, KVM_RISCV_ISA_EXT_ZALASR, + KVM_RISCV_ISA_EXT_ZICFILP, + KVM_RISCV_ISA_EXT_ZICFISS, KVM_RISCV_ISA_EXT_MAX, }; @@ -240,6 +247,9 @@ struct kvm_riscv_sbi_fwft_feature { struct kvm_riscv_sbi_fwft { struct kvm_riscv_sbi_fwft_feature misaligned_deleg; struct kvm_riscv_sbi_fwft_feature pointer_masking; + struct kvm_riscv_sbi_fwft_feature pte_ad_hw_updating; + struct kvm_riscv_sbi_fwft_feature landing_pad; + struct kvm_riscv_sbi_fwft_feature shadow_stack; }; /* If you need to interpret the index values, here is the key: */ @@ -263,12 +273,15 @@ struct kvm_riscv_sbi_fwft { #define KVM_REG_RISCV_CSR_GENERAL (0x0 << KVM_REG_RISCV_SUBTYPE_SHIFT) #define KVM_REG_RISCV_CSR_AIA (0x1 << KVM_REG_RISCV_SUBTYPE_SHIFT) #define KVM_REG_RISCV_CSR_SMSTATEEN (0x2 << KVM_REG_RISCV_SUBTYPE_SHIFT) +#define KVM_REG_RISCV_CSR_ZICFISS (0x3 << KVM_REG_RISCV_SUBTYPE_SHIFT) #define KVM_REG_RISCV_CSR_REG(name) \ (offsetof(struct kvm_riscv_csr, name) / sizeof(unsigned long)) #define KVM_REG_RISCV_CSR_AIA_REG(name) \ (offsetof(struct kvm_riscv_aia_csr, name) / sizeof(unsigned long)) #define KVM_REG_RISCV_CSR_SMSTATEEN_REG(name) \ (offsetof(struct kvm_riscv_smstateen_csr, name) / sizeof(unsigned long)) +#define KVM_REG_RISCV_CSR_ZICFISS_REG(name) \ + (offsetof(struct kvm_riscv_zicfiss_csr, name) / sizeof(unsigned long)) /* Timer registers are mapped as type 4 */ #define KVM_REG_RISCV_TIMER (0x04 << KVM_REG_RISCV_TYPE_SHIFT) diff --git a/linux-headers/linux/iommufd.h b/linux-headers/linux/iommufd.h index 10c600d71e..3c61caadbf 100644 --- a/linux-headers/linux/iommufd.h +++ b/linux-headers/linux/iommufd.h @@ -57,6 +57,7 @@ enum { IOMMUFD_CMD_IOAS_CHANGE_PROCESS = 0x92, IOMMUFD_CMD_VEVENTQ_ALLOC = 0x93, IOMMUFD_CMD_HW_QUEUE_ALLOC = 0x94, + IOMMUFD_CMD_IOAS_NOIOMMU_GET_PA = 0x95, }; /** @@ -219,6 +220,33 @@ struct iommu_ioas_map { }; #define IOMMU_IOAS_MAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_MAP) +/** + * struct iommu_ioas_noiommu_get_pa - ioctl(IOMMU_IOAS_NOIOMMU_GET_PA) + * @size: sizeof(struct iommu_ioas_noiommu_get_pa) + * @flags: Reserved, must be 0 for now + * @ioas_id: IOAS ID to query IOVA to PA mapping from + * @__reserved: Must be 0 + * @iova: IOVA to query + * @length: On input, non-zero maximum number of bytes to query starting from + * @iova. On output, number of physically contiguous bytes starting + * from @out_phys, capped by the input length. + * @out_phys: Output physical address the IOVA maps to + * + * Query the physical address backing an IOVA range. The beginning of the + * range must be mapped already and length must be non-zero. For noiommu + * devices doing unsafe DMA only. + */ +struct iommu_ioas_noiommu_get_pa { + __u32 size; + __u32 flags; + __u32 ioas_id; + __u32 __reserved; + __aligned_u64 iova; + __aligned_u64 length; + __aligned_u64 out_phys; +}; +#define IOMMU_IOAS_NOIOMMU_GET_PA _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_NOIOMMU_GET_PA) + /** * struct iommu_ioas_map_file - ioctl(IOMMU_IOAS_MAP_FILE) * @size: sizeof(struct iommu_ioas_map_file) @@ -574,11 +602,22 @@ struct iommu_hw_info_vtd { __aligned_u64 ecap_reg; }; +/** + * enum iommu_hw_info_arm_smmuv3_flags - Flags for ARM SMMUv3 hw_info + * @IOMMU_HW_INFO_ARM_SMMUV3_ERRATA_REPEAT_TLBI_CFGI: + * If set, user space must issue TLBI/CFGI+SYNC commands twice due to + * hardware erratum T264-SMMU-3. See the description at + * arm_smmu_erratum_repeat_tlbi_cfgi_key. + */ +enum iommu_hw_info_arm_smmuv3_flags { + IOMMU_HW_INFO_ARM_SMMUV3_ERRATA_REPEAT_TLBI_CFGI = 1 << 0, +}; + /** * struct iommu_hw_info_arm_smmuv3 - ARM SMMUv3 hardware information * (IOMMU_HW_INFO_TYPE_ARM_SMMUV3) * - * @flags: Must be set to 0 + * @flags: Combination of enum iommu_hw_info_arm_smmuv3_flags * @__reserved: Must be 0 * @idr: Implemented features for ARM SMMU Non-secure programming interface * @iidr: Information about the implementation and implementer of ARM SMMU, @@ -594,7 +633,7 @@ struct iommu_hw_info_vtd { * idr[0]: ST_LEVEL, TERM_MODEL, STALL_MODEL, TTENDIAN , CD2L, ASID16, TTF * idr[1]: SIDSIZE, SSIDSIZE * idr[3]: BBML, RIL - * idr[5]: VAX, GRAN64K, GRAN16K, GRAN4K + * idr[5]: VAX, GRAN64K, GRAN16K, GRAN4K, DS * * - S1P should be assumed to be true if a NESTED HWPT can be created * - VFIO/iommufd only support platforms with COHACC, it should be assumed to be @@ -602,7 +641,7 @@ struct iommu_hw_info_vtd { * - ATS is a per-device property. If the VMM describes any devices as ATS * capable in ACPI/DT it should set the corresponding idr. * - * This list may expand in future (eg E0PD, AIE, PBHA, D128, DS etc). It is + * This list may expand in future (eg E0PD, AIE, PBHA, D128 etc). It is * important that VMMs do not read bits outside the list to allow for * compatibility with future kernels. Several features in the SMMUv3 * architecture are not currently supported by the kernel for nesting: HTTU, diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h index aea4ab5c69..f21c47bf8d 100644 --- a/linux-headers/linux/kvm.h +++ b/linux-headers/linux/kvm.h @@ -987,6 +987,7 @@ struct kvm_enable_cap { #define KVM_CAP_S390_VSIE_ESAMODE 248 #define KVM_CAP_S390_HPAGE_2G 249 #define KVM_CAP_PPC_COMPAT_CAPS 250 +#define KVM_CAP_ARM_PMU_V3_STRICT 251 struct kvm_irq_routing_irqchip { __u32 irqchip; diff --git a/linux-headers/linux/userfaultfd.h b/linux-headers/linux/userfaultfd.h index 4283de22d5..c8e99df952 100644 --- a/linux-headers/linux/userfaultfd.h +++ b/linux-headers/linux/userfaultfd.h @@ -25,7 +25,8 @@ #define UFFD_API ((__u64)0xAA) #define UFFD_API_REGISTER_MODES (UFFDIO_REGISTER_MODE_MISSING | \ UFFDIO_REGISTER_MODE_WP | \ - UFFDIO_REGISTER_MODE_MINOR) + UFFDIO_REGISTER_MODE_MINOR | \ + UFFDIO_REGISTER_MODE_RWP) #define UFFD_API_FEATURES (UFFD_FEATURE_PAGEFAULT_FLAG_WP | \ UFFD_FEATURE_EVENT_FORK | \ UFFD_FEATURE_EVENT_REMAP | \ @@ -42,10 +43,13 @@ UFFD_FEATURE_WP_UNPOPULATED | \ UFFD_FEATURE_POISON | \ UFFD_FEATURE_WP_ASYNC | \ - UFFD_FEATURE_MOVE) + UFFD_FEATURE_MOVE | \ + UFFD_FEATURE_RWP | \ + UFFD_FEATURE_RWP_ASYNC) #define UFFD_API_IOCTLS \ ((__u64)1 << _UFFDIO_REGISTER | \ (__u64)1 << _UFFDIO_UNREGISTER | \ + (__u64)1 << _UFFDIO_SET_MODE | \ (__u64)1 << _UFFDIO_API) #define UFFD_API_RANGE_IOCTLS \ ((__u64)1 << _UFFDIO_WAKE | \ @@ -54,13 +58,15 @@ (__u64)1 << _UFFDIO_MOVE | \ (__u64)1 << _UFFDIO_WRITEPROTECT | \ (__u64)1 << _UFFDIO_CONTINUE | \ - (__u64)1 << _UFFDIO_POISON) + (__u64)1 << _UFFDIO_POISON | \ + (__u64)1 << _UFFDIO_RWPROTECT) #define UFFD_API_RANGE_IOCTLS_BASIC \ ((__u64)1 << _UFFDIO_WAKE | \ (__u64)1 << _UFFDIO_COPY | \ (__u64)1 << _UFFDIO_WRITEPROTECT | \ (__u64)1 << _UFFDIO_CONTINUE | \ - (__u64)1 << _UFFDIO_POISON) + (__u64)1 << _UFFDIO_POISON | \ + (__u64)1 << _UFFDIO_RWPROTECT) /* * Valid ioctl command number range with this API is from 0x00 to @@ -79,6 +85,8 @@ #define _UFFDIO_WRITEPROTECT (0x06) #define _UFFDIO_CONTINUE (0x07) #define _UFFDIO_POISON (0x08) +#define _UFFDIO_RWPROTECT (0x09) +#define _UFFDIO_SET_MODE (0x0A) #define _UFFDIO_API (0x3F) /* userfaultfd ioctl ids */ @@ -103,6 +111,10 @@ struct uffdio_continue) #define UFFDIO_POISON _IOWR(UFFDIO, _UFFDIO_POISON, \ struct uffdio_poison) +#define UFFDIO_RWPROTECT _IOWR(UFFDIO, _UFFDIO_RWPROTECT, \ + struct uffdio_rwprotect) +#define UFFDIO_SET_MODE _IOW(UFFDIO, _UFFDIO_SET_MODE, \ + struct uffdio_set_mode) /* read() structure */ struct uffd_msg { @@ -158,6 +170,7 @@ struct uffd_msg { #define UFFD_PAGEFAULT_FLAG_WRITE (1<<0) /* If this was a write fault */ #define UFFD_PAGEFAULT_FLAG_WP (1<<1) /* If reason is VM_UFFD_WP */ #define UFFD_PAGEFAULT_FLAG_MINOR (1<<2) /* If reason is VM_UFFD_MINOR */ +#define UFFD_PAGEFAULT_FLAG_RWP (1<<3) /* If reason is VM_UFFD_RWP */ struct uffdio_api { /* userland asks for an API number and the features to enable */ @@ -230,6 +243,18 @@ struct uffdio_api { * * UFFD_FEATURE_MOVE indicates that the kernel supports moving an * existing page contents from userspace. + * + * UFFD_FEATURE_RWP indicates that the kernel supports + * UFFDIO_REGISTER_MODE_RWP for read-write protection tracking. + * Pages are made inaccessible via UFFDIO_RWPROTECT and faults + * are delivered when the pages are re-accessed. + * + * UFFD_FEATURE_RWP_ASYNC indicates asynchronous mode for + * UFFDIO_REGISTER_MODE_RWP. When set, faults on read-write + * protected pages are auto-resolved by the kernel (PTE + * permissions restored immediately) without delivering a message + * to the userfaultfd handler. Use PAGEMAP_SCAN with inverted + * PAGE_IS_ACCESSED to find pages that were not re-accessed. */ #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0) #define UFFD_FEATURE_EVENT_FORK (1<<1) @@ -248,6 +273,8 @@ struct uffdio_api { #define UFFD_FEATURE_POISON (1<<14) #define UFFD_FEATURE_WP_ASYNC (1<<15) #define UFFD_FEATURE_MOVE (1<<16) +#define UFFD_FEATURE_RWP (1<<17) +#define UFFD_FEATURE_RWP_ASYNC (1<<18) __u64 features; __u64 ioctls; @@ -263,6 +290,7 @@ struct uffdio_register { #define UFFDIO_REGISTER_MODE_MISSING ((__u64)1<<0) #define UFFDIO_REGISTER_MODE_WP ((__u64)1<<1) #define UFFDIO_REGISTER_MODE_MINOR ((__u64)1<<2) +#define UFFDIO_REGISTER_MODE_RWP ((__u64)1<<3) __u64 mode; /* @@ -356,6 +384,14 @@ struct uffdio_poison { __s64 updated; }; +struct uffdio_rwprotect { + struct uffdio_range range; + /* !RWP means undo RWP-protection */ +#define UFFDIO_RWPROTECT_MODE_RWP ((__u64)1<<0) +#define UFFDIO_RWPROTECT_MODE_DONTWAKE ((__u64)1<<1) + __u64 mode; +}; + struct uffdio_move { __u64 dst; __u64 src; @@ -374,6 +410,16 @@ struct uffdio_move { __s64 move; }; +struct uffdio_set_mode { + /* + * Toggle async mode for features at runtime. + * Supported: UFFD_FEATURE_RWP_ASYNC. + * Setting a bit in both enable and disable is invalid. + */ + __u64 enable; + __u64 disable; +}; + /* * Flags for the userfaultfd(2) system call itself. */ diff --git a/linux-headers/linux/vduse.h b/linux-headers/linux/vduse.h index e19b3c0f51..bab47129db 100644 --- a/linux-headers/linux/vduse.h +++ b/linux-headers/linux/vduse.h @@ -14,6 +14,12 @@ #define VDUSE_API_VERSION_1 1 +/* The VDUSE instance expects a request for vq ready */ +#define VDUSE_F_QUEUE_READY 0 + +/* The VDUSE instance expects a request for suspend */ +#define VDUSE_F_SUSPEND 1 + /* * Get the version of VDUSE API that kernel supported (VDUSE_API_VERSION). * This is used for future extension. @@ -63,6 +69,12 @@ struct vduse_dev_config { */ #define VDUSE_DESTROY_DEV _IOW(VDUSE_BASE, 0x03, char[VDUSE_NAME_MAX]) +/* Get the VDUSE supported features */ +#define VDUSE_GET_FEATURES _IOR(VDUSE_BASE, 0x04, __u64) + +/* Set the VDUSE features */ +#define VDUSE_SET_FEATURES _IOW(VDUSE_BASE, 0x05, __u64) + /* The ioctls for VDUSE device (/dev/vduse/$NAME) */ /** @@ -325,6 +337,8 @@ enum vduse_req_type { VDUSE_SET_STATUS, VDUSE_UPDATE_IOTLB, VDUSE_SET_VQ_GROUP_ASID, + VDUSE_SET_VQ_READY, + VDUSE_SUSPEND, }; /** @@ -372,6 +386,15 @@ struct vduse_iova_range_v2 { __u32 padding; }; +/** + * struct vduse_vq_ready - Virtqueue ready request message + * @num: Virtqueue number + */ +struct vduse_vq_ready { + __u32 num; + __u32 ready; +}; + /** * struct vduse_dev_request - control request * @type: request type @@ -382,6 +405,7 @@ struct vduse_iova_range_v2 { * @iova: IOVA range for updating * @iova_v2: IOVA range for updating if API_VERSION >= 1 * @vq_group_asid: ASID of a virtqueue group + * @vq_ready: Virtqueue ready request * @padding: padding * * Structure used by read(2) on /dev/vduse/$NAME. @@ -399,6 +423,10 @@ struct vduse_dev_request { */ struct vduse_iova_range_v2 iova_v2; struct vduse_vq_group_asid vq_group_asid; + + /* Only if VDUSE_F_QUEUE_READY is negotiated */ + struct vduse_vq_ready vq_ready; + __u32 padding[32]; }; }; diff --git a/linux-headers/linux/vfio.h b/linux-headers/linux/vfio.h index f3282b8e86..c85dcbfe30 100644 --- a/linux-headers/linux/vfio.h +++ b/linux-headers/linux/vfio.h @@ -1534,6 +1534,27 @@ struct vfio_device_feature_dma_buf { */ #define VFIO_DEVICE_FEATURE_MIG_PRECOPY_INFOv2 12 +/** + * VFIO_DEVICE_FEATURE_ZPCI_ERROR feature provides PCI error information to + * userspace for vfio-pci devices on s390. On s390, PCI error recovery + * involves platform firmware and notification to operating systems is done + * by architecture specific mechanism. Exposing this information to + * userspace allows it to take appropriate actions to handle an + * error on the device. + * + * Userspace provides an opaque buffer of fixed length, and the kernel + * fills it with the zpci_ccdf_err data structure. The length of + * zpci_ccdf_err is provided to userspace via the + * VFIO_DEVICE_INFO_CAP_ZPCI_BASE capability. + * + * The ioctl returns -ENOMSG if there are no pending PCI errors. + */ +struct vfio_device_feature_zpci_err { + __aligned_u64 data; +}; + +#define VFIO_DEVICE_FEATURE_ZPCI_ERROR 13 + /* -------- API for Type1 VFIO IOMMU -------- */ /** diff --git a/linux-headers/linux/vfio_zdev.h b/linux-headers/linux/vfio_zdev.h index 77f2aff1f2..d6c1175c88 100644 --- a/linux-headers/linux/vfio_zdev.h +++ b/linux-headers/linux/vfio_zdev.h @@ -32,6 +32,8 @@ struct vfio_device_info_cap_zpci_base { /* End of version 1 */ __u32 fh; /* PCI function handle */ /* End of version 2 */ + __u32 ccdf_err_length; /* PCI CCDF length */ + /* End of version 3 */ }; /** -- 2.43.0 ^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v4 2/4] vfio/pci: Add an error handler callback 2026-08-31 18:31 [PATCH v4 0/4] Error recovery for zPCI passthrough devices Farhan Ali 2026-08-31 18:31 ` [PATCH v4 1/4] linux-headers: Update Linux header to 7.3-rc1 Farhan Ali @ 2026-08-31 18:31 ` Farhan Ali 2026-09-02 8:07 ` Cédric Le Goater 2026-08-31 18:31 ` [PATCH v4 3/4] s390x/pci: Add PCI error handling for vfio pci devices Farhan Ali ` (2 subsequent siblings) 4 siblings, 1 reply; 21+ messages in thread From: Farhan Ali @ 2026-08-31 18:31 UTC (permalink / raw) To: qemu-devel, qemu-s390x; +Cc: mjrosato, alifm, farman, cohuck, alex, clg, armbru Provide a vfio error handling callback, that can be used by devices to handle PCI errors for passthrough devices. Signed-off-by: Farhan Ali <alifm@linux.ibm.com> --- hw/vfio/pci.c | 27 +++++++++++++++++++++------ hw/vfio/pci.h | 1 + 2 files changed, 22 insertions(+), 6 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 428ab2f069..a2f489b34d 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3244,21 +3244,36 @@ void vfio_pci_put_device(VFIOPCIDevice *vdev) static void vfio_err_notifier_handler(void *opaque) { VFIOPCIDevice *vdev = opaque; + Error *err = NULL; if (!event_notifier_test_and_clear(&vdev->err_notifier)) { return; } /* - * TBD. Retrieve the error details and decide what action - * needs to be taken. One of the actions could be to pass - * the error to the guest and have the guest driver recover - * from the error. This requires that PCIe capabilities be - * exposed to the guest. For now, we just terminate the + * We can retrieve the error details and decide what action + * needs to be taken in err_handler(). One of the actions could + * be to pass the error to the guest and have the guest driver + * recover from the error. This requires that PCIe capabilities be + * exposed to the guest. + * + * If err_handler() is not implemented/fails, we just terminate the * guest to contain the error. */ - error_report("%s(%s) Unrecoverable error detected. Please collect any data possible and then kill the guest", __func__, vdev->vbasedev.name); + if (vdev->err_handler && vdev->err_handler(vdev, &err)) { + return; + } + + if (err) { + error_prepend(&err, "Unrecoverable PCIe error detected for device %s", + vdev->vbasedev.name); + error_report_err(err); + } else { + error_printf("Unrecoverable PCIe error detected for device %s", + vdev->vbasedev.name); + } + error_printf("Please collect any data possible and then kill the guest"); vm_stop(RUN_STATE_INTERNAL_ERROR); } diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index c9ab949870..c067bbbebc 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -146,6 +146,7 @@ struct VFIOPCIDevice { EventNotifier err_notifier; EventNotifier req_notifier; int (*resetfn)(struct VFIOPCIDevice *); + bool (*err_handler)(struct VFIOPCIDevice *, Error **); uint32_t vendor_id; uint32_t device_id; uint32_t sub_vendor_id; -- 2.43.0 ^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [PATCH v4 2/4] vfio/pci: Add an error handler callback 2026-08-31 18:31 ` [PATCH v4 2/4] vfio/pci: Add an error handler callback Farhan Ali @ 2026-09-02 8:07 ` Cédric Le Goater 2026-09-02 8:21 ` Shameer Kolothum Thodi ` (2 more replies) 0 siblings, 3 replies; 21+ messages in thread From: Cédric Le Goater @ 2026-09-02 8:07 UTC (permalink / raw) To: Farhan Ali, qemu-devel, qemu-s390x Cc: mjrosato, farman, cohuck, alex, armbru, Shameer Kolothum +Shameer, who is looking at forwarding AER errors to guest : https://lore.kernel.org/qemu-devel/SJ0PR12MB8614DDFEE3A9575564EDEF99ABFF2@SJ0PR12MB8614.namprd12.prod.outlook.com/ On 8/31/26 20:31, Farhan Ali wrote: > Provide a vfio error handling callback, that can be used by devices to > handle PCI errors for passthrough devices. > > Signed-off-by: Farhan Ali <alifm@linux.ibm.com> > --- > hw/vfio/pci.c | 27 +++++++++++++++++++++------ > hw/vfio/pci.h | 1 + > 2 files changed, 22 insertions(+), 6 deletions(-) > > diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c > index 428ab2f069..a2f489b34d 100644 > --- a/hw/vfio/pci.c > +++ b/hw/vfio/pci.c > @@ -3244,21 +3244,36 @@ void vfio_pci_put_device(VFIOPCIDevice *vdev) > static void vfio_err_notifier_handler(void *opaque) > { > VFIOPCIDevice *vdev = opaque; > + Error *err = NULL; > > if (!event_notifier_test_and_clear(&vdev->err_notifier)) { > return; > } > > /* > - * TBD. Retrieve the error details and decide what action > - * needs to be taken. One of the actions could be to pass > - * the error to the guest and have the guest driver recover > - * from the error. This requires that PCIe capabilities be > - * exposed to the guest. For now, we just terminate the > + * We can retrieve the error details and decide what action > + * needs to be taken in err_handler(). One of the actions could > + * be to pass the error to the guest and have the guest driver > + * recover from the error. This requires that PCIe capabilities be > + * exposed to the guest. > + * > + * If err_handler() is not implemented/fails, we just terminate the > * guest to contain the error. > */ > > - error_report("%s(%s) Unrecoverable error detected. Please collect any data possible and then kill the guest", __func__, vdev->vbasedev.name); > + if (vdev->err_handler && vdev->err_handler(vdev, &err)) { > + return; > + } > + > + if (err) { > + error_prepend(&err, "Unrecoverable PCIe error detected for device %s", > + vdev->vbasedev.name); > + error_report_err(err); > + } else { > + error_printf("Unrecoverable PCIe error detected for device %s", > + vdev->vbasedev.name); > + } > + error_printf("Please collect any data possible and then kill the guest"); how about that instead : if (err) { error_report("Unrecoverable PCIe error detected for device %s: %s", vdev->vbasedev.name, error_get_pretty(err)); error_free(err); } else { error_report("Unrecoverable PCIe error detected for device %s", vdev->vbasedev.name); } error_printf("Please collect any data possible and then kill the guest\n"); > > vm_stop(RUN_STATE_INTERNAL_ERROR); > } > diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h > index c9ab949870..c067bbbebc 100644 > --- a/hw/vfio/pci.h > +++ b/hw/vfio/pci.h > @@ -146,6 +146,7 @@ struct VFIOPCIDevice { > EventNotifier err_notifier; > EventNotifier req_notifier; > int (*resetfn)(struct VFIOPCIDevice *); > + bool (*err_handler)(struct VFIOPCIDevice *, Error **); Please add documentation, something like : /* * Platform-specific error recovery handler. * * Called when the host reports a PCI error via the VFIO error notifier. * The handler should attempt to recover the device and forward the * error to the guest if the platform supports it. * * @vdev: the VFIO PCI device that triggered the error * @errp: set with the failure reason on false return * * Return true on success, the VM continues running. * Return false on failure and set @errp, the VM will be stopped. */ bool (*err_handler)(struct VFIOPCIDevice *vdev, Error **errp); That said, I'd prefer to see AER forwarding first. Thanks, C. > uint32_t vendor_id; > uint32_t device_id; > uint32_t sub_vendor_id; ^ permalink raw reply [flat|nested] 21+ messages in thread
* RE: [PATCH v4 2/4] vfio/pci: Add an error handler callback 2026-09-02 8:07 ` Cédric Le Goater @ 2026-09-02 8:21 ` Shameer Kolothum Thodi 2026-09-02 10:39 ` Markus Armbruster 2026-09-02 16:34 ` Farhan Ali 2 siblings, 0 replies; 21+ messages in thread From: Shameer Kolothum Thodi @ 2026-09-02 8:21 UTC (permalink / raw) To: Cédric Le Goater, Farhan Ali, qemu-devel@nongnu.org, qemu-s390x@nongnu.org Cc: mjrosato@linux.ibm.com, farman@linux.ibm.com, cohuck@redhat.com, alex@shazbot.org, armbru@redhat.com > -----Original Message----- > From: Cédric Le Goater <clg@redhat.com> > Sent: 02 September 2026 09:08 > To: Farhan Ali <alifm@linux.ibm.com>; qemu-devel@nongnu.org; qemu- > s390x@nongnu.org > Cc: mjrosato@linux.ibm.com; farman@linux.ibm.com; cohuck@redhat.com; > alex@shazbot.org; armbru@redhat.com; Shameer Kolothum Thodi > <skolothumtho@nvidia.com> > Subject: Re: [PATCH v4 2/4] vfio/pci: Add an error handler callback > > External email: Use caution opening links or attachments > > > +Shameer, who is looking at forwarding AER errors to guest : Thanks Cédric. I have sent out a kernel RFC series yesterday: https://lore.kernel.org/all/20260901093217.8539-1-skolothumtho@nvidia.com/ Please take a look. Probably this series can be extended later to come with a generic AER error handling incorporating the new eventfd proposed in that RFC. I will go through this one. Thanks, Shameer > > https://lore.kernel.org/qemu- > devel/SJ0PR12MB8614DDFEE3A9575564EDEF99ABFF2@SJ0PR12MB8614.n > amprd12.prod.outlook.com/ > > On 8/31/26 20:31, Farhan Ali wrote: > > Provide a vfio error handling callback, that can be used by devices to > > handle PCI errors for passthrough devices. > > > > Signed-off-by: Farhan Ali <alifm@linux.ibm.com> > > --- > > hw/vfio/pci.c | 27 +++++++++++++++++++++------ > > hw/vfio/pci.h | 1 + > > 2 files changed, 22 insertions(+), 6 deletions(-) > > > > diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index > > 428ab2f069..a2f489b34d 100644 > > --- a/hw/vfio/pci.c > > +++ b/hw/vfio/pci.c > > @@ -3244,21 +3244,36 @@ void vfio_pci_put_device(VFIOPCIDevice > *vdev) > > static void vfio_err_notifier_handler(void *opaque) > > { > > VFIOPCIDevice *vdev = opaque; > > + Error *err = NULL; > > > > if (!event_notifier_test_and_clear(&vdev->err_notifier)) { > > return; > > } > > > > /* > > - * TBD. Retrieve the error details and decide what action > > - * needs to be taken. One of the actions could be to pass > > - * the error to the guest and have the guest driver recover > > - * from the error. This requires that PCIe capabilities be > > - * exposed to the guest. For now, we just terminate the > > + * We can retrieve the error details and decide what action > > + * needs to be taken in err_handler(). One of the actions could > > + * be to pass the error to the guest and have the guest driver > > + * recover from the error. This requires that PCIe capabilities be > > + * exposed to the guest. > > + * > > + * If err_handler() is not implemented/fails, we just terminate > > + the > > * guest to contain the error. > > */ > > > > - error_report("%s(%s) Unrecoverable error detected. Please collect any > data possible and then kill the guest", __func__, vdev->vbasedev.name); > > + if (vdev->err_handler && vdev->err_handler(vdev, &err)) { > > + return; > > + } > > + > > + if (err) { > > + error_prepend(&err, "Unrecoverable PCIe error detected for device > %s", > > + vdev->vbasedev.name); > > + error_report_err(err); > > + } else { > > + error_printf("Unrecoverable PCIe error detected for device %s", > > + vdev->vbasedev.name); > > + } > > + error_printf("Please collect any data possible and then kill the > > + guest"); > > how about that instead : > > if (err) { > error_report("Unrecoverable PCIe error detected for device %s: %s", > vdev->vbasedev.name, error_get_pretty(err)); > error_free(err); > } else { > error_report("Unrecoverable PCIe error detected for device %s", > vdev->vbasedev.name); > } > error_printf("Please collect any data possible and then kill the guest\n"); > > > > > vm_stop(RUN_STATE_INTERNAL_ERROR); > > } > > diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index > > c9ab949870..c067bbbebc 100644 > > --- a/hw/vfio/pci.h > > +++ b/hw/vfio/pci.h > > @@ -146,6 +146,7 @@ struct VFIOPCIDevice { > > EventNotifier err_notifier; > > EventNotifier req_notifier; > > int (*resetfn)(struct VFIOPCIDevice *); > > + bool (*err_handler)(struct VFIOPCIDevice *, Error **); > > Please add documentation, something like : > > /* > * Platform-specific error recovery handler. > * > * Called when the host reports a PCI error via the VFIO error notifier. > * The handler should attempt to recover the device and forward the > * error to the guest if the platform supports it. > * > * @vdev: the VFIO PCI device that triggered the error > * @errp: set with the failure reason on false return > * > * Return true on success, the VM continues running. > * Return false on failure and set @errp, the VM will be stopped. > */ > bool (*err_handler)(struct VFIOPCIDevice *vdev, Error **errp); > > > That said, I'd prefer to see AER forwarding first. > > Thanks, > > C. > > > > uint32_t vendor_id; > > uint32_t device_id; > > uint32_t sub_vendor_id; ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v4 2/4] vfio/pci: Add an error handler callback 2026-09-02 8:07 ` Cédric Le Goater 2026-09-02 8:21 ` Shameer Kolothum Thodi @ 2026-09-02 10:39 ` Markus Armbruster 2026-09-02 16:23 ` Farhan Ali 2026-09-02 16:34 ` Farhan Ali 2 siblings, 1 reply; 21+ messages in thread From: Markus Armbruster @ 2026-09-02 10:39 UTC (permalink / raw) To: Cédric Le Goater Cc: Farhan Ali, qemu-devel, qemu-s390x, mjrosato, farman, cohuck, alex, Shameer Kolothum Cédric Le Goater <clg@redhat.com> writes: > +Shameer, who is looking at forwarding AER errors to guest : > > https://lore.kernel.org/qemu-devel/SJ0PR12MB8614DDFEE3A9575564EDEF99ABFF2@SJ0PR12MB8614.namprd12.prod.outlook.com/ > > On 8/31/26 20:31, Farhan Ali wrote: >> Provide a vfio error handling callback, that can be used by devices to >> handle PCI errors for passthrough devices. >> >> Signed-off-by: Farhan Ali <alifm@linux.ibm.com> >> --- >> hw/vfio/pci.c | 27 +++++++++++++++++++++------ >> hw/vfio/pci.h | 1 + >> 2 files changed, 22 insertions(+), 6 deletions(-) >> >> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c >> index 428ab2f069..a2f489b34d 100644 >> --- a/hw/vfio/pci.c >> +++ b/hw/vfio/pci.c >> @@ -3244,21 +3244,36 @@ void vfio_pci_put_device(VFIOPCIDevice *vdev) >> static void vfio_err_notifier_handler(void *opaque) >> { >> VFIOPCIDevice *vdev = opaque; >> + Error *err = NULL; >> >> if (!event_notifier_test_and_clear(&vdev->err_notifier)) { >> return; >> } >> >> /* >> - * TBD. Retrieve the error details and decide what action >> - * needs to be taken. One of the actions could be to pass >> - * the error to the guest and have the guest driver recover >> - * from the error. This requires that PCIe capabilities be >> - * exposed to the guest. For now, we just terminate the >> + * We can retrieve the error details and decide what action >> + * needs to be taken in err_handler(). One of the actions could >> + * be to pass the error to the guest and have the guest driver >> + * recover from the error. This requires that PCIe capabilities be >> + * exposed to the guest. >> + * >> + * If err_handler() is not implemented/fails, we just terminate the >> * guest to contain the error. >> */ >> >> - error_report("%s(%s) Unrecoverable error detected. Please collect any data possible and then kill the guest", __func__, vdev->vbasedev.name); >> + if (vdev->err_handler && vdev->err_handler(vdev, &err)) { >> + return; >> + } >> + >> + if (err) { >> + error_prepend(&err, "Unrecoverable PCIe error detected for device %s", >> + vdev->vbasedev.name); This requires use of ERRP_GUARD(). See "Why, when and how to use ERRP_GUARD()" in qapi/error.h. >> + error_report_err(err); >> + } else { >> + error_printf("Unrecoverable PCIe error detected for device %s", >> + vdev->vbasedev.name); >> + } >> + error_printf("Please collect any data possible and then kill the guest"); > > how about that instead : > > if (err) { > error_report("Unrecoverable PCIe error detected for device %s: %s", > vdev->vbasedev.name, error_get_pretty(err)); > error_free(err); > } else { > error_report("Unrecoverable PCIe error detected for device %s", > vdev->vbasedev.name); > } > error_printf("Please collect any data possible and then kill the guest\n"); I prefer use of error_report_err(), because it displays hints added with error_append_hint(), if any. >> >> vm_stop(RUN_STATE_INTERNAL_ERROR); >> } [...] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v4 2/4] vfio/pci: Add an error handler callback 2026-09-02 10:39 ` Markus Armbruster @ 2026-09-02 16:23 ` Farhan Ali 2026-09-03 10:24 ` Markus Armbruster 0 siblings, 1 reply; 21+ messages in thread From: Farhan Ali @ 2026-09-02 16:23 UTC (permalink / raw) To: Markus Armbruster, Cédric Le Goater Cc: qemu-devel, qemu-s390x, mjrosato, farman, cohuck, alex, Shameer Kolothum On 9/2/2026 3:39 AM, Markus Armbruster wrote: > Cédric Le Goater <clg@redhat.com> writes: > >> +Shameer, who is looking at forwarding AER errors to guest : >> >> https://lore.kernel.org/qemu-devel/SJ0PR12MB8614DDFEE3A9575564EDEF99ABFF2@SJ0PR12MB8614.namprd12.prod.outlook.com/ >> >> On 8/31/26 20:31, Farhan Ali wrote: >>> Provide a vfio error handling callback, that can be used by devices to >>> handle PCI errors for passthrough devices. >>> >>> Signed-off-by: Farhan Ali <alifm@linux.ibm.com> >>> --- >>> hw/vfio/pci.c | 27 +++++++++++++++++++++------ >>> hw/vfio/pci.h | 1 + >>> 2 files changed, 22 insertions(+), 6 deletions(-) >>> >>> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c >>> index 428ab2f069..a2f489b34d 100644 >>> --- a/hw/vfio/pci.c >>> +++ b/hw/vfio/pci.c >>> @@ -3244,21 +3244,36 @@ void vfio_pci_put_device(VFIOPCIDevice *vdev) >>> static void vfio_err_notifier_handler(void *opaque) >>> { >>> VFIOPCIDevice *vdev = opaque; >>> + Error *err = NULL; >>> >>> if (!event_notifier_test_and_clear(&vdev->err_notifier)) { >>> return; >>> } >>> >>> /* >>> - * TBD. Retrieve the error details and decide what action >>> - * needs to be taken. One of the actions could be to pass >>> - * the error to the guest and have the guest driver recover >>> - * from the error. This requires that PCIe capabilities be >>> - * exposed to the guest. For now, we just terminate the >>> + * We can retrieve the error details and decide what action >>> + * needs to be taken in err_handler(). One of the actions could >>> + * be to pass the error to the guest and have the guest driver >>> + * recover from the error. This requires that PCIe capabilities be >>> + * exposed to the guest. >>> + * >>> + * If err_handler() is not implemented/fails, we just terminate the >>> * guest to contain the error. >>> */ >>> >>> - error_report("%s(%s) Unrecoverable error detected. Please collect any data possible and then kill the guest", __func__, vdev->vbasedev.name); >>> + if (vdev->err_handler && vdev->err_handler(vdev, &err)) { >>> + return; >>> + } >>> + >>> + if (err) { >>> + error_prepend(&err, "Unrecoverable PCIe error detected for device %s", >>> + vdev->vbasedev.name); > This requires use of ERRP_GUARD(). See "Why, when and how to use > ERRP_GUARD()" in qapi/error.h. okay, I can add the ERRP_GUARD(). AFAIU it will then look something like this? Error **errp = NULL; ERRP_GUARD(); if (vdev->err_handler && vdev->err_handler(vdev, errp)) { return; } if (*errp) { //print error with errp } else { //generic error } Thanks Farhan ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v4 2/4] vfio/pci: Add an error handler callback 2026-09-02 16:23 ` Farhan Ali @ 2026-09-03 10:24 ` Markus Armbruster 0 siblings, 0 replies; 21+ messages in thread From: Markus Armbruster @ 2026-09-03 10:24 UTC (permalink / raw) To: Farhan Ali Cc: Cédric Le Goater, qemu-devel, qemu-s390x, mjrosato, farman, cohuck, alex, Shameer Kolothum Farhan Ali <alifm@linux.ibm.com> writes: > On 9/2/2026 3:39 AM, Markus Armbruster wrote: >> Cédric Le Goater <clg@redhat.com> writes: >> >>> +Shameer, who is looking at forwarding AER errors to guest : >>> >>> https://lore.kernel.org/qemu-devel/SJ0PR12MB8614DDFEE3A9575564EDEF99ABFF2@SJ0PR12MB8614.namprd12.prod.outlook.com/ >>> >>> On 8/31/26 20:31, Farhan Ali wrote: >>>> Provide a vfio error handling callback, that can be used by devices to >>>> handle PCI errors for passthrough devices. >>>> >>>> Signed-off-by: Farhan Ali <alifm@linux.ibm.com> >>>> --- >>>> hw/vfio/pci.c | 27 +++++++++++++++++++++------ >>>> hw/vfio/pci.h | 1 + >>>> 2 files changed, 22 insertions(+), 6 deletions(-) >>>> >>>> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c >>>> index 428ab2f069..a2f489b34d 100644 >>>> --- a/hw/vfio/pci.c >>>> +++ b/hw/vfio/pci.c >>>> @@ -3244,21 +3244,36 @@ void vfio_pci_put_device(VFIOPCIDevice *vdev) >>>> static void vfio_err_notifier_handler(void *opaque) >>>> { >>>> VFIOPCIDevice *vdev = opaque; >>>> + Error *err = NULL; >>>> >>>> if (!event_notifier_test_and_clear(&vdev->err_notifier)) { >>>> return; >>>> } >>>> >>>> /* >>>> - * TBD. Retrieve the error details and decide what action >>>> - * needs to be taken. One of the actions could be to pass >>>> - * the error to the guest and have the guest driver recover >>>> - * from the error. This requires that PCIe capabilities be >>>> - * exposed to the guest. For now, we just terminate the >>>> + * We can retrieve the error details and decide what action >>>> + * needs to be taken in err_handler(). One of the actions could >>>> + * be to pass the error to the guest and have the guest driver >>>> + * recover from the error. This requires that PCIe capabilities be >>>> + * exposed to the guest. >>>> + * >>>> + * If err_handler() is not implemented/fails, we just terminate the >>>> * guest to contain the error. >>>> */ >>>> >>>> - error_report("%s(%s) Unrecoverable error detected. Please collect any data possible and then kill the guest", __func__, vdev->vbasedev.name); >>>> + if (vdev->err_handler && vdev->err_handler(vdev, &err)) { >>>> + return; >>>> + } >>>> + >>>> + if (err) { >>>> + error_prepend(&err, "Unrecoverable PCIe error detected for device %s", >>>> + vdev->vbasedev.name); >> >> This requires use of ERRP_GUARD(). See "Why, when and how to use >> ERRP_GUARD()" in qapi/error.h. > > okay, I can add the ERRP_GUARD(). AFAIU it will then look something like this? > > Error **errp = NULL; Pretty sure you mean Error *errp = NULL; > ERRP_GUARD(); Put ERRP_GUARD() right at the beginning of the function. > > if (vdev->err_handler && vdev->err_handler(vdev, errp)) { > return; > } > > if (*errp) { > //print error with errp > } else { > //generic error > } I don't this "uh, have we got an error?", to be honest. What about: if (!vdev->err_handler) { error_report([...]); } else if (!vdev->err_handler(vdev, &err)) { error_prepend(&err, [...]); error_report_err(err); } else { /* Error handled */ return; } vm_stop(RUN_STATE_INTERNAL_ERROR); This makes it quite obvious what each error report is about. Requires vdev->err_handler() to set an error when it fails, which it totally should. Matter of taste, and I'm not the maintainer here, Cédric and Alex are. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v4 2/4] vfio/pci: Add an error handler callback 2026-09-02 8:07 ` Cédric Le Goater 2026-09-02 8:21 ` Shameer Kolothum Thodi 2026-09-02 10:39 ` Markus Armbruster @ 2026-09-02 16:34 ` Farhan Ali 2026-09-03 12:08 ` Shameer Kolothum Thodi 2 siblings, 1 reply; 21+ messages in thread From: Farhan Ali @ 2026-09-02 16:34 UTC (permalink / raw) To: Cédric Le Goater, qemu-devel, qemu-s390x Cc: mjrosato, farman, cohuck, alex, armbru, Shameer Kolothum On 9/2/2026 1:07 AM, Cédric Le Goater wrote: > +Shameer, who is looking at forwarding AER errors to guest : > > https://lore.kernel.org/qemu-devel/SJ0PR12MB8614DDFEE3A9575564EDEF99ABFF2@SJ0PR12MB8614.namprd12.prod.outlook.com/ > > On 8/31/26 20:31, Farhan Ali wrote: >> Provide a vfio error handling callback, that can be used by devices to >> handle PCI errors for passthrough devices. >> >> Signed-off-by: Farhan Ali <alifm@linux.ibm.com> >> --- >> hw/vfio/pci.c | 27 +++++++++++++++++++++------ >> hw/vfio/pci.h | 1 + >> 2 files changed, 22 insertions(+), 6 deletions(-) >> >> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c >> index 428ab2f069..a2f489b34d 100644 >> --- a/hw/vfio/pci.c >> +++ b/hw/vfio/pci.c >> @@ -3244,21 +3244,36 @@ void vfio_pci_put_device(VFIOPCIDevice *vdev) >> static void vfio_err_notifier_handler(void *opaque) >> { >> VFIOPCIDevice *vdev = opaque; >> + Error *err = NULL; >> if (!event_notifier_test_and_clear(&vdev->err_notifier)) { >> return; >> } >> /* >> - * TBD. Retrieve the error details and decide what action >> - * needs to be taken. One of the actions could be to pass >> - * the error to the guest and have the guest driver recover >> - * from the error. This requires that PCIe capabilities be >> - * exposed to the guest. For now, we just terminate the >> + * We can retrieve the error details and decide what action >> + * needs to be taken in err_handler(). One of the actions could >> + * be to pass the error to the guest and have the guest driver >> + * recover from the error. This requires that PCIe capabilities be >> + * exposed to the guest. >> + * >> + * If err_handler() is not implemented/fails, we just terminate the >> * guest to contain the error. >> */ >> - error_report("%s(%s) Unrecoverable error detected. Please >> collect any data possible and then kill the guest", __func__, >> vdev->vbasedev.name); >> + if (vdev->err_handler && vdev->err_handler(vdev, &err)) { >> + return; >> + } >> + >> + if (err) { >> + error_prepend(&err, "Unrecoverable PCIe error detected for >> device %s", >> + vdev->vbasedev.name); >> + error_report_err(err); >> + } else { >> + error_printf("Unrecoverable PCIe error detected for device %s", >> + vdev->vbasedev.name); >> + } >> + error_printf("Please collect any data possible and then kill the >> guest"); > > how about that instead : > > if (err) { > error_report("Unrecoverable PCIe error detected for device %s: %s", > vdev->vbasedev.name, error_get_pretty(err)); > error_free(err); > } else { > error_report("Unrecoverable PCIe error detected for device %s", > vdev->vbasedev.name); > } > error_printf("Please collect any data possible and then kill the > guest\n"); > >> vm_stop(RUN_STATE_INTERNAL_ERROR); >> } >> diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h >> index c9ab949870..c067bbbebc 100644 >> --- a/hw/vfio/pci.h >> +++ b/hw/vfio/pci.h >> @@ -146,6 +146,7 @@ struct VFIOPCIDevice { >> EventNotifier err_notifier; >> EventNotifier req_notifier; >> int (*resetfn)(struct VFIOPCIDevice *); >> + bool (*err_handler)(struct VFIOPCIDevice *, Error **); > > Please add documentation, something like : > > /* > * Platform-specific error recovery handler. > * > * Called when the host reports a PCI error via the VFIO error > notifier. > * The handler should attempt to recover the device and forward the > * error to the guest if the platform supports it. > * > * @vdev: the VFIO PCI device that triggered the error > * @errp: set with the failure reason on false return > * > * Return true on success, the VM continues running. > * Return false on failure and set @errp, the VM will be stopped. > */ > bool (*err_handler)(struct VFIOPCIDevice *vdev, Error **errp); > Yup, I can add that. > > That said, I'd prefer to see AER forwarding first. I am curious too to understand how AER forwarding will work. Based on qemu patches (which may not be complete) [1], I can't tell how AER or an event will be sent to a guest to notify of an error. Based on the discussion on the link you posted, it looks like APEI/GHES based events would be sent to the guest? Another question regarding AER recovery series, do we need some mechanism to get the AER information from the kernel for userspace? I am trying to understand if QEMU has to forward an AER event, does it need to get information such status/mask registers etc? [1] https://github.com/shamiali2008/qemu-master/commit/e45458710cd8509fa5bd81f4376126b77b6f40be > > Thanks, > > C. > > >> uint32_t vendor_id; >> uint32_t device_id; >> uint32_t sub_vendor_id; > ^ permalink raw reply [flat|nested] 21+ messages in thread
* RE: [PATCH v4 2/4] vfio/pci: Add an error handler callback 2026-09-02 16:34 ` Farhan Ali @ 2026-09-03 12:08 ` Shameer Kolothum Thodi 2026-09-03 18:07 ` Farhan Ali 0 siblings, 1 reply; 21+ messages in thread From: Shameer Kolothum Thodi @ 2026-09-03 12:08 UTC (permalink / raw) To: Farhan Ali, Cédric Le Goater, qemu-devel@nongnu.org, qemu-s390x@nongnu.org Cc: mjrosato@linux.ibm.com, farman@linux.ibm.com, cohuck@redhat.com, alex@shazbot.org, armbru@redhat.com > -----Original Message----- > From: Farhan Ali <alifm@linux.ibm.com> > Sent: 02 September 2026 17:34 > To: Cédric Le Goater <clg@redhat.com>; qemu-devel@nongnu.org; qemu- > s390x@nongnu.org > Cc: mjrosato@linux.ibm.com; farman@linux.ibm.com; cohuck@redhat.com; > alex@shazbot.org; armbru@redhat.com; Shameer Kolothum Thodi > <skolothumtho@nvidia.com> > Subject: Re: [PATCH v4 2/4] vfio/pci: Add an error handler callback > > External email: Use caution opening links or attachments > > > On 9/2/2026 1:07 AM, Cédric Le Goater wrote: > > +Shameer, who is looking at forwarding AER errors to guest : > > > > https://lore.kernel.org/qemu- > devel/SJ0PR12MB8614DDFEE3A9575564EDEF99ABFF2@SJ0PR12MB8614.n > amprd12.prod.outlook.com/ > > > > On 8/31/26 20:31, Farhan Ali wrote: > >> Provide a vfio error handling callback, that can be used by devices to > >> handle PCI errors for passthrough devices. > >> > >> Signed-off-by: Farhan Ali <alifm@linux.ibm.com> > >> --- > >> hw/vfio/pci.c | 27 +++++++++++++++++++++------ > >> hw/vfio/pci.h | 1 + > >> 2 files changed, 22 insertions(+), 6 deletions(-) > >> > >> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c > >> index 428ab2f069..a2f489b34d 100644 > >> --- a/hw/vfio/pci.c > >> +++ b/hw/vfio/pci.c > >> @@ -3244,21 +3244,36 @@ void vfio_pci_put_device(VFIOPCIDevice > *vdev) > >> static void vfio_err_notifier_handler(void *opaque) > >> { > >> VFIOPCIDevice *vdev = opaque; > >> + Error *err = NULL; > >> if (!event_notifier_test_and_clear(&vdev->err_notifier)) { > >> return; > >> } > >> /* > >> - * TBD. Retrieve the error details and decide what action > >> - * needs to be taken. One of the actions could be to pass > >> - * the error to the guest and have the guest driver recover > >> - * from the error. This requires that PCIe capabilities be > >> - * exposed to the guest. For now, we just terminate the > >> + * We can retrieve the error details and decide what action > >> + * needs to be taken in err_handler(). One of the actions could > >> + * be to pass the error to the guest and have the guest driver > >> + * recover from the error. This requires that PCIe capabilities be > >> + * exposed to the guest. > >> + * > >> + * If err_handler() is not implemented/fails, we just terminate the > >> * guest to contain the error. > >> */ > >> - error_report("%s(%s) Unrecoverable error detected. Please > >> collect any data possible and then kill the guest", __func__, > >> vdev->vbasedev.name); > >> + if (vdev->err_handler && vdev->err_handler(vdev, &err)) { > >> + return; > >> + } > >> + > >> + if (err) { > >> + error_prepend(&err, "Unrecoverable PCIe error detected for > >> device %s", > >> + vdev->vbasedev.name); > >> + error_report_err(err); > >> + } else { > >> + error_printf("Unrecoverable PCIe error detected for device %s", > >> + vdev->vbasedev.name); > >> + } > >> + error_printf("Please collect any data possible and then kill the > >> guest"); > > > > how about that instead : > > > > if (err) { > > error_report("Unrecoverable PCIe error detected for device %s: %s", > > vdev->vbasedev.name, error_get_pretty(err)); > > error_free(err); > > } else { > > error_report("Unrecoverable PCIe error detected for device %s", > > vdev->vbasedev.name); > > } > > error_printf("Please collect any data possible and then kill the > > guest\n"); > > > >> vm_stop(RUN_STATE_INTERNAL_ERROR); > >> } > >> diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h > >> index c9ab949870..c067bbbebc 100644 > >> --- a/hw/vfio/pci.h > >> +++ b/hw/vfio/pci.h > >> @@ -146,6 +146,7 @@ struct VFIOPCIDevice { > >> EventNotifier err_notifier; > >> EventNotifier req_notifier; > >> int (*resetfn)(struct VFIOPCIDevice *); > >> + bool (*err_handler)(struct VFIOPCIDevice *, Error **); > > > > Please add documentation, something like : > > > > /* > > * Platform-specific error recovery handler. > > * > > * Called when the host reports a PCI error via the VFIO error > > notifier. > > * The handler should attempt to recover the device and forward the > > * error to the guest if the platform supports it. > > * > > * @vdev: the VFIO PCI device that triggered the error > > * @errp: set with the failure reason on false return > > * > > * Return true on success, the VM continues running. > > * Return false on failure and set @errp, the VM will be stopped. > > */ > > bool (*err_handler)(struct VFIOPCIDevice *vdev, Error **errp); > > > Yup, I can add that. > > > > > > That said, I'd prefer to see AER forwarding first. > > I am curious too to understand how AER forwarding will work. Based on > qemu patches (which may not be complete) [1], I can't tell how AER or an > event will be sent to a guest to notify of an error. Based on the > discussion on the link you posted, it looks like APEI/GHES based events > would be sent to the guest? Right, that QEMU branch is an early prototype which only observes and logs, so the forwarding is not in it. I am working on a QEMU branch which adds it, based on the kernel RFC. The plan is to use native AER rather than GHES, so QEMU injects with pcie_aer_inject_error() and an emulated pcie-root-port raises the interrupt to the guest. I will post a link once it is ready. > Another question regarding AER recovery series, do we need some > mechanism to get the AER information from the kernel for userspace? Not as it stands. The feature reports how severe the event was and what the host did, not which error occurred, so QEMU cannot forward the real status bits today. If it is something that is useful then it probably needs to be added Thanks, Shameer ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v4 2/4] vfio/pci: Add an error handler callback 2026-09-03 12:08 ` Shameer Kolothum Thodi @ 2026-09-03 18:07 ` Farhan Ali 2026-09-04 9:58 ` Shameer Kolothum Thodi 0 siblings, 1 reply; 21+ messages in thread From: Farhan Ali @ 2026-09-03 18:07 UTC (permalink / raw) To: Shameer Kolothum Thodi, Cédric Le Goater, qemu-devel@nongnu.org, qemu-s390x@nongnu.org Cc: mjrosato@linux.ibm.com, farman@linux.ibm.com, cohuck@redhat.com, alex@shazbot.org, armbru@redhat.com Hi Shameer, Thanks for your response! On 9/3/2026 5:08 AM, Shameer Kolothum Thodi wrote: >>> That said, I'd prefer to see AER forwarding first. >> I am curious too to understand how AER forwarding will work. Based on >> qemu patches (which may not be complete) [1], I can't tell how AER or an >> event will be sent to a guest to notify of an error. Based on the >> discussion on the link you posted, it looks like APEI/GHES based events >> would be sent to the guest? > Right, that QEMU branch is an early prototype which only observes and > logs, so the forwarding is not in it. > > I am working on a QEMU branch which adds it, based on the kernel RFC. > The plan is to use native AER rather than GHES, so QEMU injects with > pcie_aer_inject_error() and an emulated pcie-root-port raises the > interrupt to the guest. I will post a link once it is ready. I am very curious to know (and learn) how we would do this. >> Another question regarding AER recovery series, do we need some >> mechanism to get the AER information from the kernel for userspace? > Not as it stands. The feature reports how severe the event was and what > the host did, not which error occurred, so QEMU cannot forward the real > status bits today. If it is something that is useful then it probably > needs to be added IMHO I think there might be value in providing more detailed information to the QEMU/guest. Based on my brief look at the kernel/QEMU patch series, it looks like there is not much coordination between the guest and host recovery. So when we inject an AER in the guest, is the device access still blocked? Is the device access enabled only after a guest completed the recovery? Thanks Farhan > Thanks, > Shameer ^ permalink raw reply [flat|nested] 21+ messages in thread
* RE: [PATCH v4 2/4] vfio/pci: Add an error handler callback 2026-09-03 18:07 ` Farhan Ali @ 2026-09-04 9:58 ` Shameer Kolothum Thodi 0 siblings, 0 replies; 21+ messages in thread From: Shameer Kolothum Thodi @ 2026-09-04 9:58 UTC (permalink / raw) To: Farhan Ali, Cédric Le Goater, qemu-devel@nongnu.org, qemu-s390x@nongnu.org Cc: mjrosato@linux.ibm.com, farman@linux.ibm.com, cohuck@redhat.com, alex@shazbot.org, armbru@redhat.com Hi Farhan, > -----Original Message----- > From: Farhan Ali <alifm@linux.ibm.com> > Sent: 03 September 2026 19:07 > To: Shameer Kolothum Thodi <skolothumtho@nvidia.com>; Cédric Le Goater > <clg@redhat.com>; qemu-devel@nongnu.org; qemu-s390x@nongnu.org > Cc: mjrosato@linux.ibm.com; farman@linux.ibm.com; cohuck@redhat.com; > alex@shazbot.org; armbru@redhat.com > Subject: Re: [PATCH v4 2/4] vfio/pci: Add an error handler callback > > External email: Use caution opening links or attachments > > > Hi Shameer, > > Thanks for your response! > > On 9/3/2026 5:08 AM, Shameer Kolothum Thodi wrote: > >>> That said, I'd prefer to see AER forwarding first. > >> I am curious too to understand how AER forwarding will work. Based on > >> qemu patches (which may not be complete) [1], I can't tell how AER or > >> an event will be sent to a guest to notify of an error. Based on the > >> discussion on the link you posted, it looks like APEI/GHES based > >> events would be sent to the guest? > > Right, that QEMU branch is an early prototype which only observes and > > logs, so the forwarding is not in it. > > > > I am working on a QEMU branch which adds it, based on the kernel RFC. > > The plan is to use native AER rather than GHES, so QEMU injects with > > pcie_aer_inject_error() and an emulated pcie-root-port raises the > > interrupt to the guest. I will post a link once it is ready. > > I am very curious to know (and learn) how we would do this. I have a branch that does the AER error injection into the guest here: https://github.com/shamiali2008/qemu-master/commits/private-master-vfio-aer-test-v2/ Note: This is only for RFC test purposes. > > > >> Another question regarding AER recovery series, do we need some > >> mechanism to get the AER information from the kernel for userspace? > > Not as it stands. The feature reports how severe the event was and > > what the host did, not which error occurred, so QEMU cannot forward > > the real status bits today. If it is something that is useful then it > > probably needs to be added > > IMHO I think there might be value in providing more detailed information to > the QEMU/guest. Ok. I will have a look and see how we can propagate that into userspace. > > Based on my brief look at the kernel/QEMU patch series, it looks like there is > not much coordination between the guest and host recovery. So when we > inject an AER in the guest, is the device access still blocked? > Is the device access enabled only after a guest completed the recovery? Access is already unblocked when the guest sees the error. QEMU injects the error only after host recovery has finished, so the guest driver runs against a device it can read. You are right that there is no coordination between guest and host and that is deliberate. Please see Alex's reply here on which the RFC design is based: https://lore.kernel.org/qemu-devel/20260707161234.23ed28db@nvidia.com/ The cost is that the guest only recovers once the host has finished, and cannot influence what the host does. If the guest driver wants a reset it will do one, on a device the host has already reset. Thanks, Shameer ^ permalink raw reply [flat|nested] 21+ messages in thread
* [PATCH v4 3/4] s390x/pci: Add PCI error handling for vfio pci devices 2026-08-31 18:31 [PATCH v4 0/4] Error recovery for zPCI passthrough devices Farhan Ali 2026-08-31 18:31 ` [PATCH v4 1/4] linux-headers: Update Linux header to 7.3-rc1 Farhan Ali 2026-08-31 18:31 ` [PATCH v4 2/4] vfio/pci: Add an error handler callback Farhan Ali @ 2026-08-31 18:31 ` Farhan Ali 2026-09-02 8:22 ` Cédric Le Goater 2026-08-31 18:31 ` [PATCH v4 4/4] s390x/pci: Reset a device in error state Farhan Ali 2026-09-03 9:19 ` [PATCH v4 0/4] Error recovery for zPCI passthrough devices Cédric Le Goater 4 siblings, 1 reply; 21+ messages in thread From: Farhan Ali @ 2026-08-31 18:31 UTC (permalink / raw) To: qemu-devel, qemu-s390x; +Cc: mjrosato, alifm, farman, cohuck, alex, clg, armbru Add an s390x specific callback for vfio error handling. For s390x pci devices, we have platform specific error information. We need to retrieve this error information for passthrough devices. This is done via a VFIO_DEVICE_FEATURE ioctl which exposes that information. Once this error information is retrieved we can then inject an error into the guest, and let the guest drive the recovery. Signed-off-by: Farhan Ali <alifm@linux.ibm.com> --- hw/s390x/s390-pci-bus.c | 9 +++ hw/s390x/s390-pci-vfio-stubs.c | 6 ++ hw/s390x/s390-pci-vfio.c | 100 +++++++++++++++++++++++++++++++ include/hw/s390x/s390-pci-bus.h | 2 + include/hw/s390x/s390-pci-vfio.h | 1 + 5 files changed, 118 insertions(+) diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c index 9ecf3c30ad..b2db7f3df4 100644 --- a/hw/s390x/s390-pci-bus.c +++ b/hw/s390x/s390-pci-bus.c @@ -160,6 +160,8 @@ static void s390_pci_perform_unplug(S390PCIBusDevice *pbdev) { HotplugHandler *hotplug_ctrl; + qemu_mutex_destroy(&pbdev->err_handler_lock); + if (pbdev->pft == ZPCI_PFT_ISM) { notifier_remove(&pbdev->shutdown_notifier); } @@ -1123,6 +1125,7 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, S390pciState *s = S390_PCI_HOST_BRIDGE(hotplug_dev); PCIDevice *pdev = NULL; S390PCIBusDevice *pbdev = NULL; + Error *local_err = NULL; int rc; if (object_dynamic_cast(OBJECT(dev), TYPE_PCI_BRIDGE)) { @@ -1189,6 +1192,7 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, pbdev->iommu->pbdev = pbdev; pbdev->state = ZPCI_FS_DISABLED; set_pbdev_info(pbdev); + qemu_mutex_init(&pbdev->err_handler_lock); if (object_dynamic_cast(OBJECT(dev), "vfio-pci")) { /* @@ -1213,6 +1217,11 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, pbdev->iommu->dma_limit = s390_pci_start_dma_count(s, pbdev); /* Fill in CLP information passed via the vfio region */ s390_pci_get_clp_info(pbdev); + /* Setup error handler for error recovery */ + if (!s390_pci_setup_err_handler(pbdev, &local_err)) { + warn_report_err(local_err); + } + if (!pbdev->interp) { /* Do vfio passthrough but intercept for I/O */ pbdev->fh |= FH_SHM_VFIO; diff --git a/hw/s390x/s390-pci-vfio-stubs.c b/hw/s390x/s390-pci-vfio-stubs.c index d9882b7aad..9fc84ca135 100644 --- a/hw/s390x/s390-pci-vfio-stubs.c +++ b/hw/s390x/s390-pci-vfio-stubs.c @@ -30,3 +30,9 @@ bool s390_pci_get_host_fh(S390PCIBusDevice *pbdev, uint32_t *fh) void s390_pci_get_clp_info(S390PCIBusDevice *pbdev) { } + +bool s390_pci_setup_err_handler(S390PCIBusDevice *pbdev, Error **errp) +{ + error_setg(errp, "VFIO not available, cannot setup error handler"); + return false; +} diff --git a/hw/s390x/s390-pci-vfio.c b/hw/s390x/s390-pci-vfio.c index db6de00bd2..9a63040e22 100644 --- a/hw/s390x/s390-pci-vfio.c +++ b/hw/s390x/s390-pci-vfio.c @@ -105,6 +105,73 @@ void s390_pci_end_dma_count(S390pciState *s, S390PCIDMACount *cnt) } } +static int s390_pci_get_feature_err(VFIOPCIDevice *vfio_pci, + PciCcdfErr *ccdf, + uint32_t ccdf_err_length, + Error **errp) +{ + int ret; + size_t total_size; + struct vfio_device_feature_zpci_err *err; + g_autofree void *buf = NULL; + g_autofree struct vfio_device_feature *feature = NULL; + + total_size = sizeof(*feature) + sizeof(*err); + feature = g_malloc(total_size); + feature->argsz = total_size; + feature->flags = VFIO_DEVICE_FEATURE_GET | VFIO_DEVICE_FEATURE_ZPCI_ERROR; + + buf = g_malloc(ccdf_err_length); + err = (void *)feature->data; + err->data = (uint64_t)buf; + ret = vfio_device_get_feature(&vfio_pci->vbasedev, feature); + + if (ret) { + if (ret != -ENOMSG) { + error_setg(errp, "Failed feature get VFIO_DEVICE_FEATURE_ZPCI_ERROR" + " (rc=%d)", ret); + } + return ret; + } + + memcpy(ccdf, (PciCcdfErr *) err->data, ccdf_err_length); + + return 0; +} + +static bool s390_pci_err_handler(VFIOPCIDevice *vfio_pci, Error **errp) +{ + S390PCIBusDevice *pbdev; + PciCcdfErr ccdf; + bool success = false; + int ret = 0; + + pbdev = s390_pci_find_dev_by_target(s390_get_phb(), + DEVICE(&vfio_pci->parent_obj)->id); + + QEMU_LOCK_GUARD(&pbdev->err_handler_lock); + pbdev->state = ZPCI_FS_ERROR; + + if (sizeof(ccdf) != pbdev->ccdf_err_length) { + error_setg(errp, + "CCDF size mismatch expected size=%zu, provided size=%d", + sizeof(ccdf), pbdev->ccdf_err_length); + return false; + } + + while (ret == 0) { + ret = s390_pci_get_feature_err(vfio_pci, &ccdf, + pbdev->ccdf_err_length, errp); + if (ret) { + success = ret == -ENOMSG ? true : false; + break; + } + s390_pci_generate_error_event(ccdf.pec, pbdev->fh, pbdev->fid, 0, 0); + } + + return success; +} + static void s390_pci_read_base(S390PCIBusDevice *pbdev, struct vfio_device_info *info) { @@ -134,6 +201,10 @@ static void s390_pci_read_base(S390PCIBusDevice *pbdev, /* Store function type separately for type-specific behavior */ pbdev->pft = cap->pft; + if (hdr->version >= 3) { + pbdev->ccdf_err_length = cap->ccdf_err_length; + } + /* * If the device is a passthrough ISM device, disallow relaxed * translation. @@ -371,3 +442,32 @@ void s390_pci_get_clp_info(S390PCIBusDevice *pbdev) s390_pci_read_util(pbdev, info); s390_pci_read_pfip(pbdev, info); } + +bool s390_pci_setup_err_handler(S390PCIBusDevice *pbdev, Error **errp) +{ + int ret; + VFIOPCIDevice *vfio_pci = VFIO_PCI_DEVICE(pbdev->pdev); + uint64_t buf[DIV_ROUND_UP(sizeof(struct vfio_device_feature), + sizeof(uint64_t))] = {}; + struct vfio_device_feature *feature = (struct vfio_device_feature *)buf; + + feature->argsz = sizeof(buf); + feature->flags = VFIO_DEVICE_FEATURE_PROBE | VFIO_DEVICE_FEATURE_ZPCI_ERROR; + + ret = vfio_device_get_feature(&vfio_pci->vbasedev, feature); + + if (ret != 0) { + if (ret == -ENOTTY) { + error_setg(errp, "Automated error recovery unavailable for device"); + } else { + error_setg(errp, + "Failed to probe for VFIO_DEVICE_FEATURE_ZPCI_ERROR (ret=%d)", + ret); + } + return false; + } + + vfio_pci->err_handler = s390_pci_err_handler; + + return true; +} diff --git a/include/hw/s390x/s390-pci-bus.h b/include/hw/s390x/s390-pci-bus.h index 9228523ce8..bc67f7e065 100644 --- a/include/hw/s390x/s390-pci-bus.h +++ b/include/hw/s390x/s390-pci-bus.h @@ -364,6 +364,8 @@ struct S390PCIBusDevice { bool forwarding_assist; bool aif; bool rtr_avail; + QemuMutex err_handler_lock; + uint32_t ccdf_err_length; QTAILQ_ENTRY(S390PCIBusDevice) link; }; diff --git a/include/hw/s390x/s390-pci-vfio.h b/include/hw/s390x/s390-pci-vfio.h index f7d6149daf..c7886b63ea 100644 --- a/include/hw/s390x/s390-pci-vfio.h +++ b/include/hw/s390x/s390-pci-vfio.h @@ -20,5 +20,6 @@ S390PCIDMACount *s390_pci_start_dma_count(S390pciState *s, void s390_pci_end_dma_count(S390pciState *s, S390PCIDMACount *cnt); bool s390_pci_get_host_fh(S390PCIBusDevice *pbdev, uint32_t *fh); void s390_pci_get_clp_info(S390PCIBusDevice *pbdev); +bool s390_pci_setup_err_handler(S390PCIBusDevice *pbdev, Error **errp); #endif -- 2.43.0 ^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [PATCH v4 3/4] s390x/pci: Add PCI error handling for vfio pci devices 2026-08-31 18:31 ` [PATCH v4 3/4] s390x/pci: Add PCI error handling for vfio pci devices Farhan Ali @ 2026-09-02 8:22 ` Cédric Le Goater 2026-09-02 16:43 ` Farhan Ali 0 siblings, 1 reply; 21+ messages in thread From: Cédric Le Goater @ 2026-09-02 8:22 UTC (permalink / raw) To: Farhan Ali, qemu-devel, qemu-s390x; +Cc: mjrosato, farman, cohuck, alex, armbru On 8/31/26 20:31, Farhan Ali wrote: > Add an s390x specific callback for vfio error handling. For s390x pci devices, > we have platform specific error information. We need to retrieve this error > information for passthrough devices. This is done via a VFIO_DEVICE_FEATURE > ioctl which exposes that information. > > Once this error information is retrieved we can then inject an error into > the guest, and let the guest drive the recovery. > > Signed-off-by: Farhan Ali <alifm@linux.ibm.com> > --- > hw/s390x/s390-pci-bus.c | 9 +++ > hw/s390x/s390-pci-vfio-stubs.c | 6 ++ > hw/s390x/s390-pci-vfio.c | 100 +++++++++++++++++++++++++++++++ > include/hw/s390x/s390-pci-bus.h | 2 + > include/hw/s390x/s390-pci-vfio.h | 1 + > 5 files changed, 118 insertions(+) > > diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c > index 9ecf3c30ad..b2db7f3df4 100644 > --- a/hw/s390x/s390-pci-bus.c > +++ b/hw/s390x/s390-pci-bus.c > @@ -160,6 +160,8 @@ static void s390_pci_perform_unplug(S390PCIBusDevice *pbdev) > { > HotplugHandler *hotplug_ctrl; > > + qemu_mutex_destroy(&pbdev->err_handler_lock); > + > if (pbdev->pft == ZPCI_PFT_ISM) { > notifier_remove(&pbdev->shutdown_notifier); > } > @@ -1123,6 +1125,7 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, > S390pciState *s = S390_PCI_HOST_BRIDGE(hotplug_dev); > PCIDevice *pdev = NULL; > S390PCIBusDevice *pbdev = NULL; > + Error *local_err = NULL; > int rc; > > if (object_dynamic_cast(OBJECT(dev), TYPE_PCI_BRIDGE)) { > @@ -1189,6 +1192,7 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, > pbdev->iommu->pbdev = pbdev; > pbdev->state = ZPCI_FS_DISABLED; > set_pbdev_info(pbdev); > + qemu_mutex_init(&pbdev->err_handler_lock); > > if (object_dynamic_cast(OBJECT(dev), "vfio-pci")) { > /* > @@ -1213,6 +1217,11 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, > pbdev->iommu->dma_limit = s390_pci_start_dma_count(s, pbdev); > /* Fill in CLP information passed via the vfio region */ > s390_pci_get_clp_info(pbdev); > + /* Setup error handler for error recovery */ > + if (!s390_pci_setup_err_handler(pbdev, &local_err)) { > + warn_report_err(local_err); > + } > + > if (!pbdev->interp) { > /* Do vfio passthrough but intercept for I/O */ > pbdev->fh |= FH_SHM_VFIO; > diff --git a/hw/s390x/s390-pci-vfio-stubs.c b/hw/s390x/s390-pci-vfio-stubs.c > index d9882b7aad..9fc84ca135 100644 > --- a/hw/s390x/s390-pci-vfio-stubs.c > +++ b/hw/s390x/s390-pci-vfio-stubs.c > @@ -30,3 +30,9 @@ bool s390_pci_get_host_fh(S390PCIBusDevice *pbdev, uint32_t *fh) > void s390_pci_get_clp_info(S390PCIBusDevice *pbdev) > { > } > + > +bool s390_pci_setup_err_handler(S390PCIBusDevice *pbdev, Error **errp) > +{ > + error_setg(errp, "VFIO not available, cannot setup error handler"); > + return false; > +} > diff --git a/hw/s390x/s390-pci-vfio.c b/hw/s390x/s390-pci-vfio.c > index db6de00bd2..9a63040e22 100644 > --- a/hw/s390x/s390-pci-vfio.c > +++ b/hw/s390x/s390-pci-vfio.c > @@ -105,6 +105,73 @@ void s390_pci_end_dma_count(S390pciState *s, S390PCIDMACount *cnt) > } > } > > +static int s390_pci_get_feature_err(VFIOPCIDevice *vfio_pci, returning a bool would be preferred unless the errno is important > + PciCcdfErr *ccdf, > + uint32_t ccdf_err_length, > + Error **errp) > +{ > + int ret; > + size_t total_size; > + struct vfio_device_feature_zpci_err *err; > + g_autofree void *buf = NULL; > + g_autofree struct vfio_device_feature *feature = NULL; > + > + total_size = sizeof(*feature) + sizeof(*err); > + feature = g_malloc(total_size); > + feature->argsz = total_size; > + feature->flags = VFIO_DEVICE_FEATURE_GET | VFIO_DEVICE_FEATURE_ZPCI_ERROR; > + > + buf = g_malloc(ccdf_err_length); > + err = (void *)feature->data; > + err->data = (uint64_t)buf; > + ret = vfio_device_get_feature(&vfio_pci->vbasedev, feature); > + > + if (ret) { > + if (ret != -ENOMSG) { > + error_setg(errp, "Failed feature get VFIO_DEVICE_FEATURE_ZPCI_ERROR" > + " (rc=%d)", ret); > + } > + return ret; > + } > + > + memcpy(ccdf, (PciCcdfErr *) err->data, ccdf_err_length); > + > + return 0; > +} > + > +static bool s390_pci_err_handler(VFIOPCIDevice *vfio_pci, Error **errp) > +{ > + S390PCIBusDevice *pbdev; > + PciCcdfErr ccdf; > + bool success = false; This means errp should be set. It is not obvious from the code below. > + int ret = 0; > + > + pbdev = s390_pci_find_dev_by_target(s390_get_phb(), > + DEVICE(&vfio_pci->parent_obj)->id); can pbdev be NULL ? if (!pbdev) { error_setg(errp, "No matching zpci device found"); return false; } > + > + QEMU_LOCK_GUARD(&pbdev->err_handler_lock); > + pbdev->state = ZPCI_FS_ERROR; > + > + if (sizeof(ccdf) != pbdev->ccdf_err_length) { > + error_setg(errp, > + "CCDF size mismatch expected size=%zu, provided size=%d", > + sizeof(ccdf), pbdev->ccdf_err_length); > + return false; > + } > + > + while (ret == 0) { > + ret = s390_pci_get_feature_err(vfio_pci, &ccdf, > + pbdev->ccdf_err_length, errp); > + if (ret) { > + success = ret == -ENOMSG ? true : false; > + break; > + } > + s390_pci_generate_error_event(ccdf.pec, pbdev->fh, pbdev->fid, 0, 0); why not use ccdf.e and ccdf.faddr for more diagnostic details ? > + } > + > + return success; > +} > + > static void s390_pci_read_base(S390PCIBusDevice *pbdev, > struct vfio_device_info *info) > { > @@ -134,6 +201,10 @@ static void s390_pci_read_base(S390PCIBusDevice *pbdev, > /* Store function type separately for type-specific behavior */ > pbdev->pft = cap->pft; > > + if (hdr->version >= 3) { > + pbdev->ccdf_err_length = cap->ccdf_err_length; > + } > + > /* > * If the device is a passthrough ISM device, disallow relaxed > * translation. > @@ -371,3 +442,32 @@ void s390_pci_get_clp_info(S390PCIBusDevice *pbdev) > s390_pci_read_util(pbdev, info); > s390_pci_read_pfip(pbdev, info); > } > + > +bool s390_pci_setup_err_handler(S390PCIBusDevice *pbdev, Error **errp) > +{ > + int ret; > + VFIOPCIDevice *vfio_pci = VFIO_PCI_DEVICE(pbdev->pdev); > + uint64_t buf[DIV_ROUND_UP(sizeof(struct vfio_device_feature), > + sizeof(uint64_t))] = {}; > + struct vfio_device_feature *feature = (struct vfio_device_feature *)buf; > + > + feature->argsz = sizeof(buf); > + feature->flags = VFIO_DEVICE_FEATURE_PROBE | VFIO_DEVICE_FEATURE_ZPCI_ERROR; > + > + ret = vfio_device_get_feature(&vfio_pci->vbasedev, feature); > + > + if (ret != 0) { > + if (ret == -ENOTTY) { > + error_setg(errp, "Automated error recovery unavailable for device"); > + } else { > + error_setg(errp, > + "Failed to probe for VFIO_DEVICE_FEATURE_ZPCI_ERROR (ret=%d)", > + ret); > + } > + return false; > + } > + > + vfio_pci->err_handler = s390_pci_err_handler; > + > + return true; > +} > diff --git a/include/hw/s390x/s390-pci-bus.h b/include/hw/s390x/s390-pci-bus.h > index 9228523ce8..bc67f7e065 100644 > --- a/include/hw/s390x/s390-pci-bus.h > +++ b/include/hw/s390x/s390-pci-bus.h > @@ -364,6 +364,8 @@ struct S390PCIBusDevice { > bool forwarding_assist; > bool aif; > bool rtr_avail; > + QemuMutex err_handler_lock; > + uint32_t ccdf_err_length; > QTAILQ_ENTRY(S390PCIBusDevice) link; > }; > > diff --git a/include/hw/s390x/s390-pci-vfio.h b/include/hw/s390x/s390-pci-vfio.h > index f7d6149daf..c7886b63ea 100644 > --- a/include/hw/s390x/s390-pci-vfio.h > +++ b/include/hw/s390x/s390-pci-vfio.h > @@ -20,5 +20,6 @@ S390PCIDMACount *s390_pci_start_dma_count(S390pciState *s, > void s390_pci_end_dma_count(S390pciState *s, S390PCIDMACount *cnt); > bool s390_pci_get_host_fh(S390PCIBusDevice *pbdev, uint32_t *fh); > void s390_pci_get_clp_info(S390PCIBusDevice *pbdev); > +bool s390_pci_setup_err_handler(S390PCIBusDevice *pbdev, Error **errp); > > #endif ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v4 3/4] s390x/pci: Add PCI error handling for vfio pci devices 2026-09-02 8:22 ` Cédric Le Goater @ 2026-09-02 16:43 ` Farhan Ali 0 siblings, 0 replies; 21+ messages in thread From: Farhan Ali @ 2026-09-02 16:43 UTC (permalink / raw) To: Cédric Le Goater, qemu-devel, qemu-s390x Cc: mjrosato, farman, cohuck, alex, armbru On 9/2/2026 1:22 AM, Cédric Le Goater wrote: > On 8/31/26 20:31, Farhan Ali wrote: >> Add an s390x specific callback for vfio error handling. For s390x pci >> devices, >> we have platform specific error information. We need to retrieve this >> error >> information for passthrough devices. This is done via a >> VFIO_DEVICE_FEATURE >> ioctl which exposes that information. >> >> Once this error information is retrieved we can then inject an error >> into >> the guest, and let the guest drive the recovery. >> >> Signed-off-by: Farhan Ali <alifm@linux.ibm.com> >> --- >> hw/s390x/s390-pci-bus.c | 9 +++ >> hw/s390x/s390-pci-vfio-stubs.c | 6 ++ >> hw/s390x/s390-pci-vfio.c | 100 +++++++++++++++++++++++++++++++ >> include/hw/s390x/s390-pci-bus.h | 2 + >> include/hw/s390x/s390-pci-vfio.h | 1 + >> 5 files changed, 118 insertions(+) >> >> diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c >> index 9ecf3c30ad..b2db7f3df4 100644 >> --- a/hw/s390x/s390-pci-bus.c >> +++ b/hw/s390x/s390-pci-bus.c >> @@ -160,6 +160,8 @@ static void >> s390_pci_perform_unplug(S390PCIBusDevice *pbdev) >> { >> HotplugHandler *hotplug_ctrl; >> + qemu_mutex_destroy(&pbdev->err_handler_lock); >> + >> if (pbdev->pft == ZPCI_PFT_ISM) { >> notifier_remove(&pbdev->shutdown_notifier); >> } >> @@ -1123,6 +1125,7 @@ static void s390_pcihost_plug(HotplugHandler >> *hotplug_dev, DeviceState *dev, >> S390pciState *s = S390_PCI_HOST_BRIDGE(hotplug_dev); >> PCIDevice *pdev = NULL; >> S390PCIBusDevice *pbdev = NULL; >> + Error *local_err = NULL; >> int rc; >> if (object_dynamic_cast(OBJECT(dev), TYPE_PCI_BRIDGE)) { >> @@ -1189,6 +1192,7 @@ static void s390_pcihost_plug(HotplugHandler >> *hotplug_dev, DeviceState *dev, >> pbdev->iommu->pbdev = pbdev; >> pbdev->state = ZPCI_FS_DISABLED; >> set_pbdev_info(pbdev); >> + qemu_mutex_init(&pbdev->err_handler_lock); >> if (object_dynamic_cast(OBJECT(dev), "vfio-pci")) { >> /* >> @@ -1213,6 +1217,11 @@ static void s390_pcihost_plug(HotplugHandler >> *hotplug_dev, DeviceState *dev, >> pbdev->iommu->dma_limit = s390_pci_start_dma_count(s, >> pbdev); >> /* Fill in CLP information passed via the vfio region */ >> s390_pci_get_clp_info(pbdev); >> + /* Setup error handler for error recovery */ >> + if (!s390_pci_setup_err_handler(pbdev, &local_err)) { >> + warn_report_err(local_err); >> + } >> + >> if (!pbdev->interp) { >> /* Do vfio passthrough but intercept for I/O */ >> pbdev->fh |= FH_SHM_VFIO; >> diff --git a/hw/s390x/s390-pci-vfio-stubs.c >> b/hw/s390x/s390-pci-vfio-stubs.c >> index d9882b7aad..9fc84ca135 100644 >> --- a/hw/s390x/s390-pci-vfio-stubs.c >> +++ b/hw/s390x/s390-pci-vfio-stubs.c >> @@ -30,3 +30,9 @@ bool s390_pci_get_host_fh(S390PCIBusDevice *pbdev, >> uint32_t *fh) >> void s390_pci_get_clp_info(S390PCIBusDevice *pbdev) >> { >> } >> + >> +bool s390_pci_setup_err_handler(S390PCIBusDevice *pbdev, Error **errp) >> +{ >> + error_setg(errp, "VFIO not available, cannot setup error handler"); >> + return false; >> +} >> diff --git a/hw/s390x/s390-pci-vfio.c b/hw/s390x/s390-pci-vfio.c >> index db6de00bd2..9a63040e22 100644 >> --- a/hw/s390x/s390-pci-vfio.c >> +++ b/hw/s390x/s390-pci-vfio.c >> @@ -105,6 +105,73 @@ void s390_pci_end_dma_count(S390pciState *s, >> S390PCIDMACount *cnt) >> } >> } >> +static int s390_pci_get_feature_err(VFIOPCIDevice *vfio_pci, > > returning a bool would be preferred unless the errno is important > I can change it to bool. Don't have a strong preference here. > >> + PciCcdfErr *ccdf, >> + uint32_t ccdf_err_length, >> + Error **errp) >> +{ >> + int ret; >> + size_t total_size; >> + struct vfio_device_feature_zpci_err *err; >> + g_autofree void *buf = NULL; >> + g_autofree struct vfio_device_feature *feature = NULL; >> + >> + total_size = sizeof(*feature) + sizeof(*err); >> + feature = g_malloc(total_size); >> + feature->argsz = total_size; >> + feature->flags = VFIO_DEVICE_FEATURE_GET | >> VFIO_DEVICE_FEATURE_ZPCI_ERROR; >> + >> + buf = g_malloc(ccdf_err_length); >> + err = (void *)feature->data; >> + err->data = (uint64_t)buf; >> + ret = vfio_device_get_feature(&vfio_pci->vbasedev, feature); >> + >> + if (ret) { >> + if (ret != -ENOMSG) { >> + error_setg(errp, "Failed feature get >> VFIO_DEVICE_FEATURE_ZPCI_ERROR" >> + " (rc=%d)", ret); >> + } >> + return ret; >> + } >> + >> + memcpy(ccdf, (PciCcdfErr *) err->data, ccdf_err_length); >> + >> + return 0; >> +} >> + >> +static bool s390_pci_err_handler(VFIOPCIDevice *vfio_pci, Error **errp) >> +{ >> + S390PCIBusDevice *pbdev; >> + PciCcdfErr ccdf; >> + bool success = false; > > This means errp should be set. It is not obvious from the code below. > >> + int ret = 0; >> + >> + pbdev = s390_pci_find_dev_by_target(s390_get_phb(), >> + DEVICE(&vfio_pci->parent_obj)->id); > > can pbdev be NULL ? > > if (!pbdev) { > error_setg(errp, "No matching zpci device found"); > return false; > } >> + >> + QEMU_LOCK_GUARD(&pbdev->err_handler_lock); >> + pbdev->state = ZPCI_FS_ERROR; >> + >> + if (sizeof(ccdf) != pbdev->ccdf_err_length) { >> + error_setg(errp, >> + "CCDF size mismatch expected size=%zu, provided >> size=%d", >> + sizeof(ccdf), pbdev->ccdf_err_length); >> + return false; >> + } >> + >> + while (ret == 0) { >> + ret = s390_pci_get_feature_err(vfio_pci, &ccdf, >> + pbdev->ccdf_err_length, errp); >> + if (ret) { >> + success = ret == -ENOMSG ? true : false; >> + break; >> + } >> + s390_pci_generate_error_event(ccdf.pec, pbdev->fh, >> pbdev->fid, 0, 0); > > why not use ccdf.e and ccdf.faddr for more diagnostic details ? That's a fair point, will add that. I think this was based on the older version which only had the pec as the valid information. Thanks Farhan > > >> + } >> + >> + return success; >> +} >> + >> static void s390_pci_read_base(S390PCIBusDevice *pbdev, >> struct vfio_device_info *info) >> { >> @@ -134,6 +201,10 @@ static void s390_pci_read_base(S390PCIBusDevice >> *pbdev, >> /* Store function type separately for type-specific behavior */ >> pbdev->pft = cap->pft; >> + if (hdr->version >= 3) { >> + pbdev->ccdf_err_length = cap->ccdf_err_length; >> + } >> + >> /* >> * If the device is a passthrough ISM device, disallow relaxed >> * translation. >> @@ -371,3 +442,32 @@ void s390_pci_get_clp_info(S390PCIBusDevice *pbdev) >> s390_pci_read_util(pbdev, info); >> s390_pci_read_pfip(pbdev, info); >> } >> + >> +bool s390_pci_setup_err_handler(S390PCIBusDevice *pbdev, Error **errp) >> +{ >> + int ret; >> + VFIOPCIDevice *vfio_pci = VFIO_PCI_DEVICE(pbdev->pdev); >> + uint64_t buf[DIV_ROUND_UP(sizeof(struct vfio_device_feature), >> + sizeof(uint64_t))] = {}; >> + struct vfio_device_feature *feature = (struct >> vfio_device_feature *)buf; >> + >> + feature->argsz = sizeof(buf); >> + feature->flags = VFIO_DEVICE_FEATURE_PROBE | >> VFIO_DEVICE_FEATURE_ZPCI_ERROR; >> + >> + ret = vfio_device_get_feature(&vfio_pci->vbasedev, feature); >> + >> + if (ret != 0) { >> + if (ret == -ENOTTY) { >> + error_setg(errp, "Automated error recovery unavailable >> for device"); >> + } else { >> + error_setg(errp, >> + "Failed to probe for >> VFIO_DEVICE_FEATURE_ZPCI_ERROR (ret=%d)", >> + ret); >> + } >> + return false; >> + } >> + >> + vfio_pci->err_handler = s390_pci_err_handler; >> + >> + return true; >> +} >> diff --git a/include/hw/s390x/s390-pci-bus.h >> b/include/hw/s390x/s390-pci-bus.h >> index 9228523ce8..bc67f7e065 100644 >> --- a/include/hw/s390x/s390-pci-bus.h >> +++ b/include/hw/s390x/s390-pci-bus.h >> @@ -364,6 +364,8 @@ struct S390PCIBusDevice { >> bool forwarding_assist; >> bool aif; >> bool rtr_avail; >> + QemuMutex err_handler_lock; >> + uint32_t ccdf_err_length; >> QTAILQ_ENTRY(S390PCIBusDevice) link; >> }; >> diff --git a/include/hw/s390x/s390-pci-vfio.h >> b/include/hw/s390x/s390-pci-vfio.h >> index f7d6149daf..c7886b63ea 100644 >> --- a/include/hw/s390x/s390-pci-vfio.h >> +++ b/include/hw/s390x/s390-pci-vfio.h >> @@ -20,5 +20,6 @@ S390PCIDMACount >> *s390_pci_start_dma_count(S390pciState *s, >> void s390_pci_end_dma_count(S390pciState *s, S390PCIDMACount *cnt); >> bool s390_pci_get_host_fh(S390PCIBusDevice *pbdev, uint32_t *fh); >> void s390_pci_get_clp_info(S390PCIBusDevice *pbdev); >> +bool s390_pci_setup_err_handler(S390PCIBusDevice *pbdev, Error **errp); >> #endif > ^ permalink raw reply [flat|nested] 21+ messages in thread
* [PATCH v4 4/4] s390x/pci: Reset a device in error state 2026-08-31 18:31 [PATCH v4 0/4] Error recovery for zPCI passthrough devices Farhan Ali ` (2 preceding siblings ...) 2026-08-31 18:31 ` [PATCH v4 3/4] s390x/pci: Add PCI error handling for vfio pci devices Farhan Ali @ 2026-08-31 18:31 ` Farhan Ali 2026-09-02 8:27 ` Cédric Le Goater 2026-09-03 9:19 ` [PATCH v4 0/4] Error recovery for zPCI passthrough devices Cédric Le Goater 4 siblings, 1 reply; 21+ messages in thread From: Farhan Ali @ 2026-08-31 18:31 UTC (permalink / raw) To: qemu-devel, qemu-s390x; +Cc: mjrosato, alifm, farman, cohuck, alex, clg, armbru For passthrough devices in error state, for a guest driven reset of the device we can attempt a reset to recover the device. A reset of the device will trigger a CLP disable/enable cycle on the host to bring the device into a recovered state. Signed-off-by: Farhan Ali <alifm@linux.ibm.com> --- hw/s390x/s390-pci-bus.c | 7 +++++++ hw/s390x/s390-pci-vfio-stubs.c | 4 ++++ hw/s390x/s390-pci-vfio.c | 10 ++++++++++ include/hw/s390x/s390-pci-vfio.h | 1 + 4 files changed, 22 insertions(+) diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c index b2db7f3df4..283b5b62b0 100644 --- a/hw/s390x/s390-pci-bus.c +++ b/hw/s390x/s390-pci-bus.c @@ -1546,6 +1546,8 @@ static void s390_pci_device_reset(DeviceState *dev) return; case ZPCI_FS_STANDBY: break; + case ZPCI_FS_ERROR: + break; default: pbdev->fh &= ~FH_MASK_ENABLE; pbdev->state = ZPCI_FS_DISABLED; @@ -1558,6 +1560,11 @@ static void s390_pci_device_reset(DeviceState *dev) } else if (pbdev->summary_ind) { pci_dereg_irqs(pbdev); } + + if (pbdev->state == ZPCI_FS_ERROR) { + s390_pci_reset(pbdev); + } + if (pbdev->iommu->enabled) { pci_dereg_ioat(pbdev->iommu); } diff --git a/hw/s390x/s390-pci-vfio-stubs.c b/hw/s390x/s390-pci-vfio-stubs.c index 9fc84ca135..c68c612038 100644 --- a/hw/s390x/s390-pci-vfio-stubs.c +++ b/hw/s390x/s390-pci-vfio-stubs.c @@ -36,3 +36,7 @@ bool s390_pci_setup_err_handler(S390PCIBusDevice *pbdev, Error **errp) error_setg(errp, "VFIO not available, cannot setup error handler"); return false; } + +void s390_pci_reset(S390PCIBusDevice *pbdev) +{ +} diff --git a/hw/s390x/s390-pci-vfio.c b/hw/s390x/s390-pci-vfio.c index 9a63040e22..64eb28104b 100644 --- a/hw/s390x/s390-pci-vfio.c +++ b/hw/s390x/s390-pci-vfio.c @@ -10,6 +10,7 @@ */ #include "qemu/osdep.h" +#include "qemu/error-report.h" #include <sys/ioctl.h> #include <linux/vfio.h> @@ -172,6 +173,15 @@ static bool s390_pci_err_handler(VFIOPCIDevice *vfio_pci, Error **errp) return success; } +void s390_pci_reset(S390PCIBusDevice *pbdev) +{ + VFIOPCIDevice *vfio_pci = VFIO_PCI_DEVICE(pbdev->pdev); + if (ioctl(vfio_pci->vbasedev.fd, VFIO_DEVICE_RESET)) { + error_report("Failed to reset PCI device %s : %s ", + vfio_pci->vbasedev.name, strerror(errno)); + } +} + static void s390_pci_read_base(S390PCIBusDevice *pbdev, struct vfio_device_info *info) { diff --git a/include/hw/s390x/s390-pci-vfio.h b/include/hw/s390x/s390-pci-vfio.h index c7886b63ea..38ccf445ea 100644 --- a/include/hw/s390x/s390-pci-vfio.h +++ b/include/hw/s390x/s390-pci-vfio.h @@ -21,5 +21,6 @@ void s390_pci_end_dma_count(S390pciState *s, S390PCIDMACount *cnt); bool s390_pci_get_host_fh(S390PCIBusDevice *pbdev, uint32_t *fh); void s390_pci_get_clp_info(S390PCIBusDevice *pbdev); bool s390_pci_setup_err_handler(S390PCIBusDevice *pbdev, Error **errp); +void s390_pci_reset(S390PCIBusDevice *pbdev); #endif -- 2.43.0 ^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [PATCH v4 4/4] s390x/pci: Reset a device in error state 2026-08-31 18:31 ` [PATCH v4 4/4] s390x/pci: Reset a device in error state Farhan Ali @ 2026-09-02 8:27 ` Cédric Le Goater 2026-09-02 16:48 ` Farhan Ali 0 siblings, 1 reply; 21+ messages in thread From: Cédric Le Goater @ 2026-09-02 8:27 UTC (permalink / raw) To: Farhan Ali, qemu-devel, qemu-s390x; +Cc: mjrosato, farman, cohuck, alex, armbru On 8/31/26 20:31, Farhan Ali wrote: > For passthrough devices in error state, for a guest driven reset of the > device we can attempt a reset to recover the device. A reset of the device > will trigger a CLP disable/enable cycle on the host to bring the device > into a recovered state. > > Signed-off-by: Farhan Ali <alifm@linux.ibm.com> > --- > hw/s390x/s390-pci-bus.c | 7 +++++++ > hw/s390x/s390-pci-vfio-stubs.c | 4 ++++ > hw/s390x/s390-pci-vfio.c | 10 ++++++++++ > include/hw/s390x/s390-pci-vfio.h | 1 + > 4 files changed, 22 insertions(+) > > diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c > index b2db7f3df4..283b5b62b0 100644 > --- a/hw/s390x/s390-pci-bus.c > +++ b/hw/s390x/s390-pci-bus.c > @@ -1546,6 +1546,8 @@ static void s390_pci_device_reset(DeviceState *dev) > return; > case ZPCI_FS_STANDBY: > break; > + case ZPCI_FS_ERROR: > + break; > default: > pbdev->fh &= ~FH_MASK_ENABLE; > pbdev->state = ZPCI_FS_DISABLED; > @@ -1558,6 +1560,11 @@ static void s390_pci_device_reset(DeviceState *dev) > } else if (pbdev->summary_ind) { > pci_dereg_irqs(pbdev); > } > + > + if (pbdev->state == ZPCI_FS_ERROR) { > + s390_pci_reset(pbdev); > + } > + > if (pbdev->iommu->enabled) { > pci_dereg_ioat(pbdev->iommu); > } > diff --git a/hw/s390x/s390-pci-vfio-stubs.c b/hw/s390x/s390-pci-vfio-stubs.c > index 9fc84ca135..c68c612038 100644 > --- a/hw/s390x/s390-pci-vfio-stubs.c > +++ b/hw/s390x/s390-pci-vfio-stubs.c > @@ -36,3 +36,7 @@ bool s390_pci_setup_err_handler(S390PCIBusDevice *pbdev, Error **errp) > error_setg(errp, "VFIO not available, cannot setup error handler"); > return false; > } > + > +void s390_pci_reset(S390PCIBusDevice *pbdev) > +{ > +} > diff --git a/hw/s390x/s390-pci-vfio.c b/hw/s390x/s390-pci-vfio.c > index 9a63040e22..64eb28104b 100644 > --- a/hw/s390x/s390-pci-vfio.c > +++ b/hw/s390x/s390-pci-vfio.c > @@ -10,6 +10,7 @@ > */ > > #include "qemu/osdep.h" > +#include "qemu/error-report.h" > > #include <sys/ioctl.h> > #include <linux/vfio.h> > @@ -172,6 +173,15 @@ static bool s390_pci_err_handler(VFIOPCIDevice *vfio_pci, Error **errp) > return success; > } > > +void s390_pci_reset(S390PCIBusDevice *pbdev) > +{ > + VFIOPCIDevice *vfio_pci = VFIO_PCI_DEVICE(pbdev->pdev); > + if (ioctl(vfio_pci->vbasedev.fd, VFIO_DEVICE_RESET)) { > + error_report("Failed to reset PCI device %s : %s ", > + vfio_pci->vbasedev.name, strerror(errno)); > + } and pbdev->state remains ZPCI_FS_ERROR ? I don't see it cleared. C. > +} > + > static void s390_pci_read_base(S390PCIBusDevice *pbdev, > struct vfio_device_info *info) > { > diff --git a/include/hw/s390x/s390-pci-vfio.h b/include/hw/s390x/s390-pci-vfio.h > index c7886b63ea..38ccf445ea 100644 > --- a/include/hw/s390x/s390-pci-vfio.h > +++ b/include/hw/s390x/s390-pci-vfio.h > @@ -21,5 +21,6 @@ void s390_pci_end_dma_count(S390pciState *s, S390PCIDMACount *cnt); > bool s390_pci_get_host_fh(S390PCIBusDevice *pbdev, uint32_t *fh); > void s390_pci_get_clp_info(S390PCIBusDevice *pbdev); > bool s390_pci_setup_err_handler(S390PCIBusDevice *pbdev, Error **errp); > +void s390_pci_reset(S390PCIBusDevice *pbdev); > > #endif ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v4 4/4] s390x/pci: Reset a device in error state 2026-09-02 8:27 ` Cédric Le Goater @ 2026-09-02 16:48 ` Farhan Ali 0 siblings, 0 replies; 21+ messages in thread From: Farhan Ali @ 2026-09-02 16:48 UTC (permalink / raw) To: Cédric Le Goater, qemu-devel, qemu-s390x Cc: mjrosato, farman, cohuck, alex, armbru On 9/2/2026 1:27 AM, Cédric Le Goater wrote: > On 8/31/26 20:31, Farhan Ali wrote: >> For passthrough devices in error state, for a guest driven reset of the >> device we can attempt a reset to recover the device. A reset of the >> device >> will trigger a CLP disable/enable cycle on the host to bring the device >> into a recovered state. >> >> Signed-off-by: Farhan Ali <alifm@linux.ibm.com> >> --- >> hw/s390x/s390-pci-bus.c | 7 +++++++ >> hw/s390x/s390-pci-vfio-stubs.c | 4 ++++ >> hw/s390x/s390-pci-vfio.c | 10 ++++++++++ >> include/hw/s390x/s390-pci-vfio.h | 1 + >> 4 files changed, 22 insertions(+) >> >> diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c >> index b2db7f3df4..283b5b62b0 100644 >> --- a/hw/s390x/s390-pci-bus.c >> +++ b/hw/s390x/s390-pci-bus.c >> @@ -1546,6 +1546,8 @@ static void s390_pci_device_reset(DeviceState >> *dev) >> return; >> case ZPCI_FS_STANDBY: >> break; >> + case ZPCI_FS_ERROR: >> + break; >> default: >> pbdev->fh &= ~FH_MASK_ENABLE; >> pbdev->state = ZPCI_FS_DISABLED; >> @@ -1558,6 +1560,11 @@ static void s390_pci_device_reset(DeviceState >> *dev) >> } else if (pbdev->summary_ind) { >> pci_dereg_irqs(pbdev); >> } >> + >> + if (pbdev->state == ZPCI_FS_ERROR) { >> + s390_pci_reset(pbdev); >> + } >> + >> if (pbdev->iommu->enabled) { >> pci_dereg_ioat(pbdev->iommu); >> } >> diff --git a/hw/s390x/s390-pci-vfio-stubs.c >> b/hw/s390x/s390-pci-vfio-stubs.c >> index 9fc84ca135..c68c612038 100644 >> --- a/hw/s390x/s390-pci-vfio-stubs.c >> +++ b/hw/s390x/s390-pci-vfio-stubs.c >> @@ -36,3 +36,7 @@ bool s390_pci_setup_err_handler(S390PCIBusDevice >> *pbdev, Error **errp) >> error_setg(errp, "VFIO not available, cannot setup error >> handler"); >> return false; >> } >> + >> +void s390_pci_reset(S390PCIBusDevice *pbdev) >> +{ >> +} >> diff --git a/hw/s390x/s390-pci-vfio.c b/hw/s390x/s390-pci-vfio.c >> index 9a63040e22..64eb28104b 100644 >> --- a/hw/s390x/s390-pci-vfio.c >> +++ b/hw/s390x/s390-pci-vfio.c >> @@ -10,6 +10,7 @@ >> */ >> #include "qemu/osdep.h" >> +#include "qemu/error-report.h" >> #include <sys/ioctl.h> >> #include <linux/vfio.h> >> @@ -172,6 +173,15 @@ static bool s390_pci_err_handler(VFIOPCIDevice >> *vfio_pci, Error **errp) >> return success; >> } >> +void s390_pci_reset(S390PCIBusDevice *pbdev) >> +{ >> + VFIOPCIDevice *vfio_pci = VFIO_PCI_DEVICE(pbdev->pdev); >> + if (ioctl(vfio_pci->vbasedev.fd, VFIO_DEVICE_RESET)) { >> + error_report("Failed to reset PCI device %s : %s ", >> + vfio_pci->vbasedev.name, strerror(errno)); >> + } > > and pbdev->state remains ZPCI_FS_ERROR ? I don't see it cleared. > > C. The reset is done as part of CLP disable and on a disable the state is set to ZPCI_FS_DISABLED [1]. The device enabled again when the guest issues a CLP enabled. [1] https://elixir.bootlin.com/qemu/v11.1.1/source/hw/s390x/s390-pci-inst.c#L271 Thanks Farhan > >> +} >> + >> static void s390_pci_read_base(S390PCIBusDevice *pbdev, >> struct vfio_device_info *info) >> { >> diff --git a/include/hw/s390x/s390-pci-vfio.h >> b/include/hw/s390x/s390-pci-vfio.h >> index c7886b63ea..38ccf445ea 100644 >> --- a/include/hw/s390x/s390-pci-vfio.h >> +++ b/include/hw/s390x/s390-pci-vfio.h >> @@ -21,5 +21,6 @@ void s390_pci_end_dma_count(S390pciState *s, >> S390PCIDMACount *cnt); >> bool s390_pci_get_host_fh(S390PCIBusDevice *pbdev, uint32_t *fh); >> void s390_pci_get_clp_info(S390PCIBusDevice *pbdev); >> bool s390_pci_setup_err_handler(S390PCIBusDevice *pbdev, Error >> **errp); >> +void s390_pci_reset(S390PCIBusDevice *pbdev); >> #endif > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v4 0/4] Error recovery for zPCI passthrough devices 2026-08-31 18:31 [PATCH v4 0/4] Error recovery for zPCI passthrough devices Farhan Ali ` (3 preceding siblings ...) 2026-08-31 18:31 ` [PATCH v4 4/4] s390x/pci: Reset a device in error state Farhan Ali @ 2026-09-03 9:19 ` Cédric Le Goater 2026-09-03 17:55 ` Farhan Ali 4 siblings, 1 reply; 21+ messages in thread From: Cédric Le Goater @ 2026-09-03 9:19 UTC (permalink / raw) To: Farhan Ali, qemu-devel, qemu-s390x Cc: mjrosato, farman, cohuck, alex, armbru, Shameer Kolothum On 8/31/26 20:31, Farhan Ali wrote: > Hi, > > This patch series introduces support for error recovery for passthrough PCI > devices on System Z (s390x). This is the user space component for the Linux > kernel patches [1]. The kernel patches were merged for 7.3 and it had some > significant changes in VFIO API since the last QEMU series was posted. But the > design for QEMU component is still the same. > > For QEMU on eventfd notification for PCI error from vfio-pci driver we call the > vfio error handler. We can use a per device error handler callback to override > the default vfio error handler. For s390x specific error handler, we retrieve > the architecture specific PCI error information and inject the information into > the guest. Once the guest receives the error information, the guest drivers will > drive the error recovery. Typically recovery involves a device reset which > translate to CLP disable/enable cycle for the device. > > I would appreciate some feedback on this patch series. Hi Farhan, Have you seen Shameer's kernel series for PCI error recovery [1] ? It uses a dedicated eventfd registered via VFIO_DEVICE_FEATURE_SET, with a self-contained handler independent of the legacy err_notifier. The QEMU side [2] just probes the feature, installs the eventfd, and suppresses the legacy vm_stop path with a bool flag : no callback in generic VFIO PCI code, which is cleaner. This is also the pattern used by the other s390 VFIO devices: vfio-ccw has three dedicated eventfds (io_notifier, crw_notifier, req_notifier) and vfio-ap has two (req_notifier, cfg_notifier). For zPCI, it would mean extending VFIO_DEVICE_FEATURE_ZPCI_ERROR to support SET with an eventfd field. The ABI is still new so the struct can be extended, keeping the s390 error recovery code self-contained, with no changes to generic VFIO PCI code. I think this approach is preferable. Thanks, C. [1] https://lore.kernel.org/all/20260901093217.8539-1-skolothumtho@nvidia.com/ [2] https://github.com/shamiali2008/qemu-master/commit/e45458710cd8509fa5bd81f4376126b77b6f40be ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v4 0/4] Error recovery for zPCI passthrough devices 2026-09-03 9:19 ` [PATCH v4 0/4] Error recovery for zPCI passthrough devices Cédric Le Goater @ 2026-09-03 17:55 ` Farhan Ali 2026-09-08 19:25 ` Farhan Ali 0 siblings, 1 reply; 21+ messages in thread From: Farhan Ali @ 2026-09-03 17:55 UTC (permalink / raw) To: Cédric Le Goater, qemu-devel, qemu-s390x Cc: mjrosato, farman, cohuck, alex, armbru, Shameer Kolothum On 9/3/2026 2:19 AM, Cédric Le Goater wrote: > On 8/31/26 20:31, Farhan Ali wrote: >> Hi, >> >> This patch series introduces support for error recovery for >> passthrough PCI >> devices on System Z (s390x). This is the user space component for the >> Linux >> kernel patches [1]. The kernel patches were merged for 7.3 and it had >> some >> significant changes in VFIO API since the last QEMU series was >> posted. But the >> design for QEMU component is still the same. >> >> For QEMU on eventfd notification for PCI error from vfio-pci driver >> we call the >> vfio error handler. We can use a per device error handler callback to >> override >> the default vfio error handler. For s390x specific error handler, we >> retrieve >> the architecture specific PCI error information and inject the >> information into >> the guest. Once the guest receives the error information, the guest >> drivers will >> drive the error recovery. Typically recovery involves a device reset >> which >> translate to CLP disable/enable cycle for the device. >> >> I would appreciate some feedback on this patch series. > Hi Cedric, > Hi Farhan, > > Have you seen Shameer's kernel series for PCI error recovery [1] ? > It uses a dedicated eventfd registered via VFIO_DEVICE_FEATURE_SET, > with a self-contained handler independent of the legacy err_notifier. > > The QEMU side [2] just probes the feature, installs the eventfd, and > suppresses the legacy vm_stop path with a bool flag : no callback in > generic VFIO PCI code, which is cleaner. > > This is also the pattern used by the other s390 VFIO devices: vfio-ccw > has three dedicated eventfds (io_notifier, crw_notifier, req_notifier) > and vfio-ap has two (req_notifier, cfg_notifier). My thought process was to use the existing legacy err_notifier and let each vfio-pci device handle the error. I thought the callback would also provide some flexibility on each device handling the error. I can take a look at Shameer's approach and see if its something we can do. Just briefly looking at the QEMU code, it looks like it it ties the eventfd to the generic vfio-pci code (of course this is in flux). This would become problematic for s390x as unfortunately AER recovery will not work for zPCI devices due to platform firmware being involved in the recovery process. > For zPCI, it would mean extending VFIO_DEVICE_FEATURE_ZPCI_ERROR to > support SET with an eventfd field. The ABI is still new so the struct > can be extended, keeping the s390 error recovery code self-contained, > with no changes to generic VFIO PCI code. I think this approach is > preferable. > I would like to get some feedback from Alex to see if we could extend the ABI. Thanks Farhan > Thanks, > > C. > > [1] > https://lore.kernel.org/all/20260901093217.8539-1-skolothumtho@nvidia.com/ > [2] > https://github.com/shamiali2008/qemu-master/commit/e45458710cd8509fa5bd81f4376126b77b6f40be > > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v4 0/4] Error recovery for zPCI passthrough devices 2026-09-03 17:55 ` Farhan Ali @ 2026-09-08 19:25 ` Farhan Ali 0 siblings, 0 replies; 21+ messages in thread From: Farhan Ali @ 2026-09-08 19:25 UTC (permalink / raw) To: Cédric Le Goater, qemu-devel, qemu-s390x, Alex Williamson Cc: mjrosato, farman, cohuck, armbru, Shameer Kolothum On 9/3/2026 10:55 AM, Farhan Ali wrote: > > On 9/3/2026 2:19 AM, Cédric Le Goater wrote: >> On 8/31/26 20:31, Farhan Ali wrote: >>> Hi, >>> >>> This patch series introduces support for error recovery for >>> passthrough PCI >>> devices on System Z (s390x). This is the user space component for >>> the Linux >>> kernel patches [1]. The kernel patches were merged for 7.3 and it >>> had some >>> significant changes in VFIO API since the last QEMU series was >>> posted. But the >>> design for QEMU component is still the same. >>> >>> For QEMU on eventfd notification for PCI error from vfio-pci driver >>> we call the >>> vfio error handler. We can use a per device error handler callback >>> to override >>> the default vfio error handler. For s390x specific error handler, >>> we retrieve >>> the architecture specific PCI error information and inject the >>> information into >>> the guest. Once the guest receives the error information, the guest >>> drivers will >>> drive the error recovery. Typically recovery involves a device >>> reset which >>> translate to CLP disable/enable cycle for the device. >>> >>> I would appreciate some feedback on this patch series. >> > Hi Cedric, > >> Hi Farhan, >> >> Have you seen Shameer's kernel series for PCI error recovery [1] ? >> It uses a dedicated eventfd registered via VFIO_DEVICE_FEATURE_SET, >> with a self-contained handler independent of the legacy err_notifier. >> >> The QEMU side [2] just probes the feature, installs the eventfd, and >> suppresses the legacy vm_stop path with a bool flag : no callback in >> generic VFIO PCI code, which is cleaner. >> >> This is also the pattern used by the other s390 VFIO devices: vfio-ccw >> has three dedicated eventfds (io_notifier, crw_notifier, req_notifier) >> and vfio-ap has two (req_notifier, cfg_notifier). > > My thought process was to use the existing legacy err_notifier and let > each vfio-pci device handle the error. I thought the callback would > also provide some flexibility on each device handling the error. > > I can take a look at Shameer's approach and see if its something we > can do. Just briefly looking at the QEMU code, it looks like it it > ties the eventfd to the generic vfio-pci code (of course this is in > flux). This would become problematic for s390x as unfortunately AER > recovery will not work for zPCI devices due to platform firmware being > involved in the recovery process. > >> For zPCI, it would mean extending VFIO_DEVICE_FEATURE_ZPCI_ERROR to >> support SET with an eventfd field. The ABI is still new so the struct >> can be extended, keeping the s390 error recovery code self-contained, >> with no changes to generic VFIO PCI code. I think this approach is >> preferable. >> > I would like to get some feedback from Alex to see if we could extend > the ABI. > > I had been thinking about this a bit more, and I think having a another eventfd for s390 to just notify on error would just be unnecessary duplication. Because now we will have the legacy err_notifier + s390 error notifer eventfds for the same purpose but with different handlers. Can we replace fd handler for the legacy err_notifier from vfio_err_notifier_handler() to an s390 specific handler for s390 devices? This would also avoid any callback in the vfio core common code. Thanks Farhan > > >> Thanks, >> >> C. >> >> [1] >> https://lore.kernel.org/all/20260901093217.8539-1-skolothumtho@nvidia.com/ >> [2] >> https://github.com/shamiali2008/qemu-master/commit/e45458710cd8509fa5bd81f4376126b77b6f40be >> >> > ^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2026-09-08 19:26 UTC | newest] Thread overview: 21+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-08-31 18:31 [PATCH v4 0/4] Error recovery for zPCI passthrough devices Farhan Ali 2026-08-31 18:31 ` [PATCH v4 1/4] linux-headers: Update Linux header to 7.3-rc1 Farhan Ali 2026-08-31 18:31 ` [PATCH v4 2/4] vfio/pci: Add an error handler callback Farhan Ali 2026-09-02 8:07 ` Cédric Le Goater 2026-09-02 8:21 ` Shameer Kolothum Thodi 2026-09-02 10:39 ` Markus Armbruster 2026-09-02 16:23 ` Farhan Ali 2026-09-03 10:24 ` Markus Armbruster 2026-09-02 16:34 ` Farhan Ali 2026-09-03 12:08 ` Shameer Kolothum Thodi 2026-09-03 18:07 ` Farhan Ali 2026-09-04 9:58 ` Shameer Kolothum Thodi 2026-08-31 18:31 ` [PATCH v4 3/4] s390x/pci: Add PCI error handling for vfio pci devices Farhan Ali 2026-09-02 8:22 ` Cédric Le Goater 2026-09-02 16:43 ` Farhan Ali 2026-08-31 18:31 ` [PATCH v4 4/4] s390x/pci: Reset a device in error state Farhan Ali 2026-09-02 8:27 ` Cédric Le Goater 2026-09-02 16:48 ` Farhan Ali 2026-09-03 9:19 ` [PATCH v4 0/4] Error recovery for zPCI passthrough devices Cédric Le Goater 2026-09-03 17:55 ` Farhan Ali 2026-09-08 19:25 ` Farhan Ali
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.