Re: [PATCH v3 0/5] powerpc/eeh: Add RTAS-based error injection support on pSeries

[Narayana Murty N <nnmlinux@linux.ibm.com>]

Hi Anushree,

Thanks for testing the series.

On 18/08/26 4:31 PM, Anushree Mathur wrote:
> 
> 
> On 21/07/26 9:08 AM, Narayana Murty N wrote:
>> The pSeries EEH error-injection backend currently implements a limited
>> software-generated MMIO failure and does not use the error-injection
>> services provided by RTAS.
>>
>> This series replaces that implementation with the PAPR-defined RTAS
>> workflow based on:
>>
>>    - ibm,open-errinjct
>>    - ibm,errinjct
>>    - ibm,close-errinjct
>>
>> The implementation opens an RTAS error-injection session, prepares the
>> firmware work buffer, performs the requested injection, and closes the
>> session on both success and failure paths.
>>
>> The existing EEH userspace ABI is preserved. EEH_ERR_TYPE_32 and
>> EEH_ERR_TYPE_64 continue to represent generic 32-bit and 64-bit IOA
>> bus-error injection requests. The pSeries backend maps these values to
>> the corresponding RTAS error types, while the PowerNV backend
>> explicitly maps them to the corresponding OPAL types.
>>
>> Additional generic EEH error types (EEH_ERR_TYPE_RECOVERED_SPECIAL_EVENT,
>> EEH_ERR_TYPE_CORRUPTED_PAGE, and the cache/TLB corruption types) are now
>> defined in the UAPI header and mapped explicitly to RTAS firmware 
>> encodings
>> by the pSeries backend. Platform backends that do not support a valid
>> generic type return -EOPNOTSUPP.
>>
>> No existing userspace ABI values are changed.
>>
>> The current injection path can be exercised for VFIO-assigned devices
>> through VFIO_EEH_PE_INJECT_ERR. The guest or userspace VFIO application
>> continues to use the same generic EEH type and function values,
>> independent of whether the host platform uses RTAS or OPAL.
>>
>> The series also handles the unusual return format of
>> ibm,open-errinjct:
>>
>> rets[0] = error-injection session token
>> rets[1] = RTAS status
>>
>> rtas_call() now returns rets[1] as the status and places the session
>> token in outputs[0], preserving the normal kernel rtas_call()
>> convention.
>>
>> sys_rtas() is intentionally unchanged because it exposes the raw RTAS
>> return cells to userspace. Userspace therefore continues to receive
>> the session token and status in their PAPR-defined positions.
>>
>> The RTAS work buffer is allocated during RTAS initialization below:
>>
>> min(ppc64_rma_size, RTAS_INSTANTIATE_MAX)
>>
>> using the same accessible-memory limit used for rtas_rmo_buf. The
>> kernel populates the buffer through its virtual mapping but passes its
>> physical address to firmware.
>>
>> The complete open, inject and close sequence is serialized with a
>> mutex. RTAS busy and extended-delay return values are handled for all
>> three calls. A session token value of zero is accepted, and session
>> state is tracked independently from the token value.
>>
>> The patches are organised as follows:
>>
>> Handle the special ibm,open-errinjct return format in rtas_call().
>> Allocate an RTAS-accessible error-injection work buffer.
>> Add pSeries RTAS parameter validation and buffer encoding helpers.
>> Implement RTAS-based pSeries EEH error injection.
>> Explicitly map generic EEH error types to OPAL types on PowerNV.
>>
>> Testing was performed on PowerVM with firmware providing the RTAS
>> error-injection services and with the corresponding QEMU support:
>>
>> https://lore.kernel.org/qemu-devel/20260520095446.64206-1- 
>> nnmlinux@linux.ibm.com/
>>
>> Signed-off-by: Narayana Murty N <nnmlinux@linux.ibm.com>
>>
>> Narayana Murty N (5):
>>    powerpc/rtas: Handle ibm,open-errinjct return format
>>    powerpc/rtas: Allocate ibm,errinjct buffer below RTAS limit
>>    powerpc/pseries/eeh: Add RTAS error validation helpers
>>    powerpc/pseries/eeh: Implement RTAS-based EEH error injection
>>    powerpc/powernv/eeh: Map VFIO EEH error injection to OPAL
>>
>>   arch/powerpc/include/asm/rtas.h              |  26 ++
>>   arch/powerpc/include/uapi/asm/eeh.h          |  24 +-
>>   arch/powerpc/kernel/rtas.c                   |  68 +++-
>>   arch/powerpc/platforms/powernv/eeh-powernv.c |  36 +-
>>   arch/powerpc/platforms/pseries/eeh_pseries.c | 367 +++++++++++++++++--
>>   5 files changed, 483 insertions(+), 38 deletions(-)
>>
>> Change Log:
>> V2 -> V3:
>>   * Fixed ibm,open-errinjct return handling to correctly process 
>> firmware responses.
>>   * Allocate the error-injection buffer in RTAS-accessible memory 
>> instead of general kernel memory.
>>   * Pass the physical address of the error-injection buffer to 
>> firmware (previously incorrect address type).
>>   * Accept session token zero as a valid token (previously rejected 
>> erroneously).
>>   * Handle RTAS busy and extended-delay return codes for open-inject, 
>> and close calls.
>>   * Simplified the validation helper — reduced complexity and removed 
>> redundant checks.
>>   * Simplified the buffer-preparation helper for cleaner, more 
>> maintainable code.
>>   * Validate that all required RTAS tokens are present before 
>> attempting to open a session.
>>   * Added explicit generic EEH-to-OPAL error-type mapping for the 
>> PowerNV platform.
>> v1 -> v2: https://lore.kernel.org/all/20260527072433.94510-1- 
>> nnmlinux@linux.ibm.com/
>>   * Addressed all review comments from Sourabh Jain
>>     - Removed unnecessary empty line in rtas_call()
>>     - Enhanced comment to explain PAPR specification requirements
>>     - Corrected misleading comment about output handling
>>     - Improved else block comment for better code clarity
>>   * Fixed kernel test robot warnings
>>     - Fixed kernel-doc warning for __maybe_unused parameter
>>     - Confirmed sparse warnings are false positives (correct 
>> endianness handling)
>>   * Added PowerNV platform abstraction layer (new Patch 5)
>>     - Maps EEH error types to OPAL-specific types
>>     - Simplifies type handling by direct variable update
>>   * Improved code comments and documentation throughout
>>   * Added Reported-by tags for kernel test robot findings
>>   * Split into logical 5-patch series for better review
>>
>> RFC -> v1: https://lore.kernel.org/all/20251205094510.4671-1- 
>> nnmlinux@linux.ibm.com/
>>   * Initial 4-patch series
>>   * Fixed PAPR ibm,open-errinjct output format (token,status order)
>>   * Added pr_fmt handling for EEH subsystem compatibility
>>   * Implemented comprehensive validation helpers
>>
>> RFC: https://lore.kernel.org/all/20251107091009.43034-1- 
>> nnmlinux@linux.ibm.com/
>>   * Initial RFC implementation
> Hi Narayana,
> I have tested this patch along with the qemu patch series (https:// 
> lore.kernel.org/all/20260520095446.64206-1-nnmlinux@linux.ibm.com/)
> and I am still seeing the issue in which I am not able to trigger EEH on 
> NVME Backplane device after passthroughing it to guest.
> I tried using location code with and without mask and address and EEH is 
> not getting triggered.
> 
> 1st command :
> 
> errinjct ioa-bus-error-64  -f 6 -p U2E4A.ND0.WZS0211-P1-C0  -a 
> 0x88020000 -m 0xffffffffffff8000 -k 1
> 
> 2nd command :
> 
> errinjct ioa-bus-error-64  -f 6 -p U2E4A.ND0.WZS0211-P1-C0 -m 0 -k 1
> 
> Getting the following issue everytime
> 
> errinjct: Could not open file /proc/device-tree/ibm,loc-code, No such 
> file or directory
> 
> errinjct: Could not open file /proc/device-tree/ibm,platform-keystore/ 
> ibm,loc-code, No such file or directory
> 
> errinjct: Could not open file /proc/device-tree/memory@0/ibm,loc-code, 
> No such file or directory
> 
> Injecting an ioa-bus-error...
> errinjct: RTAS error injection failed!
> 
> errinjct: RTAS: ioa-bus-error: Argument error (-3)
> 
> This error may have occurred because error injection
> is disabled for this partition. Please check the
> FSP and ensure you have error injection enabled.
> 
> Please look into this.
> 
> Thank you,
> Anushree Mathur
> 
> 
The commands shown below appear to be running the errinjct tool on the 
host. That path uses the host RTAS interface through sys_rtas() and 
therefore does not exercise the VFIO/QEMU passthrough error-injection 
path added by this kernel and QEMU series.

The intention of these patches is to allow an EEH error-injection 
request originating from the guest for a PCI device assigned through 
vfio-pci to be propagated through QEMU/VFIO to the host EEH backend.

Could you please retry the test from inside the guest after passing the 
NVMe backplane device through with vfio-pci?

The expected path for this test is roughly:

guest errinjct
→ guest ibm,open-errinjct / ibm,errinjct / ibm,close-errinjct
→ QEMU RTAS emulation
→ VFIO_EEH_PE_INJECT_ERR
→ host EEH/RTAS error-injection backend

In contrast, invoking errinjct directly on the host goes through 
sys_rtas() and talks directly to the host firmware. The 
/proc/device-tree/.../ibm,loc-code lookup failures and the RTAS Argument 
error (-3) reported in your log are therefore from that host-side 
errinjct path and are not testing the guest VFIO passthrough support 
provided by this series.

Please try the same error injection from the guest against the 
passed-through NVMe device and share the guest errinjct output together 
with the corresponding guest, QEMU, and host logs if it still fails.

Thanks,
Narayana