Is PCI passthrough of the AMD XDNA device possible?

Is PCI passthrough of the AMD XDNA device possible?

[Marcello Sylvester Bauer]

[-- Attachment #1.1.1: Type: text/plain, Size: 2470 bytes --]

Greetings,

I have some questions about making the AMD XDNA NPU accessible in a 
virtualized environment. I tried using VFIO on QEMU, but I could not get 
it to work.

Here is a brief rundown of the resulting issues:

Simply adding the device to QEMU ("-device 
vfio-pci,host=0000:c4:00.1,addr=4") results in the following error:
```
amdxdna 0000:c4:00.1: [drm] *ERROR* aie2_init: Enable PASID failed, ret -19
amdxdna 0000:c4:00.1: [drm] *ERROR* amdxdna_probe: Hardware init failed, 
ret -19
```
This makes sense, as the drive relies on the IOMMU for the PASID extension.

Fortunately, QEMU supports hardware-assisted IOMMU virtualization with 
the AMD-vIOMMU device ("-device amd-iommu,intremap=on").
While the driver can enable the PASID extension with this device, it 
fails to establish communication with PSP during firmware validation:
```
amdxdna 0000:00:04.1: [drm] *ERROR* fw return error 0x8
amdxdna 0000:00:04.1: [drm] *ERROR* failed to validate fw, ret -5
amdxdna 0000:00:04.1: [drm] *ERROR* aie2_hw_start: failed to start psp, 
ret -5
amdxdna 0000:00:04.1: [drm] *ERROR* aie2_smu_exec: smu cmd 4 timed out
amdxdna 0000:00:04.1: [drm] *ERROR* aie2_smu_fini: Power off failed, ret 
-110
amdxdna 0000:00:04.1: [drm] *ERROR* aie2_init: start npu failed, ret -5
amdxdna 0000:00:04.1: [drm] *ERROR* amdxdna_probe: Hardware init failed, 
ret -5
amdxdna 0000:00:04.1: probe with driver amdxdna failed with error -5
```

Note that it also causes other devices on the host system to 
malfunction, such as the GPU:
```
amdgpu 0000:c3:00.0: amdgpu: SMU: I'm not done with your previous 
command: SMN_C2PMSG_66:0x0000000F SMN_C2PMSG_82:0x00000007
amdgpu 0000:c3:00.0: amdgpu: SMU: I'm not done with your previous 
command: SMN_C2PMSG_66:0x0000000F SMN_C2PMSG_82:0x00000007
amdgpu 0000:c3:00.0: amdgpu: SMU: I'm not done with your previous 
command: SMN_C2PMSG_66:0x0000000F SMN_C2PMSG_82:0x00000007
amdgpu 0000:c3:00.0: amdgpu: Failed to disable gfxoff!
```

Now to the questions in this regard:

Are there any known limitations or information regarding the 
virtualization of AMD XNDA?
Is there public documentation on how the device interacts with the PSP 
BAR, and if not, what does firmware error 8 mean?

Thanks for the support.

Kind regards,
Marcello
--
hardware: Lenovo ThinkPad T14s Gen 6 (AMD Ryzen AI 7 PRO 360)
QEMU: v10.0.0
Linux host: v6.15.4
Linux guest: 6.14.0-22-generic (Ubuntu 25.04)



[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 10181 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Is PCI passthrough of the AMD XDNA device possible?

[Lizhi Hou]

On 7/2/25 02:14, Marcello Sylvester Bauer wrote:
> Greetings,
>
> I have some questions about making the AMD XDNA NPU accessible in a 
> virtualized environment. I tried using VFIO on QEMU, but I could not 
> get it to work.
>
> Here is a brief rundown of the resulting issues:
>
> Simply adding the device to QEMU ("-device 
> vfio-pci,host=0000:c4:00.1,addr=4") results in the following error:
> ```
> amdxdna 0000:c4:00.1: [drm] *ERROR* aie2_init: Enable PASID failed, 
> ret -19
> amdxdna 0000:c4:00.1: [drm] *ERROR* amdxdna_probe: Hardware init 
> failed, ret -19
> ```
> This makes sense, as the drive relies on the IOMMU for the PASID 
> extension.
>
> Fortunately, QEMU supports hardware-assisted IOMMU virtualization with 
> the AMD-vIOMMU device ("-device amd-iommu,intremap=on").
> While the driver can enable the PASID extension with this device, it 
> fails to establish communication with PSP during firmware validation:
> ```
> amdxdna 0000:00:04.1: [drm] *ERROR* fw return error 0x8
> amdxdna 0000:00:04.1: [drm] *ERROR* failed to validate fw, ret -5
> amdxdna 0000:00:04.1: [drm] *ERROR* aie2_hw_start: failed to start 
> psp, ret -5
> amdxdna 0000:00:04.1: [drm] *ERROR* aie2_smu_exec: smu cmd 4 timed out
> amdxdna 0000:00:04.1: [drm] *ERROR* aie2_smu_fini: Power off failed, 
> ret -110
> amdxdna 0000:00:04.1: [drm] *ERROR* aie2_init: start npu failed, ret -5
> amdxdna 0000:00:04.1: [drm] *ERROR* amdxdna_probe: Hardware init 
> failed, ret -5
> amdxdna 0000:00:04.1: probe with driver amdxdna failed with error -5
> ```
>
> Note that it also causes other devices on the host system to 
> malfunction, such as the GPU:
> ```
> amdgpu 0000:c3:00.0: amdgpu: SMU: I'm not done with your previous 
> command: SMN_C2PMSG_66:0x0000000F SMN_C2PMSG_82:0x00000007
> amdgpu 0000:c3:00.0: amdgpu: SMU: I'm not done with your previous 
> command: SMN_C2PMSG_66:0x0000000F SMN_C2PMSG_82:0x00000007
> amdgpu 0000:c3:00.0: amdgpu: SMU: I'm not done with your previous 
> command: SMN_C2PMSG_66:0x0000000F SMN_C2PMSG_82:0x00000007
> amdgpu 0000:c3:00.0: amdgpu: Failed to disable gfxoff!
> ```
>
> Now to the questions in this regard:
>
> Are there any known limitations or information regarding the 
> virtualization of AMD XNDA?
> Is there public documentation on how the device interacts with the PSP 
> BAR, and if not, what does firmware error 8 mean?

Unfortunately, XDNA hardware/firmware does not support passthrough. The 
validate fw error with passthrough setup is known. If you would use XDNA 
in VM, you may try virtio solution: https://github.com/amd/virtio-npu


Thanks,

Lizhi

>
> Thanks for the support.
>
> Kind regards,
> Marcello
> -- 
> hardware: Lenovo ThinkPad T14s Gen 6 (AMD Ryzen AI 7 PRO 360)
> QEMU: v10.0.0
> Linux host: v6.15.4
> Linux guest: 6.14.0-22-generic (Ubuntu 25.04)
>
>