Re: [PATCH v2] tests/functional: test device passthrough on aarch64

["Alex Bennée" <alex.bennee@linaro.org>]

Cédric Le Goater <clg@kaod.org> writes:

> Hello,
>
> On 6/27/25 22:02, Pierrick Bouvier wrote:
>> This test allows to document and exercise device passthrough, using a
>> nested virtual machine setup. Two disks are generated and passed to the
>> VM, and their content is compared to original images.
>> Guest and nested guests commands are executed through two scripts,
>> and
>> init used in both system is configured to trigger a kernel panic in case
>> any command fails. This is more reliable and readable than executing all
>> commands through prompt injection and trying to guess what failed.
>> Initially, this test was supposed to test smmuv3 nested emulation
>> (combining both stages of translation), but I could not find any setup
>> (kernel + vmm) able to do the passthrough correctly, despite several
>> tries.
>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> ---
>>   tests/functional/meson.build                  |   2 +
>>   .../test_aarch64_device_passthrough.py        | 142 ++++++++++++++++++
>>   2 files changed, 144 insertions(+)
>>   create mode 100755 tests/functional/test_aarch64_device_passthrough.py
>> diff --git a/tests/functional/meson.build
>> b/tests/functional/meson.build
>> index 3021928a9d4..6cc78abb123 100644
>> --- a/tests/functional/meson.build
>> +++ b/tests/functional/meson.build
>> @@ -13,6 +13,7 @@ endif
>>   test_timeouts = {
>>     'aarch64_aspeed_ast2700' : 600,
>>     'aarch64_aspeed_ast2700fc' : 600,
>> +  'aarch64_device_passthrough' : 720,
>>     'aarch64_imx8mp_evk' : 240,
>>     'aarch64_raspi4' : 480,
>>     'aarch64_reverse_debug' : 180,
>> @@ -84,6 +85,7 @@ tests_aarch64_system_quick = [
>>   tests_aarch64_system_thorough = [
>>     'aarch64_aspeed_ast2700',
>>     'aarch64_aspeed_ast2700fc',
>> +  'aarch64_device_passthrough',
>>     'aarch64_imx8mp_evk',
>>     'aarch64_raspi3',
>>     'aarch64_raspi4',
>> diff --git a/tests/functional/test_aarch64_device_passthrough.py b/tests/functional/test_aarch64_device_passthrough.py
>> new file mode 100755
>> index 00000000000..1f3f158a9ff
>> --- /dev/null
>> +++ b/tests/functional/test_aarch64_device_passthrough.py
>> @@ -0,0 +1,142 @@
>> +#!/usr/bin/env python3
>> +#
>> +# Boots a nested guest and compare content of a device (passthrough) to a
>> +# reference image. Both vfio group and iommufd passthrough methods are tested.
>> +#
>> +# Copyright (c) 2025 Linaro Ltd.
>> +#
>> +# Author: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-or-later
>> +
>> +import os
>> +
>> +from qemu_test import QemuSystemTest, Asset
>> +from qemu_test import exec_command, wait_for_console_pattern
>> +from qemu_test import exec_command_and_wait_for_pattern
>> +from random import randbytes
>> +
>> +guest_script = '''
>> +#!/usr/bin/env bash
>> +
>> +set -euo pipefail
>> +set -x
>> +
>> +# find disks from nvme serial
>> +dev_vfio=$(lsblk --nvme | grep vfio | cut -f 1 -d ' ')
>> +dev_iommufd=$(lsblk --nvme | grep iommufd | cut -f 1 -d ' ')
>> +pci_vfio=$(basename $(readlink -f /sys/block/$dev_vfio/../../../))
>> +pci_iommufd=$(basename $(readlink -f /sys/block/$dev_iommufd/../../../))
>> +
>> +# bind disks to vfio
>> +for p in "$pci_vfio" "$pci_iommufd"; do
>> +    if [ "$(cat /sys/bus/pci/devices/$p/driver_override)" == vfio-pci ]; then
>> +        continue
>> +    fi
>> +    echo $p > /sys/bus/pci/drivers/nvme/unbind
>> +    echo vfio-pci > /sys/bus/pci/devices/$p/driver_override
>> +    echo $p > /sys/bus/pci/drivers/vfio-pci/bind
>> +done
>> +
>> +# boot nested guest and execute /host/nested_guest.sh
>> +# one disk is passed through vfio group, the other, through iommufd
>> +qemu-system-aarch64 \
>
> Is this binary on the host.ext4 image ?
>
> If so, the test is testing a chosen QEMU version compiled in the
> L1 guest image but not the current QEMU version, which is the one
> running in the L0.
>
> Anyhow this is a very nice test and an excellent base to build on.
> As a next step, I’d suggest including tests with NICs using igb
> devices and igb virtual functions (VFs).
>
> It would also be great to run the L1 environment using the current
> version of QEMU. I haven't found a clean way to achieve that yet :/

I sometimes boot up with a virtiofsd mapped to $HOME but it gets a
little unstable over time and I haven't had a chance to figure out where
things where going wrong.

We have the containers to reliably build a cross image of QEMU but we
would have to ensure the guest image matches so we don't run into
library skew issues. I have had a static build working but thats not a
very well supported configuration for qemu-system.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro