Re: [PATCH v2] tests/functional: test device passthrough on aarch64

[Pierrick Bouvier <pierrick.bouvier@linaro.org>]

On 7/2/25 5:51 AM, Cédric Le Goater wrote:
> Hello,
> 
> On 6/27/25 22:02, Pierrick Bouvier wrote:
>> This test allows to document and exercise device passthrough, using a
>> nested virtual machine setup. Two disks are generated and passed to the
>> VM, and their content is compared to original images.
>>
>> Guest and nested guests commands are executed through two scripts, and
>> init used in both system is configured to trigger a kernel panic in case
>> any command fails. This is more reliable and readable than executing all
>> commands through prompt injection and trying to guess what failed.
>>
>> Initially, this test was supposed to test smmuv3 nested emulation
>> (combining both stages of translation), but I could not find any setup
>> (kernel + vmm) able to do the passthrough correctly, despite several
>> tries.
>>
>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> ---
>>    tests/functional/meson.build                  |   2 +
>>    .../test_aarch64_device_passthrough.py        | 142 ++++++++++++++++++
>>    2 files changed, 144 insertions(+)
>>    create mode 100755 tests/functional/test_aarch64_device_passthrough.py
>>
>> diff --git a/tests/functional/meson.build b/tests/functional/meson.build
>> index 3021928a9d4..6cc78abb123 100644
>> --- a/tests/functional/meson.build
>> +++ b/tests/functional/meson.build
>> @@ -13,6 +13,7 @@ endif
>>    test_timeouts = {
>>      'aarch64_aspeed_ast2700' : 600,
>>      'aarch64_aspeed_ast2700fc' : 600,
>> +  'aarch64_device_passthrough' : 720,
>>      'aarch64_imx8mp_evk' : 240,
>>      'aarch64_raspi4' : 480,
>>      'aarch64_reverse_debug' : 180,
>> @@ -84,6 +85,7 @@ tests_aarch64_system_quick = [
>>    tests_aarch64_system_thorough = [
>>      'aarch64_aspeed_ast2700',
>>      'aarch64_aspeed_ast2700fc',
>> +  'aarch64_device_passthrough',
>>      'aarch64_imx8mp_evk',
>>      'aarch64_raspi3',
>>      'aarch64_raspi4',
>> diff --git a/tests/functional/test_aarch64_device_passthrough.py b/tests/functional/test_aarch64_device_passthrough.py
>> new file mode 100755
>> index 00000000000..1f3f158a9ff
>> --- /dev/null
>> +++ b/tests/functional/test_aarch64_device_passthrough.py
>> @@ -0,0 +1,142 @@
>> +#!/usr/bin/env python3
>> +#
>> +# Boots a nested guest and compare content of a device (passthrough) to a
>> +# reference image. Both vfio group and iommufd passthrough methods are tested.
>> +#
>> +# Copyright (c) 2025 Linaro Ltd.
>> +#
>> +# Author: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-or-later
>> +
>> +import os
>> +
>> +from qemu_test import QemuSystemTest, Asset
>> +from qemu_test import exec_command, wait_for_console_pattern
>> +from qemu_test import exec_command_and_wait_for_pattern
>> +from random import randbytes
>> +
>> +guest_script = '''
>> +#!/usr/bin/env bash
>> +
>> +set -euo pipefail
>> +set -x
>> +
>> +# find disks from nvme serial
>> +dev_vfio=$(lsblk --nvme | grep vfio | cut -f 1 -d ' ')
>> +dev_iommufd=$(lsblk --nvme | grep iommufd | cut -f 1 -d ' ')
>> +pci_vfio=$(basename $(readlink -f /sys/block/$dev_vfio/../../../))
>> +pci_iommufd=$(basename $(readlink -f /sys/block/$dev_iommufd/../../../))
>> +
>> +# bind disks to vfio
>> +for p in "$pci_vfio" "$pci_iommufd"; do
>> +    if [ "$(cat /sys/bus/pci/devices/$p/driver_override)" == vfio-pci ]; then
>> +        continue
>> +    fi
>> +    echo $p > /sys/bus/pci/drivers/nvme/unbind
>> +    echo vfio-pci > /sys/bus/pci/devices/$p/driver_override
>> +    echo $p > /sys/bus/pci/drivers/vfio-pci/bind
>> +done
>> +
>> +# boot nested guest and execute /host/nested_guest.sh
>> +# one disk is passed through vfio group, the other, through iommufd
>> +qemu-system-aarch64 \
> 
> Is this binary on the host.ext4 image ?
> 

Yes, it is the "vanilla" QEMU package from debian:trixie.
It would be better to cross compile current QEMU revision, and run that, 
but it's not something easily feasible with current CI architecture.

> If so, the test is testing a chosen QEMU version compiled in the
> L1 guest image but not the current QEMU version, which is the one
> running in the L0.
> 

Yes, this limits the scope of the test, I agree.

> Anyhow this is a very nice test and an excellent base to build on.
> As a next step, I’d suggest including tests with NICs using igb
> devices and igb virtual functions (VFs).
>

Sure, it can easily be extended with any scenario.
The key point (where I would like to get review from tests maintainers) 
is to know if using embedded scripts is ok, vs sequential interaction 
with stdio.

> It would also be great to run the L1 environment using the current
> version of QEMU. I haven't found a clean way to achieve that yet :/
>

Any scenario where we start to trigger a compilation will be painfully 
slow unfortunately.
It's clearly a limit of the testing infrastructure we have, and it 
prevents having more nested scenarios.

> Thanks,
> 
> C.
>