Re: [RFC PATCH] tests/functional/ppc64: add pseries virtio-blk hotplug test

["Alex Bennée" <alex.bennee@linaro.org>]

Chinmay Rath <rathc@linux.ibm.com> writes:

> On 11/3/26 19:49, Anushree Mathur wrote:
>> Add a functional test that hot-plugs and hot-unplugs a virtio-blk-pci
>> disk on the pseries machine.  It runs with TCG, and also with KVM if
>> KVM is enabled.
>>
>> On pseries a PCI device added through QMP is only enumerated by the
>> guest once rtas_errd (ppc64-diag) handles the RTAS hotplug event and
>>
>> runs drmgr (powerpc-utils).  Neither is in the buildroot image used by
>> the other pseries tests, so the test uses rootfs-hotplug.ext2: the same
>> image with those two tools added.
>>
>> The test boots with a second PHB (phb1) present so that the guest
>> registers its slots, adds a null-co backend, hot-plugs virtio-blk-pci
>> into phb1.0 and waits for /dev/vda, then unplugs it and waits for both
>> the QMP DEVICE_DELETED event and /dev/vda to go away.  The root drive is
>> used with snapshot=on so the cached asset is not modified by the guest.
>>
>> Signed-off-by: Anushree Mathur <anushree.mathur@linux.ibm.com>
>> ---
>> Hi all,
>>
>> This adds a functional test for virtio-blk hotplug on the pseries
>> machine, with a TCG and a KVM variant.
>>
>> The test boots pseries with a second PCI host bridge (phb1) present at
>> boot, hot-plugs a virtio-blk-pci device into it over QMP, and waits for
>> /dev/vda to show up in the guest.  It then unplugs the device and waits
>> for the DEVICE_DELETED event and for /dev/vda to disappear again.
>>
>> On pseries the guest does not enumerate a hot-plugged PCI device by
>> itself: rtas_errd (ppc64-diag) has to handle the RTAS hotplug event and
>> run drmgr (powerpc-utils).  The buildroot image the other pseries tests
>> use has neither, so the test needs a rootfs that includes them.
>>
>> I am sending this as an RFC because that rootfs is not available yet.
>> It is a new file, rootfs-hotplug.ext2, which I proposed for the
>> qemu-ppc-boot repository next to the existing pseries images:
>>
>>    https://github.com/legoater/qemu-ppc-boot/pull/5

I can't see the contents of this PR but I was wondering to the tuxrun
kernels have what you need?

>
> Hi Cedric,
>
> I heard from Anushree that you do not plan to accept PRs in the above
> repository going forward.
>
> If you're okay with it, I'd like to fork your repository and maintain
> it for future PPC test development and usage.
> Please let me know if that works for you.
>
> Thanks,
> Chinmay
>
>>
>> Until that is merged, the Asset URL in the test returns 404.  The sha256
>> in the test matches the image in that pull request.  The image is the
>> existing 2023.11 ppc64le pseries rootfs with rtas_errd, drmgr and their
>> libraries added, built with the legoater/buildroot qemu-ppc toolchain.
>>
>> Tested on a ppc64le POWER host with both TCG and KVM; checkpatch.pl and
>> pylint are clean.
>>
>>   MAINTAINERS                                |   1 +
>>   tests/functional/ppc64/meson.build         |   2 +
>>   tests/functional/ppc64/test_hotplug_blk.py | 188 +++++++++++++++++++++
>>   3 files changed, 191 insertions(+)
>>   create mode 100644 tests/functional/ppc64/test_hotplug_blk.py
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 3b12ba90f2..806d8e6ce6 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -1704,6 +1704,7 @@ F: tests/qtest/libqos/rtas*
>>   F: tests/functional/ppc64/test_pseries.py
>>   F: tests/functional/ppc64/test_hv.py
>>   F: tests/functional/ppc64/test_tuxrun.py
>> +F: tests/functional/ppc64/test_hotplug_blk.py
>>     PowerNV (Non-Virtualized)
>>   M: Nicholas Piggin <npiggin@gmail.com>
>> diff --git a/tests/functional/ppc64/meson.build b/tests/functional/ppc64/meson.build
>> index cb3c745624..e1e8211dcb 100644
>> --- a/tests/functional/ppc64/meson.build
>> +++ b/tests/functional/ppc64/meson.build
>> @@ -2,6 +2,7 @@
>>     test_ppc64_timeouts = {
>>     'fadump' : 480,
>> +  'hotplug_blk' : 600,
>>     'hv' : 1000,
>>     'mac99' : 120,
>>     'openbsd' : 240,
>> @@ -19,6 +20,7 @@ tests_ppc64_system_quick = [
>>   tests_ppc64_system_thorough = [
>>     'e500',
>>     'fadump',
>> +  'hotplug_blk',
>>     'hv',
>>     'mac99',
>>     'openbsd',
>> diff --git a/tests/functional/ppc64/test_hotplug_blk.py b/tests/functional/ppc64/test_hotplug_blk.py
>> new file mode 100644
>> index 0000000000..2737e4b486
>> --- /dev/null
>> +++ b/tests/functional/ppc64/test_hotplug_blk.py
>> @@ -0,0 +1,188 @@
>> +#!/usr/bin/env python3
>> +#
>> +# Functional test that hot-plugs and hot-unplugs a virtio-blk-pci disk
>> +# on the pseries machine type.
>> +#
>> +# Copyright (c) 2025 IBM Corporation
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-or-later
>> +
>> +import platform
>> +
>> +from qemu_test import LinuxKernelTest, Asset
>> +from qemu_test import exec_command_and_wait_for_pattern
>> +
>> +
>> +class PseriesHotplugBlk(LinuxKernelTest):
>> +    """
>> +    Test virtio-blk-pci hotplug/unplug on the pseries machine type.
>> +
>> +    A pre-built rootfs image (based on the buildroot image used by
>> +    test_hotplug_cpu.py) is used as the guest OS.  The image includes
>> +    rtas_errd from ppc64-diag, which is required on pseries to call the
>> +    ibm,configure-connector RTAS function that makes a QMP-hotplugged
>> +    PCI device visible to the kernel.
>> +
>> +    ppc64-diag is not packaged in Alpine Linux, so the Alpine+apk
>> +    approach used by test_hv.py cannot be used here.  Instead a
>> +    self-contained rootfs with rtas_errd and drmgr (from powerpc-utils)
>> +    already installed is referenced as an Asset, following the same
>> +    pattern as test_hotplug_cpu.py.
>> +
>> +    The test is run with both TCG and KVM; the KVM variant needs a
>> +    ppc64 host with kvm-hv loaded and is skipped automatically otherwise.
>> +
>> +    A second spapr-pci-host-bridge (phb1, index=1) is present at VM
>> +    launch so the guest kernel registers its PCI slots with rpaphp
>> +    during boot-time PCI probe.  Runtime DLPAR hotplug of
>> +    virtio-blk-pci into phb1.0 then works via the rtas_errd path.
>> +    """
>> +
>> +    timeout = 600
>> +
>> +    ASSET_KERNEL = Asset(
>> +        ('https://github.com/legoater/qemu-ppc-boot/raw/refs/heads/main/'
>> +         'buildroot/qemu_ppc64le_pseries-2023.11-8-gdcd9f0f6eb-20240104/'
>> +         'vmlinux'),
>> +        'fc20d5820ad83ed514ac5f88af7e2692d47ab7f36dab290cdb137d05b98502f5')
>> +
>> +    # Pre-built rootfs based on the buildroot pseries image above, with
>> +    # rtas_errd (from ppc64-diag) and drmgr (from powerpc-utils) added.
>> +    # rtas_errd is required to process RTAS hotplug events on pseries and
>> +    # it execs drmgr to configure the device; without them rpaphp
>> +    # registers the slots but the device is never enumerated by the kernel.
>> +    ASSET_ROOTFS = Asset(
>> +        ('https://github.com/legoater/qemu-ppc-boot/raw/refs/heads/main/'
>> +         'buildroot/qemu_ppc64le_pseries-2023.11-8-gdcd9f0f6eb-20240104/'
>> +         'rootfs-hotplug.ext2'),
>> +        '01ab3421d313e7765176eb27193fdb5837281dc3c8484eed140de8a3e65908eb')
>> +
>> +    def _boot_pseries(self, accel):
>> +        """Boot pseries with phb1 present and rtas_errd running.
>> +
>> +        accel must be 'tcg' or 'kvm'.
>> +        """
>> +        kernel_path = self.ASSET_KERNEL.fetch()
>> +        rootfs_path = self.ASSET_ROOTFS.fetch()
>> +
>> +        self.set_machine('pseries')
>> +        if accel == 'tcg':
>> +            self.vm.add_args('-accel', 'tcg,thread=multi')
>> +        else:
>> +            self.vm.add_args('-accel', 'kvm')
>> +        self.vm.add_args('-smp', '2', '-m', '512M')
>> +        # Second PHB present at boot so the guest kernel registers its
>> +        # slots with rpaphp.  Required for runtime DLPAR hotplug of
>> +        # virtio-blk-pci into phb1.0.
>> +        self.vm.add_args('-device', 'spapr-pci-host-bridge,id=phb1,index=1')
>> +        # snapshot=on keeps the cached rootfs asset unmodified
>> +        self.vm.add_args(
>> +            '-drive',
>> +            f'file={rootfs_path},if=scsi,index=0,format=raw,snapshot=on')
>> +        self.vm.add_args(
>> +            '-kernel', kernel_path,
>> +            '-append', 'root=/dev/sda console=hvc0 rootwait printk.time=0',
>> +        )
>> +        self.vm.set_console()
>> +        self.vm.launch()
>> +
>> +        self.wait_for_console_pattern('buildroot login:')
>> +        exec_command_and_wait_for_pattern(self, 'root', '#')
>> +
>> +        # Start rtas_errd so that RTAS hotplug events are processed.
>> +        # rtas_errd daemonizes by default (no -d flag).  Run mkdir and
>> +        # rtas_errd unconditionally then echo a sentinel so the pattern
>> +        # match never depends on their exit codes.
>> +        exec_command_and_wait_for_pattern(
>> +            self,
>> +            'touch /var/log/platform; mkdir -p /var/log/dump;'
>> +            ' /usr/sbin/rtas_errd;'
>> +            ' echo rtas_errd_started',
>> +            'rtas_errd_started',
>> +        )
>> +
>> +    def _do_blk_hotplug(self):
>> +
>> +        # Confirm /dev/vda does not exist before hotplug.
>> +        exec_command_and_wait_for_pattern(
>> +            self,
>> +            'test -b /dev/vda && echo present || echo absent',
>> +            'absent',
>> +        )
>> +
>> +        # Add a null-co block backend (no image file needed).
>> +        self.vm.cmd(
>> +            'blockdev-add',
>> +            **{
>> +                'driver': 'null-co',
>> +                'size': 1073741824,
>> +                'node-name': 'testdisk',
>> +            },
>> +        )
>> +
>> +        # Hotplug virtio-blk-pci into phb1.0.  The RTAS hotplug event
>> +        # (type 229) is delivered to the guest; rtas_errd calls
>> +        # ibm,configure-connector, the kernel enumerates the device and
>> +        # udev creates /dev/vda.
>> +        self.vm.cmd(
>> +            'device_add',
>> +            driver='virtio-blk-pci',
>> +            drive='testdisk',
>> +            id='hotblk0',
>> +            bus='phb1.0',
>> +            addr='0x0',
>> +        )
>> +
>> +        # Poll until /dev/vda appears.  Use a shell variable sentinel so
>> +        # the string 'vda_present' never appears in the typed command,
>> +        # avoiding a false match on the echoed command text.
>> +        exec_command_and_wait_for_pattern(
>> +            self,
>> +            'until test -b /dev/vda; do sleep 0.2; done'
>> +            '; R=present; echo vda_$R',
>> +            'vda_present',
>> +        )
>> +
>> +        # Unplug the device and wait for the QMP deletion event.
>> +        self.vm.cmd('device_del', id='hotblk0')
>> +        self.vm.event_wait(
>> +            'DEVICE_DELETED',
>> +            match={'data': {'device': 'hotblk0'}},
>> +        )
>> +
>> +        # Poll until /dev/vda disappears.
>> +        exec_command_and_wait_for_pattern(
>> +            self,
>> +            'while test -b /dev/vda; do sleep 0.2; done'
>> +            '; R=absent; echo vda_$R',
>> +            'vda_absent',
>> +        )
>> +
>> +    def test_pseries_hotplug_blk_tcg(self):
>> +        """Hotplug then unplug a virtio-blk-pci device on pseries (TCG)."""
>> +        self.require_accelerator('tcg')
>> +        self._boot_pseries('tcg')
>> +        self._do_blk_hotplug()
>> +
>> +    def test_pseries_hotplug_blk_kvm(self):
>> +        """Hotplug then unplug a virtio-blk-pci device on pseries (KVM).
>> +
>> +        Requires a ppc64 host with kvm-hv loaded; skipped automatically
>> +        on other hosts.
>> +        """
>> +        self.require_accelerator('kvm')
>> +        # On ppc64 the base kvm module can be loaded (and /dev/kvm created)
>> +        # without kvm-hv, so checking /dev/kvm alone is not sufficient.
>> +        if platform.machine() in ('ppc64le', 'ppc64'):
>> +            try:
>> +                with open('/proc/modules', encoding='utf-8') as f:
>> +                    if not any(line.split()[0] == 'kvm_hv' for line in f):
>> +                        self.skipTest('kvm-hv module is not loaded')
>> +            except OSError:
>> +                pass
>> +        self._boot_pseries('kvm')
>> +        self._do_blk_hotplug()
>> +
>> +
>> +if __name__ == '__main__':
>> +    PseriesHotplugBlk.main()

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro