Re: [RFC v2 2/2] tests/qtest: add SMMUv3 smoke test using smmu-testdev DMA source

["Alex Bennée" <alex.bennee@linaro.org>]

tangtao1634 <tangtao1634@phytium.com.cn> writes:

> From: Tao Tang <tangtao1634@phytium.com.cn>
>
> Introduce a bare-metal qtest that drives the new smmu-testdev to exercise
> the SMMUv3 emulation without guest firmware or drivers. The test programs
> a minimal Non-Secure context (STE/CD/PTE), triggers a DMA, and asserts
> translation results.
>
> Motivation
> ----------
> SMMU testing in emulation often requires a large software stack and a
> realistic PCIe fabric, which adds flakiness and obscures failures. This
> qtest keeps the surface small and deterministic by using a hermetic DMA
> source that feeds the SMMU directly.
>
> What the test covers
> --------------------
> * Builds a Non-Secure STE/CD/PTE for a chosen stream_id/ssid.
> * Primes source and destination host buffers.
> * Kicks a DMA via smmu-testdev and waits for completion.
> * Verifies translated access and payload equality.
>
> Non-goals and scope limits
> --------------------------
> * Secure bank flows are omitted because Secure SMMU support is still RFC.
>   A local Secure test exists and can be posted once the upstream series
>   lands.
> * PCIe discovery, MSI/INTx, ATS/PRI, and driver bring-up are out of scope
>   as smmu-testdev is not a realistic PCIe Endpoint nor a platform device.
>
> Rationale for a dedicated path
> ------------------------------
> Using a generic PCI or virtio device would still require driver init and a
> richer bus model, undermining determinism for this focused purpose. This
> qtest, paired with smmu-testdev, keeps failures attributable to the SMMU
> translation path.
>
> Finally we document the smmu-testdev device in docs/specs.
>
> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
> ---
>  docs/specs/index.rst             |   1 +
>  docs/specs/smmu-testdev.rst      |  45 ++++++
>  tests/qtest/meson.build          |   1 +
>  tests/qtest/smmu-testdev-qtest.c | 238 +++++++++++++++++++++++++++++++
>  4 files changed, 285 insertions(+)
>  create mode 100644 docs/specs/smmu-testdev.rst
>  create mode 100644 tests/qtest/smmu-testdev-qtest.c
>
> diff --git a/docs/specs/index.rst b/docs/specs/index.rst
> index f19d73c9f6..47a18c48f1 100644
> --- a/docs/specs/index.rst
> +++ b/docs/specs/index.rst
> @@ -39,3 +39,4 @@ guest hardware that is specific to QEMU.
>     riscv-iommu
>     riscv-aia
>     aspeed-intc
> +   smmu-testdev
> \ No newline at end of file
> diff --git a/docs/specs/smmu-testdev.rst b/docs/specs/smmu-testdev.rst
> new file mode 100644
> index 0000000000..2599b46e4f
> --- /dev/null
> +++ b/docs/specs/smmu-testdev.rst
> @@ -0,0 +1,45 @@
> +smmu-testdev — Minimal SMMUv3 DMA test device
> +=============================================
> +
> +Overview
> +--------
> +``smmu-testdev`` is a tiny, test-only DMA source intended to exercise the
> +SMMUv3 emulation without booting firmware or a guest OS. It lets tests
> +populate STE/CD/PTE with known values and trigger a DMA that flows through
> +the SMMU translation path. It is **not** a faithful PCIe endpoint nor a
> +platform device and must be considered a QEMU-internal test vehicle.
> +
> +Status
> +------
> +* Location: ``hw/misc/smmu-testdev.c``
> +* Build guard: ``CONFIG_SMMU_TESTDEV``
> +* Default machines: none (tests instantiate it explicitly)
> +* Intended use: qtests under ``tests/qtest/smmu-testdev-qtest.c``
> +
> +Running the qtest
> +-----------------
> +The smoke test ships with this device and is the recommended entry point::
> +
> +    QTEST_QEMU_BINARY=qemu-system-aarch64 ./tests/qtest/smmu-testdev-qtest
> +     --tap -k
> +
> +This programs a minimal Non-Secure SMMU context, kicks a DMA, and verifies
> +translation + data integrity.
> +
> +Instantiation (advanced)
> +------------------------
> +The device is not wired into any board by default. For ad-hoc experiments,
> +tests (or developers) can create it dynamically via qtest or the QEMU
> +monitor. It exposes a single MMIO window that the test drives directly.
> +
> +Limitations
> +-----------
> +* Non-Secure bank only in this version; Secure SMMU tests are planned once
> +  upstream Secure support lands.
> +* No PCIe discovery, MSI, ATS/PRI, or driver bring-up is modeled.
> +* The device is test-only; do not rely on it for machine realism.
> +
> +See also
> +--------
> +* ``tests/qtest/smmu-testdev-qtest.c`` — the companion smoke test
> +* SMMUv3 emulation and documentation under ``hw/arm/smmu*``
> diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
> index 669d07c06b..bcdb51e141 100644
> --- a/tests/qtest/meson.build
> +++ b/tests/qtest/meson.build
> @@ -263,6 +263,7 @@ qtests_aarch64 = \
>     config_all_devices.has_key('CONFIG_TPM_TIS_I2C') ? ['tpm-tis-i2c-test'] : []) + \
>    (config_all_devices.has_key('CONFIG_ASPEED_SOC') ? qtests_aspeed64 : []) + \
>    (config_all_devices.has_key('CONFIG_NPCM8XX') ? qtests_npcm8xx : []) + \
> +  (config_all_devices.has_key('CONFIG_SMMU_TESTDEV') ? ['smmu-testdev-qtest'] : []) + \
>    qtests_cxl +                                                                                  \
>    ['arm-cpu-features',
>     'numa-test',
> diff --git a/tests/qtest/smmu-testdev-qtest.c b/tests/qtest/smmu-testdev-qtest.c
> new file mode 100644
> index 0000000000..d89e45757b
> --- /dev/null
> +++ b/tests/qtest/smmu-testdev-qtest.c
> @@ -0,0 +1,238 @@
> +/*
> + * QTest for smmu-testdev
> + *
> + * This QTest file is used to test the smmu-testdev so that we can test SMMU
> + * without any guest kernel or firmware.
> + *
> + * Copyright (c) 2025 Phytium Technology
> + *
> + * Author:
> + *  Tao Tang <tangtao1634@phytium.com.cn>
> + *
> + * SPDX-License-Identifier: GPL-2.0-or-later
> + */
> +
> +#include "qemu/osdep.h"
> +#include "libqtest.h"
> +#include "libqos/pci.h"
> +#include "libqos/generic-pcihost.h"
> +#include "hw/pci/pci_regs.h"
> +#include "hw/misc/smmu-testdev.h"
> +
> +#define VIRT_SMMU_BASE    0x0000000009050000ULL
> +#define DMA_LEN           0x20U
> +
> +static inline uint64_t smmu_bank_base(uint64_t base, SMMUTestDevSpace sp)
> +{
> +    /* Map only the Non-Secure bank for now; future domains may offset. */
> +    (void)sp;
> +    return base;
> +}
> +
> +static uint32_t expected_dma_result(uint32_t mode,
> +                                    SMMUTestDevSpace s1_space,
> +                                    SMMUTestDevSpace s2_space)
> +{
> +    (void)mode;
> +    if (s1_space != STD_SPACE_NONSECURE || s2_space != STD_SPACE_NONSECURE) {
> +        return STD_DMA_ERR_TX_FAIL;
> +    }
> +    return 0u;
> +}
> +
> +static void smmu_prog_bank(QTestState *qts, uint64_t B, SMMUTestDevSpace sp)
> +{
> +    g_assert_cmpuint(sp, ==, STD_SPACE_NONSECURE);
> +    /* Program minimal SMMUv3 state in a given control bank. */
> +    qtest_writel(qts, B + 0x0044, 0x80000000); /* GBPA UPDATE */
> +    qtest_writel(qts, B + 0x0020, 0x0);       /* CR0 */
> +    qtest_writel(qts, B + 0x0028, 0x0d75);    /* CR1 */
> +    {
> +        /* CMDQ_BASE: add address-space offset (S/NS/Root/Realm). */
> +        uint64_t v = 0x400000000e16b00aULL + std_space_offset(sp);
> +        qtest_writeq(qts, B + 0x0090, v);
> +    }
> +    qtest_writel(qts, B + 0x009c, 0x0);       /* CMDQ_CONS */
> +    qtest_writel(qts, B + 0x0098, 0x0);       /* CMDQ_PROD */
> +    {
> +        /* EVENTQ_BASE: add address-space offset (S/NS/Root/Realm). */
> +        uint64_t v = 0x400000000e17000aULL + std_space_offset(sp);
> +        qtest_writeq(qts, B + 0x00a0, v);
> +    }
> +    qtest_writel(qts, B + 0x00a8, 0x0);       /* EVENTQ_PROD */
> +    qtest_writel(qts, B + 0x00ac, 0x0);       /* EVENTQ_CONS */
> +    qtest_writel(qts, B + 0x0088, 0x5);       /* STRTAB_BASE_CFG */
> +    {
> +        /* STRTAB_BASE: add address-space offset (S/NS/Root/Realm). */
> +        uint64_t v = 0x400000000e179000ULL + std_space_offset(sp);
> +        qtest_writeq(qts, B + 0x0080, v);
> +    }
> +    qtest_writel(qts, B + 0x003C, 0x1);       /* INIT */
> +    qtest_writel(qts, B + 0x0020, 0xD);       /* CR0 */
> +}
> +
> +static void smmu_prog_minimal(QTestState *qts, SMMUTestDevSpace space)
> +{
> +    /* Always program Non-Secure bank, then the requested space. */
> +    uint64_t ns_base = smmu_bank_base(VIRT_SMMU_BASE, STD_SPACE_NONSECURE);
> +    smmu_prog_bank(qts, ns_base, STD_SPACE_NONSECURE);
> +
> +    uint64_t sp_base = smmu_bank_base(VIRT_SMMU_BASE, space);
> +    if (sp_base != ns_base) {
> +        smmu_prog_bank(qts, sp_base, space);
> +    }
> +}
> +
> +static uint32_t poll_dma_result(QPCIDevice *dev, QPCIBar bar,
> +                                QTestState *qts)
> +{
> +    /* Trigger side effects (DMA) via REG_ID read once. */
> +    (void)qpci_io_readl(dev, bar, STD_REG_ID);
> +
> +    /* Poll until not BUSY, then return the result. */
> +    for (int i = 0; i < 1000; i++) {
> +        uint32_t r = qpci_io_readl(dev, bar, STD_REG_DMA_RESULT);
> +        if (r != STD_DMA_RESULT_BUSY) {
> +            return r;
> +        }
> +        /* Small backoff to avoid busy spinning. */
> +        g_usleep(1000);
> +    }
> +    /* Timeout treated as failure-like non-zero. */
> +    return STD_DMA_RESULT_BUSY;
> +}
> +
> +static void test_mmio_access(void)
> +{
> +    QTestState *qts;
> +    QGenericPCIBus gbus;
> +    QPCIDevice *dev;
> +    QPCIBar bar;
> +    uint8_t buf[DMA_LEN];
> +    uint32_t attr_ns;
> +    qts = qtest_init("-machine virt,acpi=off,gic-version=3,iommu=smmuv3 " \
> +                     "-display none -smp 1  -m 512 -cpu max -net none "
> +                     "-device smmu-testdev,device=0x0,function=0x1 ");
> +
> +    qpci_init_generic(&gbus, qts, NULL, false);
> +
> +    /* Find device by vendor/device ID to avoid slot surprises. */
> +    dev = NULL;

might as well init when you declare.

> +    for (int slot = 0; slot < 32 && !dev; slot++) {
> +        for (int fn = 0; fn < 8 && !dev; fn++) {
> +            QPCIDevice *cand = qpci_device_find(&gbus.bus,
> +                                               QPCI_DEVFN(slot, fn));
> +            if (!cand) {
> +                continue;
> +            }
> +            uint16_t vid = qpci_config_readw(cand, PCI_VENDOR_ID);
> +            uint16_t did = qpci_config_readw(cand, PCI_DEVICE_ID);
> +            if (vid == 0x1b36 && did == 0x0005) {
> +                dev = cand;
> +            } else {
> +                g_free(cand);
> +            }
> +        }
> +    }
> +    g_assert_nonnull(dev);

surely g_assert(dev) would do.

> +
> +    qpci_device_enable(dev);
> +    bar = qpci_iomap(dev, 0, NULL);
> +    g_assert_false(bar.is_io);
> +
> +    /* Baseline attribute reads. */
> +    attr_ns = qpci_io_readl(dev, bar, STD_REG_ATTR_NS);
> +    g_assert_cmpuint(attr_ns, ==, 0x2);
> +
> +    /* Program SMMU base and DMA parameters. */
> +    qpci_io_writel(dev, bar, STD_REG_SMMU_BASE_LO, (uint32_t)VIRT_SMMU_BASE);
> +    qpci_io_writel(dev, bar, STD_REG_SMMU_BASE_HI,
> +                   (uint32_t)(VIRT_SMMU_BASE >> 32));
> +    qpci_io_writel(dev, bar, STD_REG_DMA_IOVA_LO, (uint32_t)STD_IOVA);
> +    qpci_io_writel(dev, bar, STD_REG_DMA_IOVA_HI,
> +                   (uint32_t)(STD_IOVA >> 32));
> +    qpci_io_writel(dev, bar, STD_REG_DMA_LEN, DMA_LEN);
> +    qpci_io_writel(dev, bar, STD_REG_DMA_DIR, 0); /* device -> host */
> +
> +    qtest_memset(qts, STD_IOVA, 0x00, DMA_LEN);
> +    qtest_memread(qts, STD_IOVA, buf, DMA_LEN);
> +
> +    /* Refresh attrs via write to ensure legacy functionality still works. */
> +    qpci_io_writel(dev, bar, STD_REG_ID, 0x1);
> +    /*
> +     * invoke translation builder for multiple
> +     * stage/security-space combinations (readable/refactored).
> +     */
> +    const uint32_t modes[] = { 0u, 1u, 2u }; /* Stage1, Stage2, Nested stage */
> +    const SMMUTestDevSpace spaces[] = { STD_SPACE_NONSECURE };

top of block.

> +    /* Use attrs-DMA path for end-to-end */
> +    qpci_io_writel(dev, bar, STD_REG_DMA_MODE, 1);
> +    for (size_t mi = 0; mi < sizeof(modes) / sizeof(modes[0]); mi++) {
> +        const SMMUTestDevSpace *s1_set = NULL;
> +        size_t s1_count = 0;
> +        const SMMUTestDevSpace *s2_set = NULL;
> +        size_t s2_count = 0;
> +
> +        switch (modes[mi]) {
> +        case 0u:
> +        case 1u:
> +        case 2u:
> +            s1_set = spaces;
> +            s1_count = sizeof(spaces) / sizeof(spaces[0]);
> +            s2_set = spaces;
> +            s2_count = sizeof(spaces) / sizeof(spaces[0]);
> +            break;
> +        default:
> +            g_assert_not_reached();
> +        }
> +
> +        for (size_t si = 0; si < s1_count; si++) {
> +            for (size_t sj = 0; sj < s2_count; sj++) {
> +                qpci_io_writel(dev, bar, STD_REG_TRANS_MODE, modes[mi]);
> +                qpci_io_writel(dev, bar, STD_REG_S1_SPACE, s1_set[si]);
> +                qpci_io_writel(dev, bar, STD_REG_S2_SPACE, s2_set[sj]);
> +                qpci_io_writel(dev, bar, STD_REG_TRANS_DBELL, 0x2);
> +                qpci_io_writel(dev, bar, STD_REG_TRANS_DBELL, 0x1);
> +
> +                uint32_t st = qpci_io_readl(dev, bar,
> +                                            STD_REG_TRANS_STATUS);
> +                g_test_message("build: stage=%s s1=%s s2=%s status=0x%x",
> +                                std_mode_to_str(modes[mi]),
> +                                std_space_to_str(s1_set[si]),
> +                                std_space_to_str(s2_set[sj]), st);
> +                /* Program SMMU registers in selected control bank. */
> +                smmu_prog_minimal(qts, s1_set[si]);
> +
> +                /* End-to-end DMA using tx_space per mode. */
> +                SMMUTestDevSpace tx_space =
> +                    (modes[mi] == 0u) ? s1_set[si] : s2_set[sj];
> +                uint32_t dma_attrs = ((uint32_t)tx_space << 1);
> +                qpci_io_writel(dev, bar, STD_REG_DMA_ATTRS,
> +                                dma_attrs);
> +                qpci_io_writel(dev, bar, STD_REG_DMA_DBELL, 1);
> +                /* Wait for DMA completion and assert success. */
> +                {
> +                    uint32_t dr = poll_dma_result(dev, bar, qts);
> +                    uint32_t exp = expected_dma_result(modes[mi],
> +                                                        spaces[si],
> +                                                        spaces[sj]);
> +                    g_assert_cmpuint(dr, ==, exp);
> +                    g_test_message("polling end. attrs=0x%x res=0x%x",
> +                                   dma_attrs, dr);
> +                }
> +                /* Clear CD/STE/PTE built by the device for next round. */
> +                qpci_io_writel(dev, bar, STD_REG_TRANS_CLEAR, 1);
> +                g_test_message("clear cache end.");
> +            }
> +        }
> +    }

I suspect this function could be broken up a bit as new tests are added
and functionality shared?

> +
> +    qtest_quit(qts);
> +}
> +
> +int main(int argc, char **argv)
> +{
> +    g_test_init(&argc, &argv, NULL);
> +    qtest_add_func("/smmu-testdev/mmio", test_mmio_access);
> +    return g_test_run();
> +}

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro