From: Donald Dutile <ddutile@redhat.com>
To: Nathan Chen <nathanc@nvidia.com>,
Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>,
qemu-arm@nongnu.org, qemu-devel@nongnu.org
Cc: eric.auger@redhat.com, peter.maydell@linaro.org, jgg@nvidia.com,
nicolinc@nvidia.com, berrange@redhat.com, mochs@nvidia.com,
smostafa@google.com, linuxarm@huawei.com,
wangzhou1@hisilicon.com, jiangkunkun@huawei.com,
jonathan.cameron@huawei.com, zhangfei.gao@linaro.org
Subject: Re: [PATCH v3 0/6] hw/arm/virt: Add support for user creatable SMMUv3 device
Date: Thu, 5 Jun 2025 16:58:54 -0400 [thread overview]
Message-ID: <3d003b4c-5efa-487f-9156-9344613ead55@redhat.com> (raw)
In-Reply-To: <90957693-dc12-4731-960f-0ee295d297ec@nvidia.com>
On 6/5/25 1:58 PM, Nathan Chen wrote:
>
>
> On 6/4/2025 7:34 PM, Donald Dutile wrote:
>> On 6/4/25 10:02 PM, Nathan Chen wrote:
>>> On 6/2/2025 8:41 AM, Shameer Kolothum wrote:
>>>> This patch series introduces support for a user-creatable SMMUv3 device
>>>> (-device arm-smmuv3) in QEMU.
>>>
>>> Tested-by: Nathan Chen <nathanc@nvidia.com>
>>>
>>> I am able to create 16 SMMUv3 devices in a qemu VM with emulated devices properly associated with the guest SMMUs in guest sysfs - verified with some guest SMMUs having two or three emulated NICs assigned to them while other guest SMMUs have a minimum of one assigned.
>>>
>>> Thanks,
>>> Nathan
>>>
>> Nathan,
>> Great test!
>> Can you share the xml &/or qemu cmdline, to demonstrate this broad capability?
>> Also would be a good reference when (we know it isn't 'if') a new/ different config
>> that wasn't tested shows a bug, and we'll have more info to work with.
>
> Yes, here is the qemu command line I used. smmuv3.15 is associated with two devices, smmuv3.0 is associated with two devices, and smmuv3.1 is associated with three devices:
>
> qemu-system-aarch64 \
> -machine hmat=on -machine virt,accel=kvm,gic-version=3,ras=on \
> -cpu host -smp cpus=4 -m size=16G,slots=4,maxmem=32G -nographic \
> -bios /usr/share/AAVMF/AAVMF_CODE.fd \
> -device nvme,drive=nvme0,serial=deadbeaf1,bus=pcie.0 \
> -drive file=/home/nvidia/nathanc/noble-server-cloudimg-arm64.qcow2,index=0,media=disk,format=qcow2,if=none,id=nvme0 \
> -device e1000,romfile=/root/nathanc/efi-e1000.rom,netdev=net0,bus=pcie.0 \
> -netdev user,id=net0,hostfwd=tcp::5558-:22,hostfwd=tcp::5586-:5586 \
> -netdev user,id=net1 \
> -netdev user,id=net2 \
> -netdev user,id=net3 \
> -netdev user,id=net4 \
> -netdev user,id=net5 \
> -netdev user,id=net6 \
> -netdev user,id=net7 \
> -netdev user,id=net8 \
> -netdev user,id=net9 \
> -netdev user,id=net10 \
> -netdev user,id=net11 \
> -netdev user,id=net12 \
> -netdev user,id=net13 \
> -netdev user,id=net14 \
> -netdev user,id=net15 \
> -netdev user,id=net16 \
> -netdev user,id=net17 \
> -netdev user,id=net18 \
> -netdev user,id=net19 \
> -netdev user,id=net20 \
> -device arm-smmuv3,primary-bus=pcie.0,id=smmuv3.0 \
> -device virtio-net-pci,bus=pcie.0,netdev=net1 \
> -device virtio-net-pci,bus=pcie.0,netdev=net20 \
> -device pxb-pcie,id=pcie.1,bus_nr=200,bus=pcie.0 \
> -device arm-smmuv3,primary-bus=pcie.1,id=smmuv3.1 \
> -device pcie-root-port,id=pcie.port1,bus=pcie.1,slot=1,chassis=1 \
> -device virtio-net-pci,bus=pcie.port1,netdev=net2 \
> -device pcie-root-port,id=pcie.port17,bus=pcie.1,slot=17,chassis=17 \
> -device virtio-net-pci,bus=pcie.port17,netdev=net18 \
> -device pcie-root-port,id=pcie.port18,bus=pcie.1,slot=18,chassis=18 \
> -device virtio-net-pci,bus=pcie.port18,netdev=net19 \
> -device pxb-pcie,id=pcie.2,bus_nr=196,bus=pcie.0 \
> -device arm-smmuv3,primary-bus=pcie.2,id=smmuv3.2 \
> -device pcie-root-port,id=pcie.port2,bus=pcie.2,slot=2,chassis=2 \
> -device virtio-net-pci,bus=pcie.port2,netdev=net3 \
> -device pxb-pcie,id=pcie.3,bus_nr=192,bus=pcie.0 \
> -device arm-smmuv3,primary-bus=pcie.3,id=smmuv3.3 \
> -device pcie-root-port,id=pcie.port3,bus=pcie.3,slot=3,chassis=3 \
> -device virtio-net-pci,bus=pcie.port3,netdev=net4 \
> -device pxb-pcie,id=pcie.4,bus_nr=188,bus=pcie.0 \
> -device arm-smmuv3,primary-bus=pcie.4,id=smmuv3.4 \
> -device pcie-root-port,id=pcie.port4,bus=pcie.4,slot=4,chassis=4 \
> -device virtio-net-pci,bus=pcie.port4,netdev=net5 \
> -device pxb-pcie,id=pcie.5,bus_nr=184,bus=pcie.0 \
> -device arm-smmuv3,primary-bus=pcie.5,id=smmuv3.5 \
> -device pcie-root-port,id=pcie.port5,bus=pcie.5,slot=5,chassis=5 \
> -device virtio-net-pci,bus=pcie.port5,netdev=net6 \
> -device pxb-pcie,id=pcie.6,bus_nr=180,bus=pcie.0 \
> -device arm-smmuv3,primary-bus=pcie.6,id=smmuv3.6 \
> -device pcie-root-port,id=pcie.port6,bus=pcie.6,slot=6,chassis=6 \
> -device virtio-net-pci,bus=pcie.port6,netdev=net7 \
> -device pxb-pcie,id=pcie.7,bus_nr=176,bus=pcie.0 \
> -device arm-smmuv3,primary-bus=pcie.7,id=smmuv3.7 \
> -device pcie-root-port,id=pcie.port7,bus=pcie.7,slot=7,chassis=7 \
> -device virtio-net-pci,bus=pcie.port7,netdev=net8 \
> -device pxb-pcie,id=pcie.8,bus_nr=172,bus=pcie.0 \
> -device arm-smmuv3,primary-bus=pcie.8,id=smmuv3.8 \
> -device pcie-root-port,id=pcie.port8,bus=pcie.8,slot=8,chassis=8 \
> -device virtio-net-pci,bus=pcie.port8,netdev=net9 \
> -device pxb-pcie,id=pcie.9,bus_nr=168,bus=pcie.0 \
> -device arm-smmuv3,primary-bus=pcie.9,id=smmuv3.9 \
> -device pcie-root-port,id=pcie.port9,bus=pcie.9,slot=9,chassis=9 \
> -device virtio-net-pci,bus=pcie.port9,netdev=net10 \
> -device pxb-pcie,id=pcie.10,bus_nr=164,bus=pcie.0 \
> -device arm-smmuv3,primary-bus=pcie.10,id=smmuv3.10 \
> -device pcie-root-port,id=pcie.port10,bus=pcie.10,slot=10,chassis=10 \
> -device virtio-net-pci,bus=pcie.port10,netdev=net11 \
> -device pxb-pcie,id=pcie.11,bus_nr=160,bus=pcie.0 \
> -device arm-smmuv3,primary-bus=pcie.11,id=smmuv3.11 \
> -device pcie-root-port,id=pcie.port11,bus=pcie.11,slot=11,chassis=11 \
> -device virtio-net-pci,bus=pcie.port11,netdev=net12 \
> -device pxb-pcie,id=pcie.12,bus_nr=156,bus=pcie.0 \
> -device arm-smmuv3,primary-bus=pcie.12,id=smmuv3.12 \
> -device pcie-root-port,id=pcie.port12,bus=pcie.12,slot=12,chassis=12 \
> -device virtio-net-pci,bus=pcie.port12,netdev=net13 \
> -device pxb-pcie,id=pcie.13,bus_nr=152,bus=pcie.0 \
> -device arm-smmuv3,primary-bus=pcie.13,id=smmuv3.13 \
> -device pcie-root-port,id=pcie.port13,bus=pcie.13,slot=13,chassis=13 \
> -device virtio-net-pci,bus=pcie.port13,netdev=net14 \
> -device pxb-pcie,id=pcie.14,bus_nr=148,bus=pcie.0 \
> -device arm-smmuv3,primary-bus=pcie.14,id=smmuv3.14 \
> -device pcie-root-port,id=pcie.port14,bus=pcie.14,slot=14,chassis=14 \
> -device virtio-net-pci,bus=pcie.port14,netdev=net15 \
> -device pxb-pcie,id=pcie.15,bus_nr=144,bus=pcie.0 \
> -device arm-smmuv3,primary-bus=pcie.15,id=smmuv3.15 \
> -device pcie-root-port,id=pcie.port15,bus=pcie.15,slot=15,chassis=15 \
> -device virtio-net-pci,bus=pcie.port15,netdev=net16 \
> -device pcie-root-port,id=pcie.port16,bus=pcie.15,slot=16,chassis=16 \
> -device virtio-net-pci,bus=pcie.port16,netdev=net17
>
> Here is the guest topology:
>
> $ lspci -tv
> -+-[0000:00]-+-00.0 Red Hat, Inc. QEMU PCIe Host bridge
> | +-01.0 Red Hat, Inc. QEMU NVM Express Controller
> | +-02.0 Intel Corporation 82540EM Gigabit Ethernet Controller
> | +-03.0 Red Hat, Inc. Virtio network device
> | +-04.0 Red Hat, Inc. Virtio network device
> | +-05.0 Red Hat, Inc. QEMU PCIe Expander bridge
> | +-06.0 Red Hat, Inc. QEMU PCIe Expander bridge
> | +-07.0 Red Hat, Inc. QEMU PCIe Expander bridge
> | +-08.0 Red Hat, Inc. QEMU PCIe Expander bridge
> | +-09.0 Red Hat, Inc. QEMU PCIe Expander bridge
> | +-0a.0 Red Hat, Inc. QEMU PCIe Expander bridge
> | +-0b.0 Red Hat, Inc. QEMU PCIe Expander bridge
> | +-0c.0 Red Hat, Inc. QEMU PCIe Expander bridge
> | +-0d.0 Red Hat, Inc. QEMU PCIe Expander bridge
> | +-0e.0 Red Hat, Inc. QEMU PCIe Expander bridge
> | +-0f.0 Red Hat, Inc. QEMU PCIe Expander bridge
> | +-10.0 Red Hat, Inc. QEMU PCIe Expander bridge
> | +-11.0 Red Hat, Inc. QEMU PCIe Expander bridge
> | +-12.0 Red Hat, Inc. QEMU PCIe Expander bridge
> | \-13.0 Red Hat, Inc. QEMU PCIe Expander bridge
> +-[0000:90]-+-00.0-[91]----00.0 Red Hat, Inc. Virtio 1.0 network device
> | \-01.0-[92]----00.0 Red Hat, Inc. Virtio 1.0 network device
> +-[0000:94]---00.0-[95]----00.0 Red Hat, Inc. Virtio 1.0 network device
> +-[0000:98]---00.0-[99]----00.0 Red Hat, Inc. Virtio 1.0 network device
> +-[0000:9c]---00.0-[9d]----00.0 Red Hat, Inc. Virtio 1.0 network device
> +-[0000:a0]---00.0-[a1]----00.0 Red Hat, Inc. Virtio 1.0 network device
> +-[0000:a4]---00.0-[a5]----00.0 Red Hat, Inc. Virtio 1.0 network device
> +-[0000:a8]---00.0-[a9]----00.0 Red Hat, Inc. Virtio 1.0 network device
> +-[0000:ac]---00.0-[ad]----00.0 Red Hat, Inc. Virtio 1.0 network device
> +-[0000:b0]---00.0-[b1]----00.0 Red Hat, Inc. Virtio 1.0 network device
> +-[0000:b4]---00.0-[b5]----00.0 Red Hat, Inc. Virtio 1.0 network device
> +-[0000:b8]---00.0-[b9]----00.0 Red Hat, Inc. Virtio 1.0 network device
> +-[0000:bc]---00.0-[bd]----00.0 Red Hat, Inc. Virtio 1.0 network device
> +-[0000:c0]---00.0-[c1]----00.0 Red Hat, Inc. Virtio 1.0 network device
> +-[0000:c4]---00.0-[c5]----00.0 Red Hat, Inc. Virtio 1.0 network device
> \-[0000:c8]-+-00.0-[c9]----00.0 Red Hat, Inc. Virtio 1.0 network device
> +-01.0-[ca]----00.0 Red Hat, Inc. Virtio 1.0 network device
> \-02.0-[cb]----00.0 Red Hat, Inc. Virtio 1.0 network device
>
> Thanks,
> Nathan
>
Needs a device-assigned NVIDIA GPU! ;-)
Thanks for the config!
- Don
prev parent reply other threads:[~2025-06-05 21:00 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-06-02 15:41 [PATCH v3 0/6] hw/arm/virt: Add support for user creatable SMMUv3 device Shameer Kolothum via
2025-06-02 15:41 ` [PATCH v3 1/6] hw/arm/smmuv3: Check SMMUv3 has PCIe Root Complex association Shameer Kolothum via
2025-06-05 9:13 ` Eric Auger
2025-06-05 9:53 ` Eric Auger
2025-06-05 10:02 ` Eric Auger via
2025-06-05 11:15 ` Shameerali Kolothum Thodi via
2025-06-05 10:55 ` Igor Mammedov
2025-06-05 11:29 ` Shameerali Kolothum Thodi via
2025-06-05 12:19 ` Igor Mammedov
2025-06-05 12:36 ` Shameerali Kolothum Thodi via
2025-06-05 13:05 ` Igor Mammedov
2025-06-02 15:41 ` [PATCH v3 2/6] hw/arm/virt-acpi-build: Re-arrange SMMUv3 IORT build Shameer Kolothum via
2025-06-05 9:39 ` Eric Auger
2025-06-05 11:10 ` Shameerali Kolothum Thodi via
2025-06-02 15:41 ` [PATCH v3 3/6] hw/arm/virt-acpi-build: Update IORT for multiple smmuv3 devices Shameer Kolothum via
2025-06-05 9:57 ` Eric Auger
2025-06-05 11:14 ` Shameerali Kolothum Thodi via
2025-06-02 15:41 ` [PATCH v3 4/6] hw/arm/virt: Factor out common SMMUV3 dt bindings code Shameer Kolothum via
2025-06-02 15:41 ` [PATCH v3 5/6] hw/arm/virt: Add an SMMU_IO_LEN macro Shameer Kolothum via
2025-06-02 15:41 ` [PATCH v3 6/6] hw/arm/virt: Allow user-creatable SMMUv3 dev instantiation Shameer Kolothum via
2025-06-05 12:45 ` Eric Auger
2025-06-05 2:02 ` [PATCH v3 0/6] hw/arm/virt: Add support for user creatable SMMUv3 device Nathan Chen
2025-06-05 2:34 ` Donald Dutile
2025-06-05 17:58 ` Nathan Chen
2025-06-05 20:58 ` Donald Dutile [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=3d003b4c-5efa-487f-9156-9344613ead55@redhat.com \
--to=ddutile@redhat.com \
--cc=berrange@redhat.com \
--cc=eric.auger@redhat.com \
--cc=jgg@nvidia.com \
--cc=jiangkunkun@huawei.com \
--cc=jonathan.cameron@huawei.com \
--cc=linuxarm@huawei.com \
--cc=mochs@nvidia.com \
--cc=nathanc@nvidia.com \
--cc=nicolinc@nvidia.com \
--cc=peter.maydell@linaro.org \
--cc=qemu-arm@nongnu.org \
--cc=qemu-devel@nongnu.org \
--cc=shameerali.kolothum.thodi@huawei.com \
--cc=smostafa@google.com \
--cc=wangzhou1@hisilicon.com \
--cc=zhangfei.gao@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).