RE: [PATCH v2 0/8] Initial support for SMMUv3 nested translation

Shameerali Kolothum Thodi
2024-08-30T09:10:03Z

> -----Original Message-----
> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Thursday, August 29, 2024 5:10 PM
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: Jason Gunthorpe <jgg@nvidia.com>; acpica-devel@lists.linux.dev;
> Guohanjun (Hanjun Guo) <guohanjun@huawei.com>; iommu@lists.linux.dev;
> Joerg Roedel <joro@8bytes.org>; Kevin Tian <kevin.tian@intel.com>;
> kvm@vger.kernel.org; Len Brown <lenb@kernel.org>; linux-
> acpi@vger.kernel.org; linux-arm-kernel@lists.infradead.org; Lorenzo Pieralisi
> <lpieralisi@kernel.org>; Rafael J. Wysocki <rafael@kernel.org>; Robert Moore
> <robert.moore@intel.com>; Robin Murphy <robin.murphy@arm.com>; Sudeep
> Holla <sudeep.holla@arm.com>; Will Deacon <will@kernel.org>; Alex
> Williamson <alex.williamson@redhat.com>; Eric Auger
> <eric.auger@redhat.com>; Jean-Philippe Brucker <jean-philippe@linaro.org>;
> Moritz Fischer <mdf@kernel.org>; Michael Shavit <mshavit@google.com>;
> patches@lists.linux.dev; Mostafa Saleh <smostafa@google.com>
> Subject: Re: [PATCH v2 0/8] Initial support for SMMUv3 nested translation
>
> On Thu, Aug 29, 2024 at 02:52:23PM +0000, Shameerali Kolothum Thodi wrote:
> > > That makes some progress. But still I am not seeing the assigned dev in
> > > Guest.
> > >
> > > -device vfio-pci-nohotplug,host=0000:75:00.1,iommufd=iommufd0
> > >
> > > root@ubuntu:/# lspci -tv#
> > >
> > > root@ubuntu:/# lspci -tv
> > > -+-[0000:ca]---00.0-[cb]--
> > > \-[0000:00]-+-00.0 Red Hat, Inc. QEMU PCIe Host bridge
> > > +-01.0 Red Hat, Inc Virtio network device
> > > +-02.0 Red Hat, Inc. QEMU PCIe Expander bridge
> > > +-03.0 Red Hat, Inc. QEMU PCIe Expander bridge
> > > +-04.0 Red Hat, Inc. QEMU PCIe Expander bridge
> > > +-05.0 Red Hat, Inc. QEMU PCIe Expander bridge
> > > +-06.0 Red Hat, Inc. QEMU PCIe Expander bridge
> > > +-07.0 Red Hat, Inc. QEMU PCIe Expander bridge
> > > +-08.0 Red Hat, Inc. QEMU PCIe Expander bridge
> > > \-09.0 Red Hat, Inc. QEMU PCIe Expander bridge
>
> Hmm, the tree looks correct..
>
> > > The new root port is created, but no device attached.
> > It looks like Guest finds the config invalid:
> >
> > [ 0.283618] PCI host bridge to bus 0000:ca
> > [ 0.284064] ACPI BIOS Error (bug):
> \_SB.PCF7.PCEE.PCE5.PCDC.PCD3.PCCA._DSM: Excess arguments - ASL declared
> 5, ACPI requires 4 (20240322/nsarguments-162)
>
> Looks like the DSM change wasn't clean. Yet, this might not be the
> root cause, as mine could boot with it.

Yes. This is not the culprit in this case and was reported earlier as well,

https://patchew.org/QEMU/20211005085313.493858-1-eric.auger@redhat.com/20211005085313.493858-2-eric.auger@redhat.com/

> Here is mine (I added a print to that conflict part, for success):
>
> [ 0.340733] ACPI BIOS Error (bug): \_SB.PCF7.PCEE.PCE5.PCDC._DSM: Excess
> arguments - ASL declared 5, ACPI requires 4 (20230628/nsarguments-162)
> [ 0.341776] pci 0000:dc:00.0: [1b36:000c] type 01 class 0x060400 PCIe Root
> Port
> [ 0.344895] pci 0000:dc:00.0: BAR 0 [mem 0x10400000-0x10400fff]
> [ 0.347935] pci 0000:dc:00.0: PCI bridge to [bus dd]
> [ 0.348410] pci 0000:dc:00.0: bridge window [mem 0x10200000-0x103fffff]
> [ 0.349483] pci 0000:dc:00.0: bridge window [mem 0x42000000000-
> 0x44080ffffff 64bit pref]
> [ 0.351459] pci_bus 0000:dd: busn_res: insert [bus dd] under [bus dc-dd]
>
> In my case:
> [root bus (00)] <---[pxb (dc)] <--- [root-port (dd)] <--- dev
>
> In your case:
> [root bus (00)] <---[pxb (ca)] <--- [root-port (cb)] <--- dev
>
> > [ 0.285533] pci_bus 0000:ca: root bus resource [bus ca]
> > [ 0.286214] pci 0000:ca:00.0: [1b36:000c] type 01 class 0x060400 PCIe Root
> Port
> > [ 0.287717] pci 0000:ca:00.0: BAR 0 [mem 0x00000000-0x00000fff]
> > [ 0.288431] pci 0000:ca:00.0: PCI bridge to [bus 00]
>
> This starts to diff. Somehow the link is reversed? It should be:
> [ 0.288431] pci 0000:ca:00.0: PCI bridge to [bus cb]
>
> > [ 0.290649] pci 0000:ca:00.0: bridge configuration invalid ([bus 00-00]),
> reconfiguring
> > [ 0.292476] pci_bus 0000:cb: busn_res: can not insert [bus cb-ca] under [bus
> ca] (conflicts with (null) [bus ca])
> > [ 0.293597] pci_bus 0000:cb: busn_res: [bus cb-ca] end is updated to cb
> > [ 0.294300] pci_bus 0000:cb: busn_res: can not insert [bus cb] under [bus ca]
> (conflicts with (null) [bus ca])
>
> And then everything went south...
>
> Would you please try adding some prints?
> ----------------------------------------------------------------------
> @@ -1556,6 +1556,7 @@ static char *create_new_pcie_port(VirtNestedSmmu
> *nested_smmu, Error **errp)
> uint32_t bus_nr = pci_bus_num(nested_smmu->pci_bus);
> DeviceState *dev;
> char *name_port;
> + bool ret;
>
> /* Create a root port */
> dev = qdev_new("pcie-root-port");
> @@ -1571,7 +1572,9 @@ static char *create_new_pcie_port(VirtNestedSmmu
> *nested_smmu, Error **errp)
> qdev_prop_set_uint32(dev, "chassis", chassis_nr);
> qdev_prop_set_uint32(dev, "slot", port_nr);
> qdev_prop_set_uint64(dev, "io-reserve", 0);
> - qdev_realize_and_unref(dev, BUS(nested_smmu->pci_bus), &error_fatal);
> + ret = qdev_realize_and_unref(dev, BUS(nested_smmu->pci_bus),
> &error_fatal);
> + fprintf(stderr, "ret=%d, pcie-root-port ID: %s, added to pxb_bus num: %x,
> chassis: %d\n",
> + ret, name_port, pci_bus_num(nested_smmu->pci_bus), chassis_nr);
> return name_port;
> }

Print shows everything fine:
create_new_pcie_port: name_port smmu_bus0xca_port0, bus_nr 0xca chassis_nr 0xfd, nested_smmu->index 0x2, pci_bus_num 0xca, ret 1

It looks like a problem with old QEMU_EFI.fd(2022 build and before).
I tried with 2023 QEMU_EFI.fd and with that it looks fine.

root@ubuntu:/# lspci -tv
-+-[0000:ca]---00.0-[cb]----00.0 Huawei Technologies Co., Ltd. Device a251
\-[0000:00]-+-00.0 Red Hat, Inc. QEMU PCIe Host bridge
+-01.0 Red Hat, Inc Virtio network device
+-02.0 Red Hat, Inc. QEMU PCIe Expander bridge
+-03.0 Red Hat, Inc. QEMU PCIe Expander bridge
+-04.0 Red Hat, Inc. QEMU PCIe Expander bridge
+-05.0 Red Hat, Inc. QEMU PCIe Expander bridge
+-06.0 Red Hat, Inc. QEMU PCIe Expander bridge
+-07.0 Red Hat, Inc. QEMU PCIe Expander bridge
+-08.0 Red Hat, Inc. QEMU PCIe Expander bridge
\-09.0 Red Hat, Inc. QEMU PCIe Expander bridge

So for now, I can proceed.

Thanks,
Shameer