* [PATCH v4] PCI: vmd: Clean up domain before enumeration
@ 2021-10-25 18:29 Nirmal Patel
2021-11-02 17:55 ` Patel, Nirmal
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Nirmal Patel @ 2021-10-25 18:29 UTC (permalink / raw)
To: Nirmal Patel, linux-pci, jonathan.derrick
During VT-d pass-through, the VMD driver occasionally fails to
enumerate underlying NVMe devices when repetitive reboots are
performed in the guest OS. The issue can be resolved by resetting
VMD root ports for proper enumeration and triggering secondary bus
reset which will also propagate reset through downstream bridges.
Signed-off-by: Nirmal Patel <nirmal.patel@linux.intel.com>
Reviewed-by: Jon Derrick <jonathan.derrick@linux.dev>
---
v3->v4: Using pci_reset_bus function for secondary bus reset instead of
manually triggering secondary bus reset, addressing review
comments of v3.
v2->v3: Combining two functions into one, Remove redundant definations
and Formatting fixes
drivers/pci/controller/vmd.c | 37 ++++++++++++++++++++++++++++++++++++
1 file changed, 37 insertions(+)
diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index a5987e52700e..79f8b86ee45b 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -498,6 +498,40 @@ static inline void vmd_acpi_begin(void) { }
static inline void vmd_acpi_end(void) { }
#endif /* CONFIG_ACPI */
+static void vmd_domain_reset(struct vmd_dev *vmd)
+{
+ u16 bus, max_buses = resource_size(&vmd->resources[0]);
+ u8 dev, functions, fn, hdr_type;
+ char __iomem *base;
+
+ for (bus = 0; bus < max_buses; bus++) {
+ for (dev = 0; dev < 32; dev++) {
+ base = vmd->cfgbar + PCIE_ECAM_OFFSET(bus,
+ PCI_DEVFN(dev, 0), 0);
+
+ hdr_type = readb(base + PCI_HEADER_TYPE) &
+ PCI_HEADER_TYPE_MASK;
+
+ functions = !!(hdr_type & 0x80) ? 8 : 1;
+ for (fn = 0; fn < functions; fn++) {
+ base = vmd->cfgbar + PCIE_ECAM_OFFSET(bus,
+ PCI_DEVFN(dev, fn), 0);
+
+ hdr_type = readb(base + PCI_HEADER_TYPE) &
+ PCI_HEADER_TYPE_MASK;
+
+ if (hdr_type != PCI_HEADER_TYPE_BRIDGE ||
+ (readw(base + PCI_CLASS_DEVICE) !=
+ PCI_CLASS_BRIDGE_PCI))
+ continue;
+
+ memset_io(base + PCI_IO_BASE, 0,
+ PCI_ROM_ADDRESS1 - PCI_IO_BASE);
+ }
+ }
+ }
+}
+
static void vmd_attach_resources(struct vmd_dev *vmd)
{
vmd->dev->resource[VMD_MEMBAR1].child = &vmd->resources[1];
@@ -801,6 +835,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
vmd_acpi_begin();
pci_scan_child_bus(vmd->bus);
+ vmd_domain_reset(vmd);
+ list_for_each_entry(child, &vmd->bus->children, node)
+ pci_reset_bus(child->self);
pci_assign_unassigned_bus_resources(vmd->bus);
/*
--
2.27.0
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [PATCH v4] PCI: vmd: Clean up domain before enumeration
2021-10-25 18:29 [PATCH v4] PCI: vmd: Clean up domain before enumeration Nirmal Patel
@ 2021-11-02 17:55 ` Patel, Nirmal
2021-11-09 15:32 ` Patel, Nirmal
2021-11-14 0:16 ` Krzysztof Wilczyński
2 siblings, 0 replies; 5+ messages in thread
From: Patel, Nirmal @ 2021-11-02 17:55 UTC (permalink / raw)
To: linux-pci, jonathan.derrick; +Cc: lorenzo.pieralisi, helgaas
On 10/25/2021 11:29 AM, Nirmal Patel wrote:
> During VT-d pass-through, the VMD driver occasionally fails to
> enumerate underlying NVMe devices when repetitive reboots are
> performed in the guest OS. The issue can be resolved by resetting
> VMD root ports for proper enumeration and triggering secondary bus
> reset which will also propagate reset through downstream bridges.
>
> Signed-off-by: Nirmal Patel <nirmal.patel@linux.intel.com>
> Reviewed-by: Jon Derrick <jonathan.derrick@linux.dev>
> ---
> v3->v4: Using pci_reset_bus function for secondary bus reset instead of
> manually triggering secondary bus reset, addressing review
> comments of v3.
> v2->v3: Combining two functions into one, Remove redundant definations
> and Formatting fixes
>
> drivers/pci/controller/vmd.c | 37 ++++++++++++++++++++++++++++++++++++
> 1 file changed, 37 insertions(+)
>
> diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
> index a5987e52700e..79f8b86ee45b 100644
> --- a/drivers/pci/controller/vmd.c
> +++ b/drivers/pci/controller/vmd.c
> @@ -498,6 +498,40 @@ static inline void vmd_acpi_begin(void) { }
> static inline void vmd_acpi_end(void) { }
> #endif /* CONFIG_ACPI */
>
> +static void vmd_domain_reset(struct vmd_dev *vmd)
> +{
> + u16 bus, max_buses = resource_size(&vmd->resources[0]);
> + u8 dev, functions, fn, hdr_type;
> + char __iomem *base;
> +
> + for (bus = 0; bus < max_buses; bus++) {
> + for (dev = 0; dev < 32; dev++) {
> + base = vmd->cfgbar + PCIE_ECAM_OFFSET(bus,
> + PCI_DEVFN(dev, 0), 0);
> +
> + hdr_type = readb(base + PCI_HEADER_TYPE) &
> + PCI_HEADER_TYPE_MASK;
> +
> + functions = !!(hdr_type & 0x80) ? 8 : 1;
> + for (fn = 0; fn < functions; fn++) {
> + base = vmd->cfgbar + PCIE_ECAM_OFFSET(bus,
> + PCI_DEVFN(dev, fn), 0);
> +
> + hdr_type = readb(base + PCI_HEADER_TYPE) &
> + PCI_HEADER_TYPE_MASK;
> +
> + if (hdr_type != PCI_HEADER_TYPE_BRIDGE ||
> + (readw(base + PCI_CLASS_DEVICE) !=
> + PCI_CLASS_BRIDGE_PCI))
> + continue;
> +
> + memset_io(base + PCI_IO_BASE, 0,
> + PCI_ROM_ADDRESS1 - PCI_IO_BASE);
> + }
> + }
> + }
> +}
> +
> static void vmd_attach_resources(struct vmd_dev *vmd)
> {
> vmd->dev->resource[VMD_MEMBAR1].child = &vmd->resources[1];
> @@ -801,6 +835,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
> vmd_acpi_begin();
>
> pci_scan_child_bus(vmd->bus);
> + vmd_domain_reset(vmd);
> + list_for_each_entry(child, &vmd->bus->children, node)
> + pci_reset_bus(child->self);
> pci_assign_unassigned_bus_resources(vmd->bus);
>
> /*
Hi Lorenzo,
Please let me know if you are okay with these changes. Thanks.
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH v4] PCI: vmd: Clean up domain before enumeration
2021-10-25 18:29 [PATCH v4] PCI: vmd: Clean up domain before enumeration Nirmal Patel
2021-11-02 17:55 ` Patel, Nirmal
@ 2021-11-09 15:32 ` Patel, Nirmal
2021-11-09 15:54 ` Bjorn Helgaas
2021-11-14 0:16 ` Krzysztof Wilczyński
2 siblings, 1 reply; 5+ messages in thread
From: Patel, Nirmal @ 2021-11-09 15:32 UTC (permalink / raw)
To: linux-pci, lorenzo.pieralisi, Bjorn Helgaas; +Cc: jonathan.derrick
On 10/25/2021 11:29 AM, Nirmal Patel wrote:
> During VT-d pass-through, the VMD driver occasionally fails to
> enumerate underlying NVMe devices when repetitive reboots are
> performed in the guest OS. The issue can be resolved by resetting
> VMD root ports for proper enumeration and triggering secondary bus
> reset which will also propagate reset through downstream bridges.
>
> Signed-off-by: Nirmal Patel <nirmal.patel@linux.intel.com>
> Reviewed-by: Jon Derrick <jonathan.derrick@linux.dev>
> ---
> v3->v4: Using pci_reset_bus function for secondary bus reset instead of
> manually triggering secondary bus reset, addressing review
> comments of v3.
> v2->v3: Combining two functions into one, Remove redundant definations
> and Formatting fixes
>
> drivers/pci/controller/vmd.c | 37 ++++++++++++++++++++++++++++++++++++
> 1 file changed, 37 insertions(+)
>
> diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
> index a5987e52700e..79f8b86ee45b 100644
> --- a/drivers/pci/controller/vmd.c
> +++ b/drivers/pci/controller/vmd.c
> @@ -498,6 +498,40 @@ static inline void vmd_acpi_begin(void) { }
> static inline void vmd_acpi_end(void) { }
> #endif /* CONFIG_ACPI */
>
> +static void vmd_domain_reset(struct vmd_dev *vmd)
> +{
> + u16 bus, max_buses = resource_size(&vmd->resources[0]);
> + u8 dev, functions, fn, hdr_type;
> + char __iomem *base;
> +
> + for (bus = 0; bus < max_buses; bus++) {
> + for (dev = 0; dev < 32; dev++) {
> + base = vmd->cfgbar + PCIE_ECAM_OFFSET(bus,
> + PCI_DEVFN(dev, 0), 0);
> +
> + hdr_type = readb(base + PCI_HEADER_TYPE) &
> + PCI_HEADER_TYPE_MASK;
> +
> + functions = !!(hdr_type & 0x80) ? 8 : 1;
> + for (fn = 0; fn < functions; fn++) {
> + base = vmd->cfgbar + PCIE_ECAM_OFFSET(bus,
> + PCI_DEVFN(dev, fn), 0);
> +
> + hdr_type = readb(base + PCI_HEADER_TYPE) &
> + PCI_HEADER_TYPE_MASK;
> +
> + if (hdr_type != PCI_HEADER_TYPE_BRIDGE ||
> + (readw(base + PCI_CLASS_DEVICE) !=
> + PCI_CLASS_BRIDGE_PCI))
> + continue;
> +
> + memset_io(base + PCI_IO_BASE, 0,
> + PCI_ROM_ADDRESS1 - PCI_IO_BASE);
> + }
> + }
> + }
> +}
> +
> static void vmd_attach_resources(struct vmd_dev *vmd)
> {
> vmd->dev->resource[VMD_MEMBAR1].child = &vmd->resources[1];
> @@ -801,6 +835,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
> vmd_acpi_begin();
>
> pci_scan_child_bus(vmd->bus);
> + vmd_domain_reset(vmd);
> + list_for_each_entry(child, &vmd->bus->children, node)
> + pci_reset_bus(child->self);
> pci_assign_unassigned_bus_resources(vmd->bus);
>
> /*
Hi
Gentle ping. Please let me know if you are okay with these changes (with Jon's Reviewed-by). Thanks.
-nirmal
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH v4] PCI: vmd: Clean up domain before enumeration
2021-11-09 15:32 ` Patel, Nirmal
@ 2021-11-09 15:54 ` Bjorn Helgaas
0 siblings, 0 replies; 5+ messages in thread
From: Bjorn Helgaas @ 2021-11-09 15:54 UTC (permalink / raw)
To: Patel, Nirmal; +Cc: linux-pci, lorenzo.pieralisi, jonathan.derrick
On Tue, Nov 09, 2021 at 08:32:37AM -0700, Patel, Nirmal wrote:
> On 10/25/2021 11:29 AM, Nirmal Patel wrote:
> > During VT-d pass-through, the VMD driver occasionally fails to
> > enumerate underlying NVMe devices when repetitive reboots are
> > performed in the guest OS. The issue can be resolved by resetting
> > VMD root ports for proper enumeration and triggering secondary bus
> > reset which will also propagate reset through downstream bridges.
> >
> > Signed-off-by: Nirmal Patel <nirmal.patel@linux.intel.com>
> > Reviewed-by: Jon Derrick <jonathan.derrick@linux.dev>
> Gentle ping. Please let me know if you are okay with these changes
> (with Jon's Reviewed-by). Thanks.
We're in the middle of the merge window now. We'll start applying
patches after -rc1 is tagged, which will probably happen Nov 14.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v4] PCI: vmd: Clean up domain before enumeration
2021-10-25 18:29 [PATCH v4] PCI: vmd: Clean up domain before enumeration Nirmal Patel
2021-11-02 17:55 ` Patel, Nirmal
2021-11-09 15:32 ` Patel, Nirmal
@ 2021-11-14 0:16 ` Krzysztof Wilczyński
2 siblings, 0 replies; 5+ messages in thread
From: Krzysztof Wilczyński @ 2021-11-14 0:16 UTC (permalink / raw)
To: Nirmal Patel; +Cc: linux-pci, jonathan.derrick
Hi Nirmal,
> +static void vmd_domain_reset(struct vmd_dev *vmd)
> +{
> + u16 bus, max_buses = resource_size(&vmd->resources[0]);
> + u8 dev, functions, fn, hdr_type;
> + char __iomem *base;
> +
> + for (bus = 0; bus < max_buses; bus++) {
> + for (dev = 0; dev < 32; dev++) {
> + base = vmd->cfgbar + PCIE_ECAM_OFFSET(bus,
> + PCI_DEVFN(dev, 0), 0);
> +
> + hdr_type = readb(base + PCI_HEADER_TYPE) &
> + PCI_HEADER_TYPE_MASK;
> +
> + functions = !!(hdr_type & 0x80) ? 8 : 1;
A small nitpick: there is no benefit in converting the result of the
expression in the brackets (alebit, keep the brackets for readibility)
to a boolean alike result.
> + for (fn = 0; fn < functions; fn++) {
> + base = vmd->cfgbar + PCIE_ECAM_OFFSET(bus,
> + PCI_DEVFN(dev, fn), 0);
> +
Thank you for using the ECAM macros!
Reviewed-by: Krzysztof Wilczyński <kw@linux.com>
Krzysztof
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-11-14 0:16 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-10-25 18:29 [PATCH v4] PCI: vmd: Clean up domain before enumeration Nirmal Patel
2021-11-02 17:55 ` Patel, Nirmal
2021-11-09 15:32 ` Patel, Nirmal
2021-11-09 15:54 ` Bjorn Helgaas
2021-11-14 0:16 ` Krzysztof Wilczyński
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).