Re: [PATCH v8 2/8] PCI: vmd: Add vmd_configure_cfgbar() helper function

Re: [PATCH v8 2/8] PCI: vmd: Add vmd_configure_cfgbar() helper function

[sashiko-bot]

> Move the VMD CFGBAR initialization code to a new helper
> vmd_configure_cfgbar(). No functional changes.
> 
> Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
> Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260921164316.3860312-1-szymon.durawa@linux.intel.com?part=2

Re: [PATCH v8 3/8] PCI: vmd: Add vmd_configure_membar() and vmd_configure_membar1_membar2()

[sashiko-bot]

> Move the MEMBAR1 and MEMBAR2 registry initialization code to new helpers
> vmd_configure_membar() and vmd_configure_membar1_membar2().
> The refactor preserves MEMBAR address/offset/flags programming, but
> resource name construction is now generated in the helper ("VMD MEMBAR%d")
> instead of using two separate string literals.
> 
> Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
> Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260921164316.3860312-1-szymon.durawa@linux.intel.com?part=3

Re: [PATCH v8 1/8] PCI: vmd: Add vmd_bus_enumeration() helper function

[sashiko-bot]

> Move the vmd bus enumeration code to a new helper vmd_bus_enumeration().
> No functional changes.
> 
> Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
> Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260921164316.3860312-1-szymon.durawa@linux.intel.com?part=1

Re: [PATCH v8 5/8] PCI: vmd: Replace hardcoded values with enum and defines

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] pci: incomplete refactoring, hardcoded index '0' used instead of VMD_RES_CFGBAR
--

commit d6af2294e28d1c964e8b78ed89f34bf845a9c3ed
Author: Szymon Durawa <szymon.durawa@linux.intel.com>

PCI: vmd: Replace hardcoded values with enum and defines

Add enum vmd_resource type to replace hardcoded values. Add defines for
vmd bus start number based on VMD restriction value. No functional
changes.

> diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
> index 8b59b369f1263..b7c4e22deee58 100644
> --- a/drivers/pci/controller/vmd.c
> +++ b/drivers/pci/controller/vmd.c

[ ... ]

> @@ -562,7 +574,7 @@ static resource_size_t vmd_cfgbar_ecam_space(struct vmd_dev *vmd)
>  }
>  static void vmd_domain_reset(struct vmd_dev *vmd)
>  {
> -	u16 bus, max_buses = resource_size(&vmd->resources[0]);
> +	u16 bus, max_buses = resource_size(&vmd->resources[VMD_RES_CFGBAR]);
>  	u8 dev, functions, fn, hdr_type;
>  	unsigned int ecam_bus;
>  	char __iomem *base;

[Severity: Low]
Does this refactoring miss other instances of the hardcoded 0 index?

For example, in the preceding function vmd_cfgbar_ecam_space():

	ecam_start = vmd_bus_to_ecam(vmd, vmd->resources[0].start);

And a few lines further down in vmd_domain_reset():

		ecam_bus = vmd_bus_to_ecam(vmd, vmd->resources[0].start + bus);

Could these also be updated to use VMD_RES_CFGBAR for consistency?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260921164316.3860312-1-szymon.durawa@linux.intel.com?part=5

Re: [PATCH v8 6/8] PCI: vmd: Convert bus and busn_start to an array

[sashiko-bot]

> Convert bus and busn_start from scalar to an array to support
> multiple VMD buses in the future. No functional changes.
> 
> Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
> Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260921164316.3860312-1-szymon.durawa@linux.intel.com?part=6

Re: [PATCH v8 4/8] PCI: vmd: Add vmd_create_bus()

[sashiko-bot]

> Move the VMD bus initialization code to a new helper vmd_create_bus().
> No functional changes.
> 
> Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
> Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260921164316.3860312-1-szymon.durawa@linux.intel.com?part=4

Re: [PATCH v8 7/8] PCI: vmd: Add support for second rootbus under VMD

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] pci: vmd: root ports skipped during domain reset due to CFGBAR start offset resize
- [High] pci: vmd: unprotected modification of global resource tree pointers and structures
--

commit 21e75586431de95d78578f413b1e8f1a06a83836
Author: Szymon Durawa <szymon.durawa@linux.intel.com>
Subject: PCI: vmd: Add support for second rootbus under VMD

This commit introduces support for a second root bus under VMD, which is
needed starting from Intel Arrow Lake. It enumerates BUS1, divides the
MMIO space to be shared between both root buses, and resizes the CFGBAR
accordingly.

> diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
> index c6b5e693916cb..51b8c2d9b7f0b 100644
> --- a/drivers/pci/controller/vmd.c
> +++ b/drivers/pci/controller/vmd.c

[ ... ]

> @@ -632,12 +664,24 @@ static void vmd_attach_resources(struct vmd_dev *vmd)
>  {
>  	vmd->dev->resource[VMD_MEMBAR1].child = &vmd->resources[VMD_RES_MBAR_1];
>  	vmd->dev->resource[VMD_MEMBAR2].child = &vmd->resources[VMD_RES_MBAR_2];
> +
> +	if (vmd->bus1_rootbus) {
> +		vmd->resources[VMD_RES_MBAR_1].sibling =
> +			&vmd->resources[VMD_RES_BUS1_MBAR_1];
> +		vmd->resources[VMD_RES_MBAR_2].sibling =
> +			&vmd->resources[VMD_RES_BUS1_MBAR_2];
> +	}

[Severity: High]
Does manually assigning the .sibling pointers here introduce a data race?

These assignments link the BUS1 resources directly into the global iomem
resource tree without acquiring the resource_lock.

On the vmd_create_bus() error path later in vmd_enable_domain(), these
sibling pointers are cleared and the resource structures are zeroed, also
without locking:

drivers/pci/controller/vmd.c:vmd_enable_domain() {
	...
	vmd->resources[VMD_RES_MBAR_1].sibling = NULL;
	vmd->resources[VMD_RES_MBAR_2].sibling = NULL;
	vmd->resources[VMD_RES_BUS1_CFGBAR] = (struct resource){};
	...
}

Can concurrent readers (like /proc/iomem iterators traversing the tree)
encounter corrupted or zeroed structures, leading to a crash or invalid reads?

>  }

[ ... ]

> @@ -936,6 +997,62 @@ static void vmd_configure_cfgbar(struct vmd_dev *vmd)
>  		.end   = busn_end,
>  		.flags = IORESOURCE_BUS | IORESOURCE_PCI_FIXED,
>  	};
> +
> +	if (vmd->bus1_rootbus) {

[ ... ]

> +		/*
> +		 * Resize BUS0 CFGBAR range to make space for BUS1
> +		 * owned devices by adjusting range end with value stored in
> +		 * VMD_BUSRANGE0 register.
> +		 */
> +		vmd->resources[VMD_RES_CFGBAR].start = bus0_start;
> +		vmd->resources[VMD_RES_CFGBAR].end = bus0_end;

[Severity: High]
Will this resize of the CFGBAR start address cause vmd_domain_reset() to
skip the root buses?

When vmd_configure_cfgbar() resizes vmd->resources[VMD_RES_CFGBAR].start to
the secondary bus start (e.g., 0xE2), vmd_domain_reset() will later read
this modified start address:

drivers/pci/controller/vmd.c:vmd_domain_reset() {
	...
	for (bus = 0; bus < max_buses; bus++) {
		ecam_bus = vmd_bus_to_ecam(vmd, vmd->resources[0].start + bus);
	...
}

Consequently, the ECAM loop skips the primary root buses (0xE0 and 0xE1).
Does this leave their bridge windows with potentially stale BIOS configurations
that conflict with OS enumeration?

> +
> +		vmd->resources[VMD_RES_BUS1_CFGBAR] = (struct resource){
> +			.name = "VMD CFGBAR BUS1",
> +			.start = bus1_start,
> +			.end = bus1_end,
> +			.flags = IORESOURCE_BUS | IORESOURCE_PCI_FIXED,
> +		};
> +	}
> +
> +	return 0;
>  }

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260921164316.3860312-1-szymon.durawa@linux.intel.com?part=7

Re: [PATCH v8 8/8] PCI: vmd: Workaround for hardwired BUS1 bus number

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] pci: vmd: fake root bus number 0x80 causes resource conflict and breaks hotplug
--

commit 96fa114ac906af61b74b8a37fc6def3f417dbf5b
Author: Szymon Durawa <szymon.durawa@linux.intel.com>

    PCI: vmd: Workaround for hardwired BUS1 bus number

This commit implements a workaround for the VMD BUS1 root bus number being
fixed to 0x80 in hardware. It overrides the root bus number in vmd_create_bus()
and attempts to handle the translation during config accesses and ACPI matching.

> diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
> index 51b8c2d9b7f0b..91c5b02bf91d1 100644
> --- a/drivers/pci/controller/vmd.c
> +++ b/drivers/pci/controller/vmd.c
> @@ -467,10 +467,23 @@ static unsigned int vmd_bus_to_ecam(struct vmd_dev *vmd, unsigned int busnr)
>  static void __iomem *vmd_cfg_addr(struct vmd_dev *vmd, struct pci_bus *bus,
>  				  unsigned int devfn, int reg, int len)
>  {
[ ... ]
> +	if (vmd->bus1_rootbus && bus->number == VMD_PRIMARY_BUS1)
> +		bus_number = vmd->busn_start[VMD_BUS_1];
> +	else
> +		bus_number = bus->number;

[Severity: High]
Does this break configuration space accesses for dynamically assigned downstream
bus numbers?

During dynamic bus assignment like hotplug or SR-IOV on root ports behind
VMD BUS1, the PCI core sees 0x80 as the parent bus number and assigns child
buses starting from 0x81 instead of the physical downstream window (e.g., 0xE1).

When accessing these dynamically assigned buses, this code falls through to the
untranslated bus->number (e.g., 0x81). It then subtracts the physical ECAM
start (e.g., 0xE0) from 0x81, leading to a massive unsigned underflow
(0xFFFFFFA1).

This out-of-bounds offset will cause vmd_cfg_addr() to return NULL, silently
breaking all configuration accesses.

[ ... ]
> @@ -1188,6 +1223,7 @@ static int vmd_create_bus(struct vmd_dev *vmd, enum vmd_rootbus bus_number,
>  			  struct pci_sysdata *sd, resource_size_t *offset,
>  			  bool vmd_in_guest, u8 primary)
>  {
[ ... ]
> +	/*
> +	 * Workaround: set VMD_BUS_1's bus number to VMD_PRIMARY_BUS1 so it
> +	 * matches vmd->bus[VMD_BUS_1]->primary and bypasses bus number
> +	 * reconfiguration.
> +	 */
> +	if (bus_number == VMD_BUS_1 && vmd->bus1_rootbus)
> +		root_busnr = VMD_PRIMARY_BUS1;

[Severity: High]
Does overriding the root bus number to 0x80 here initiate a resource conflict
that breaks the dynamic allocator's base range?

By overriding the BUS1 root bus number to 0x80 instead of its physical ECAM
base (e.g., 0xE1), pci_register_host_bridge() requests the bus resource range
[0x80-0xFF]. This overlaps with BUS0's physical range (e.g., [0xE0-0xF0]).

pci_bus_insert_busn_res() detects this conflict and fails to attach the bus
resource of BUS1 to the domain, breaking hotplug and SR-IOV completely on
VMD BUS1.

> +	else
> +		root_busnr = vmd->busn_start[bus_number];
> +
> +	vmd_bus = pci_create_root_bus(&vmd->dev->dev, root_busnr, &vmd_ops, sd,
>  				      &resources);

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260921164316.3860312-1-szymon.durawa@linux.intel.com?part=8

[PATCH v8 0/8] VMD add second root bus support

[Szymon Durawa]

This series adds support for a second VMD root bus.

Today the VMD driver can enumerate and manage devices only behind the
first VMD root bus. Starting with Intel Arrow Lake, VMD may expose a
second root bus (BUS1), and devices behind that bus are not discovered
by the current driver.

The hardware model for Intel Arrow Lake VMD is slightly different
from earlier VMD generations:

- CFGBAR, MEMBAR1, and MEMBAR2 are shared by both VMD root buses.
- BUS1 uses a fixed primary bus number in hardware.
- Interrupt handling remains based on the existing VMD MSI/MSI-X and
  irq_domain model; this series does not introduce a separate interrupt
  domain for BUS1.

To support that layout, the series first refactors the current code so
resource setup, bus creation, and enumeration are split into helpers.
It then partitions the shared CFGBAR/MEMBAR apertures between BUS0 and
BUS1, creates a second root bus, and adds a VMD-specific workaround for
the fixed BUS1 primary bus number so PCI core bridge scanning does not
reassign BUS1 and lose subordinate device discovery.

Patch overview:

- Patches 1-4 extract helper functions for bus enumeration, CFGBAR and
  MEMBAR configuration, and root-bus creation.
- Patch 5 replaces hardcoded resource indices and constants with enums
  and defines.
- Patch 6 converts single-bus state into per-root-bus arrays.
- Patch 7 adds BUS1 resource partitioning and second-root-bus support.
- Patch 8 keeps the BUS1 root bus number aligned with the hardware-fixed
  primary bus number so PCI core bridge scanning does not reconfigure it
  and lose BUS1 subordinate enumeration.

Tested on an Arrow Lake platform with NVMe devices present behind both
VMD root buses. With the series applied, devices behind BUS0 and BUS1
are enumerated and become visible to the kernel. Without patch 8,
devices behind BUS1 are not discovered.

Changes from v1:
- splitting series into more commits, requested by Bjorn
- adding helper functions, suggested by Bjorn
- minor typos and unclear wording updated, suggested by Bjorn

Changes from v2:
- wording update in commit logs, suggested by Bjorn

Changes from v3:
- using GENMASK() instead of manual bits shifting, suggested by Bjorn
- converting decimal number to hex representation, suggested by Bjorn
- wording update in commit logs, suggested by Bjorn

Changes from v4:
- Update Dan's email address
- Resending the whole series to the correct mailing list

Changes from v5:
- Fixed correctness issues in BUS/resource layout in patches 7 and 8
- Fixed lifetime and use-after-free issues in patches 3 and 7
- Fixed resource tree integrity issues in patch 7
- Fixed error-path leaks across patches 3, 4, and 5
- Fixed allocation failure handling in patch 3
- Note on Sashiko feedback: Pre-existing warnings (unrelated to
  this feature series) were left untouched.

Changes from v6:
- Refactoring the series to align with recent changes in link below:
 https://patchwork.kernel.org/project/linux-pci/patch/20260629165025.268836-1-nirmal.patel@linux.intel.com/
- Fixed double free of the IRQ domain and emulated domain number
- Fixed a race with concurrent readers when modifying sibling pointers
- Fixed the MEMBAR2 split equations in
  vmd_configure_membar1_membar2()
- Fixed hardcoded resource indices in vmd_configure_membar()

Changes from v7:
- wording update in commit logs in patch 8, suggested by Bjorn

Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Dan Williams <djbw@kernel.org>
Cc: Lukas Wunner <lukas@wunner.de>
Cc: linux-pci@vger.kernel.org
Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>

Szymon Durawa (8):
  PCI: vmd: Add vmd_bus_enumeration() helper function
  PCI: vmd: Add vmd_configure_cfgbar() helper function
  PCI: vmd: Add vmd_configure_membar() and
    vmd_configure_membar1_membar2()
  PCI: vmd: Add vmd_create_bus()
  PCI: vmd: Replace hardcoded values with enum and defines
  PCI: vmd: Convert bus and busn_start to an array
  PCI: vmd: Add support for second rootbus under VMD
  PCI: vmd: Workaround for hardwired BUS1 bus number

 drivers/pci/controller/vmd.c | 633 ++++++++++++++++++++++++++++-------
 1 file changed, 510 insertions(+), 123 deletions(-)

-- 
2.43.0

[PATCH v8 1/8] PCI: vmd: Add vmd_bus_enumeration() helper function

[Szymon Durawa]

Move the vmd bus enumeration code to a new helper vmd_bus_enumeration().
No functional changes.

Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>
---
 drivers/pci/controller/vmd.c | 89 ++++++++++++++++++++----------------
 1 file changed, 49 insertions(+), 40 deletions(-)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index 241023ecf677..57d5e917dc45 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -905,6 +905,54 @@ static int vmd_pm_enable_quirk(struct pci_dev *pdev, void *userdata)
 	return 0;
 }
 
+static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features)
+{
+	struct pci_bus *child;
+	struct pci_dev *dev;
+	int ret;
+
+	vmd_acpi_begin();
+
+	pci_scan_child_bus(bus);
+	vmd_domain_reset(vmd_from_bus(bus));
+
+	/*
+	 * When Intel VMD is enabled, the OS does not discover the Root Ports
+	 * owned by Intel VMD within the MMCFG space. pci_reset_bus() applies
+	 * a reset to the parent of the PCI device supplied as argument. This
+	 * is why we pass a child device, so the reset can be triggered at
+	 * the Intel bridge level and propagated to all the children in the
+	 * hierarchy.
+	 */
+	list_for_each_entry(child, &bus->children, node) {
+		if (!list_empty(&child->devices)) {
+			dev = list_first_entry(&child->devices, struct pci_dev,
+					       bus_list);
+			ret = pci_reset_bus(dev);
+			if (ret)
+				pci_warn(dev, "can't reset device: %d\n", ret);
+
+			break;
+		}
+	}
+
+	pci_assign_unassigned_bus_resources(bus);
+
+	pci_walk_bus(bus, vmd_pm_enable_quirk, &features);
+
+	/*
+	 * VMD root buses are virtual and don't return true on pci_is_pcie()
+	 * and will fail pcie_bus_configure_settings() early. It can instead be
+	 * run on each of the real root ports.
+	 */
+	list_for_each_entry(child, &bus->children, node)
+		pcie_bus_configure_settings(child);
+
+	pci_bus_add_devices(bus);
+
+	vmd_acpi_end();
+}
+
 static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 {
 	struct pci_sysdata *sd = &vmd->sysdata;
@@ -915,8 +963,6 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	resource_size_t offset[2] = {0};
 	resource_size_t membar2_offset = 0x2000;
 	resource_size_t busn_end;
-	struct pci_bus *child;
-	struct pci_dev *dev;
 	bool vmd_in_guest;
 	int ret;
 
@@ -1047,45 +1093,8 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
 			       "domain"), "Can't create symlink to domain\n");
 
-	vmd_acpi_begin();
-
-	pci_scan_child_bus(vmd->bus);
-	vmd_domain_reset(vmd);
+	vmd_bus_enumeration(vmd->bus, features);
 
-	/* When Intel VMD is enabled, the OS does not discover the Root Ports
-	 * owned by Intel VMD within the MMCFG space. pci_reset_bus() applies
-	 * a reset to the parent of the PCI device supplied as argument. This
-	 * is why we pass a child device, so the reset can be triggered at
-	 * the Intel bridge level and propagated to all the children in the
-	 * hierarchy.
-	 */
-	list_for_each_entry(child, &vmd->bus->children, node) {
-		if (!list_empty(&child->devices)) {
-			dev = list_first_entry(&child->devices,
-					       struct pci_dev, bus_list);
-			ret = pci_reset_bus(dev);
-			if (ret)
-				pci_warn(dev, "can't reset device: %d\n", ret);
-
-			break;
-		}
-	}
-
-	pci_assign_unassigned_bus_resources(vmd->bus);
-
-	pci_walk_bus(vmd->bus, vmd_pm_enable_quirk, &features);
-
-	/*
-	 * VMD root buses are virtual and don't return true on pci_is_pcie()
-	 * and will fail pcie_bus_configure_settings() early. It can instead be
-	 * run on each of the real root ports.
-	 */
-	list_for_each_entry(child, &vmd->bus->children, node)
-		pcie_bus_configure_settings(child);
-
-	pci_bus_add_devices(vmd->bus);
-
-	vmd_acpi_end();
 	return 0;
 }
 
-- 
2.43.0

[PATCH v8 2/8] PCI: vmd: Add vmd_configure_cfgbar() helper function

[Szymon Durawa]

Move the VMD CFGBAR initialization code to a new helper
vmd_configure_cfgbar(). No functional changes.

Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>
---
 drivers/pci/controller/vmd.c | 28 +++++++++++++++++-----------
 1 file changed, 17 insertions(+), 11 deletions(-)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index 57d5e917dc45..2a8c10604c82 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -905,6 +905,22 @@ static int vmd_pm_enable_quirk(struct pci_dev *pdev, void *userdata)
 	return 0;
 }
 
+static void vmd_configure_cfgbar(struct vmd_dev *vmd)
+{
+	struct resource *res = &vmd->dev->resource[VMD_CFGBAR];
+	resource_size_t busn_end;
+
+	/* Do not let resource[0] end go out of bound.*/
+	busn_end = vmd->busn_start + (resource_size(res) >> 20) - 1;
+	busn_end = min_t(resource_size_t, busn_end, 0xff);
+	vmd->resources[0] = (struct resource) {
+		.name  = "VMD CFGBAR",
+		.start = vmd->busn_start,
+		.end   = busn_end,
+		.flags = IORESOURCE_BUS | IORESOURCE_PCI_FIXED,
+	};
+}
+
 static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features)
 {
 	struct pci_bus *child;
@@ -962,7 +978,6 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	LIST_HEAD(resources);
 	resource_size_t offset[2] = {0};
 	resource_size_t membar2_offset = 0x2000;
-	resource_size_t busn_end;
 	bool vmd_in_guest;
 	int ret;
 
@@ -971,16 +986,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	if (ret)
 		return ret;
 
-	/* Do not let resource[0] end go out of bounds */
-	res = &vmd->dev->resource[VMD_CFGBAR];
-	busn_end = vmd->busn_start + (resource_size(res) >> 20) - 1;
-	busn_end = min_t(resource_size_t, busn_end, 0xff);
-	vmd->resources[0] = (struct resource) {
-		.name  = "VMD CFGBAR",
-		.start = vmd->busn_start,
-		.end   = busn_end,
-		.flags = IORESOURCE_BUS | IORESOURCE_PCI_FIXED,
-	};
+	vmd_configure_cfgbar(vmd);
 
 	/*
 	 * If the window is below 4GB, clear IORESOURCE_MEM_64 so we can
-- 
2.43.0

[PATCH v8 3/8] PCI: vmd: Add vmd_configure_membar() and vmd_configure_membar1_membar2()

[Szymon Durawa]

Move the MEMBAR1 and MEMBAR2 registry initialization code to new helpers
vmd_configure_membar() and vmd_configure_membar1_membar2().
The refactor preserves MEMBAR address/offset/flags programming, but
resource name construction is now generated in the helper ("VMD MEMBAR%d")
instead of using two separate string literals.

Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>
---
 drivers/pci/controller/vmd.c | 98 +++++++++++++++++++++++++-----------
 1 file changed, 68 insertions(+), 30 deletions(-)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index 2a8c10604c82..ed6d03656885 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -921,6 +921,69 @@ static void vmd_configure_cfgbar(struct vmd_dev *vmd)
 	};
 }
 
+/*
+ * vmd_configure_membar - Configure VMD MemBAR register, which points
+ * to MMIO address assigned by the OS or BIOS.
+ * @vmd: the VMD device
+ * @resource_number: resource buffer number to be filled in
+ * @membar_number: number of the MemBAR
+ * @start_offset: 4K aligned offset applied to start of VMD’s MEMBAR MMIO space
+ * @end_offset: 4K aligned offset applied to end of VMD’s MEMBAR MMIO space
+ *
+ * Function fills resource buffer inside the VMD structure.
+ *
+ * Return: 0 on success, -ENOMEM on allocation failure.
+ */
+static int vmd_configure_membar(struct vmd_dev *vmd, u8 resource_number,
+				u8 membar_number, resource_size_t start_offset,
+				resource_size_t end_offset)
+{
+	char *name;
+	u32 upper_bits;
+	unsigned long flags;
+
+	struct resource *res = &vmd->dev->resource[membar_number];
+
+	upper_bits = upper_32_bits(res->end);
+	flags = res->flags & ~IORESOURCE_SIZEALIGN;
+	if (!upper_bits)
+		flags &= ~IORESOURCE_MEM_64;
+
+	name = devm_kasprintf(&vmd->dev->dev, GFP_KERNEL, "VMD MEMBAR%d",
+			      resource_number);
+	if (!name)
+		return -ENOMEM;
+
+	vmd->resources[resource_number] = (struct resource){
+		.name = name,
+		.start = res->start + start_offset,
+		.end = res->end - end_offset,
+		.flags = flags,
+		.parent = res,
+	};
+
+	return 0;
+}
+
+static int vmd_configure_membar1_membar2(struct vmd_dev *vmd,
+					 resource_size_t mbar2_ofs)
+{
+	int ret;
+
+	ret = vmd_configure_membar(vmd, 1, VMD_MEMBAR1, 0, 0);
+	if (ret)
+		return ret;
+
+	ret = vmd_configure_membar(vmd, 2, VMD_MEMBAR2, mbar2_ofs, 0);
+	if (ret) {
+		devm_kfree(&vmd->dev->dev, (void *)vmd->resources[1].name);
+		memset(&vmd->resources[1], 0, sizeof(vmd->resources[1]));
+		return ret;
+	}
+
+	return 0;
+}
+
 static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features)
 {
 	struct pci_bus *child;
@@ -972,9 +1035,6 @@ static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features)
 static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 {
 	struct pci_sysdata *sd = &vmd->sysdata;
-	struct resource *res;
-	u32 upper_bits;
-	unsigned long flags;
 	LIST_HEAD(resources);
 	resource_size_t offset[2] = {0};
 	resource_size_t membar2_offset = 0x2000;
@@ -1000,36 +1060,14 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	 *
 	 * The only way we could use a 64-bit non-prefetchable MEMBAR is
 	 * if its address is <4GB so that we can convert it to a 32-bit
-	 * resource.  To be visible to the host OS, all VMD endpoints must
+	 * resource. To be visible to the host OS, all VMD endpoints must
 	 * be initially configured by platform BIOS, which includes setting
-	 * up these resources.  We can assume the device is configured
+	 * up these resources. We can assume the device is configured
 	 * according to the platform needs.
 	 */
-	res = &vmd->dev->resource[VMD_MEMBAR1];
-	upper_bits = upper_32_bits(res->end);
-	flags = res->flags & ~IORESOURCE_SIZEALIGN;
-	if (!upper_bits)
-		flags &= ~IORESOURCE_MEM_64;
-	vmd->resources[1] = (struct resource) {
-		.name  = "VMD MEMBAR1",
-		.start = res->start,
-		.end   = res->end,
-		.flags = flags,
-		.parent = res,
-	};
-
-	res = &vmd->dev->resource[VMD_MEMBAR2];
-	upper_bits = upper_32_bits(res->end);
-	flags = res->flags & ~IORESOURCE_SIZEALIGN;
-	if (!upper_bits)
-		flags &= ~IORESOURCE_MEM_64;
-	vmd->resources[2] = (struct resource) {
-		.name  = "VMD MEMBAR2",
-		.start = res->start + membar2_offset,
-		.end   = res->end,
-		.flags = flags,
-		.parent = res,
-	};
+	ret = vmd_configure_membar1_membar2(vmd, membar2_offset);
+	if (ret)
+		return ret;
 
 	/* Non-zero offset means guest/direct assign view. */
 	vmd_in_guest = offset[0] || offset[1];
-- 
2.43.0

[PATCH v8 4/8] PCI: vmd: Add vmd_create_bus()

[Szymon Durawa]

Move the VMD bus initialization code to a new helper vmd_create_bus().
No functional changes.

Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>
---
 drivers/pci/controller/vmd.c | 63 ++++++++++++++++++++++--------------
 1 file changed, 38 insertions(+), 25 deletions(-)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index ed6d03656885..8b59b369f126 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -984,6 +984,40 @@ static int vmd_configure_membar1_membar2(struct vmd_dev *vmd,
 	return 0;
 }
 
+static int vmd_create_bus(struct vmd_dev *vmd, struct pci_sysdata *sd,
+			  resource_size_t *offset, bool vmd_in_guest)
+{
+	LIST_HEAD(resources);
+
+	pci_add_resource(&resources, &vmd->resources[0]);
+	pci_add_resource_offset(&resources, &vmd->resources[1], offset[0]);
+	pci_add_resource_offset(&resources, &vmd->resources[2], offset[1]);
+
+	vmd->bus = pci_create_root_bus(&vmd->dev->dev, vmd->busn_start,
+				       &vmd_ops, sd, &resources);
+	if (!vmd->bus) {
+		pci_free_resource_list(&resources);
+		return -ENODEV;
+	}
+
+	/*
+	 * Don't copy _OSC control flags from root bridge if running in a VM, as
+	 * they don't reflect the physical root bridge capabilities.
+	 */
+	if (!vmd_in_guest)
+		vmd_copy_host_bridge_flags(pci_find_host_bridge(vmd->dev->bus),
+					   to_pci_host_bridge(vmd->bus->bridge));
+
+	vmd_attach_resources(vmd);
+	if (vmd->irq_domain)
+		dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
+	else
+		dev_set_msi_domain(&vmd->bus->dev,
+				   dev_get_msi_domain(&vmd->dev->dev));
+
+	return 0;
+}
+
 static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features)
 {
 	struct pci_bus *child;
@@ -1035,7 +1069,6 @@ static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features)
 static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 {
 	struct pci_sysdata *sd = &vmd->sysdata;
-	LIST_HEAD(resources);
 	resource_size_t offset[2] = {0};
 	resource_size_t membar2_offset = 0x2000;
 	bool vmd_in_guest;
@@ -1092,10 +1125,6 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 		vmd_set_msi_remapping(vmd, false);
 	}
 
-	pci_add_resource(&resources, &vmd->resources[0]);
-	pci_add_resource_offset(&resources, &vmd->resources[1], offset[0]);
-	pci_add_resource_offset(&resources, &vmd->resources[2], offset[1]);
-
 	sd->vmd_dev = vmd->dev;
 
 	/*
@@ -1110,30 +1139,14 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 
 	sd->node = pcibus_to_node(vmd->dev->bus);
 
-	vmd->bus = pci_create_root_bus(&vmd->dev->dev, vmd->busn_start,
-				       &vmd_ops, sd, &resources);
-	if (!vmd->bus) {
+	ret = vmd_create_bus(vmd, sd, offset, vmd_in_guest);
+	if (ret) {
+		pci_err(vmd->dev, "Can't create bus: %d\n", ret);
 		pci_bus_release_emul_domain_nr(sd->domain);
-		pci_free_resource_list(&resources);
 		vmd_remove_irq_domain(vmd);
-		return -ENODEV;
+		return ret;
 	}
 
-	/*
-	 * Don't copy _OSC control flags from root bridge if running in a VM, as
-	 * they don't reflect the physical root bridge capabilities.
-	 */
-	if (!vmd_in_guest)
-		vmd_copy_host_bridge_flags(pci_find_host_bridge(vmd->dev->bus),
-					 to_pci_host_bridge(vmd->bus->bridge));
-
-	vmd_attach_resources(vmd);
-	if (vmd->irq_domain)
-		dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
-	else
-		dev_set_msi_domain(&vmd->bus->dev,
-				   dev_get_msi_domain(&vmd->dev->dev));
-
 	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
 			       "domain"), "Can't create symlink to domain\n");
 
-- 
2.43.0

[PATCH v8 5/8] PCI: vmd: Replace hardcoded values with enum and defines

[Szymon Durawa]

Add enum vmd_resource type to replace hardcoded values. Add defines for
vmd bus start number based on VMD restriction value. No functional
changes.

Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>
---
 drivers/pci/controller/vmd.c | 45 ++++++++++++++++++++++++------------
 1 file changed, 30 insertions(+), 15 deletions(-)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index 8b59b369f126..b7c4e22deee5 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -26,6 +26,11 @@
 #define VMD_MEMBAR1	2
 #define VMD_MEMBAR2	4
 
+/* VMD restriction value determines secondary start bus number */
+#define VMD_RESTRICT_0_BUS_START 0x0
+#define VMD_RESTRICT_1_BUS_START 0x80
+#define VMD_RESTRICT_2_BUS_START 0xE0
+
 #define PCI_REG_VMCAP		0x40
 #define BUS_RESTRICT_CAP(vmcap)	(vmcap & 0x1)
 #define PCI_REG_VMCONFIG	0x44
@@ -43,6 +48,13 @@
 #define BASE_ID_REG_28C1		0x2840
 #define MEMBAR2_OFFSET_28C1		0x30d0
 
+enum vmd_resource {
+	VMD_RES_CFGBAR = 0, /* VMD Bus0 Config BAR */
+	VMD_RES_MBAR_1, /* VMD Bus0 Resource MemBAR 1 */
+	VMD_RES_MBAR_2, /* VMD Bus0 Resource MemBAR 2 */
+	VMD_RES_COUNT
+};
+
 enum vmd_features {
 	/*
 	 * Device may contain registers which hint the physical location of the
@@ -150,7 +162,7 @@ struct vmd_dev {
 	struct vmd_irq_list	*irqs;
 
 	struct pci_sysdata	sysdata;
-	struct resource		resources[3];
+	struct resource		resources[VMD_RES_COUNT];
 	struct irq_domain	*irq_domain;
 	struct pci_bus		*bus;
 	u8			busn_start;
@@ -562,7 +574,7 @@ static resource_size_t vmd_cfgbar_ecam_space(struct vmd_dev *vmd)
 }
 static void vmd_domain_reset(struct vmd_dev *vmd)
 {
-	u16 bus, max_buses = resource_size(&vmd->resources[0]);
+	u16 bus, max_buses = resource_size(&vmd->resources[VMD_RES_CFGBAR]);
 	u8 dev, functions, fn, hdr_type;
 	unsigned int ecam_bus;
 	char __iomem *base;
@@ -613,8 +625,8 @@ static void vmd_domain_reset(struct vmd_dev *vmd)
 
 static void vmd_attach_resources(struct vmd_dev *vmd)
 {
-	vmd->dev->resource[VMD_MEMBAR1].child = &vmd->resources[1];
-	vmd->dev->resource[VMD_MEMBAR2].child = &vmd->resources[2];
+	vmd->dev->resource[VMD_MEMBAR1].child = &vmd->resources[VMD_RES_MBAR_1];
+	vmd->dev->resource[VMD_MEMBAR2].child = &vmd->resources[VMD_RES_MBAR_2];
 }
 
 static void vmd_detach_resources(struct vmd_dev *vmd)
@@ -690,14 +702,14 @@ static int vmd_get_bus_number_start(struct vmd_dev *vmd)
 
 		switch (BUS_RESTRICT_CFG(reg)) {
 		case 0:
-			vmd->busn_start = 0;
+			vmd->busn_start = VMD_RESTRICT_0_BUS_START;
 			break;
 		case 1:
-			vmd->busn_start = 128;
+			vmd->busn_start = VMD_RESTRICT_1_BUS_START;
 			break;
 		case 3:
 		case 2:
-			vmd->busn_start = 224;
+			vmd->busn_start = VMD_RESTRICT_2_BUS_START;
 			break;
 		default:
 			pci_err(dev, "Unknown Bus Offset Setting (%d)\n",
@@ -913,7 +925,7 @@ static void vmd_configure_cfgbar(struct vmd_dev *vmd)
 	/* Do not let resource[0] end go out of bound.*/
 	busn_end = vmd->busn_start + (resource_size(res) >> 20) - 1;
 	busn_end = min_t(resource_size_t, busn_end, 0xff);
-	vmd->resources[0] = (struct resource) {
+	vmd->resources[VMD_RES_CFGBAR] = (struct resource) {
 		.name  = "VMD CFGBAR",
 		.start = vmd->busn_start,
 		.end   = busn_end,
@@ -970,14 +982,15 @@ static int vmd_configure_membar1_membar2(struct vmd_dev *vmd,
 {
 	int ret;
 
-	ret = vmd_configure_membar(vmd, 1, VMD_MEMBAR1, 0, 0);
+	ret = vmd_configure_membar(vmd, VMD_RES_MBAR_1, VMD_MEMBAR1, 0, 0);
 	if (ret)
 		return ret;
 
-	ret = vmd_configure_membar(vmd, 2, VMD_MEMBAR2, mbar2_ofs, 0);
+	ret = vmd_configure_membar(vmd, VMD_RES_MBAR_2, VMD_MEMBAR2, mbar2_ofs, 0);
 	if (ret) {
-		devm_kfree(&vmd->dev->dev, (void *)vmd->resources[1].name);
-		memset(&vmd->resources[1], 0, sizeof(vmd->resources[1]));
+		devm_kfree(&vmd->dev->dev, (void *)vmd->resources[VMD_RES_MBAR_1].name);
+		memset(&vmd->resources[VMD_RES_MBAR_1], 0,
+		       sizeof(vmd->resources[VMD_RES_MBAR_1]));
 		return ret;
 	}
 
@@ -989,9 +1002,11 @@ static int vmd_create_bus(struct vmd_dev *vmd, struct pci_sysdata *sd,
 {
 	LIST_HEAD(resources);
 
-	pci_add_resource(&resources, &vmd->resources[0]);
-	pci_add_resource_offset(&resources, &vmd->resources[1], offset[0]);
-	pci_add_resource_offset(&resources, &vmd->resources[2], offset[1]);
+	pci_add_resource(&resources, &vmd->resources[VMD_RES_CFGBAR]);
+	pci_add_resource_offset(&resources, &vmd->resources[VMD_RES_MBAR_1],
+				offset[0]);
+	pci_add_resource_offset(&resources, &vmd->resources[VMD_RES_MBAR_2],
+				offset[1]);
 
 	vmd->bus = pci_create_root_bus(&vmd->dev->dev, vmd->busn_start,
 				       &vmd_ops, sd, &resources);
-- 
2.43.0

[PATCH v8 6/8] PCI: vmd: Convert bus and busn_start to an array

[Szymon Durawa]

Convert bus and busn_start from scalar to an array to support
multiple VMD buses in the future. No functional changes.

Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>
---
 drivers/pci/controller/vmd.c | 47 +++++++++++++++++++++---------------
 1 file changed, 27 insertions(+), 20 deletions(-)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index b7c4e22deee5..c6b5e693916c 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -55,6 +55,11 @@ enum vmd_resource {
 	VMD_RES_COUNT
 };
 
+enum vmd_rootbus {
+	VMD_BUS_0 = 0,
+	VMD_BUS_COUNT
+};
+
 enum vmd_features {
 	/*
 	 * Device may contain registers which hint the physical location of the
@@ -164,8 +169,8 @@ struct vmd_dev {
 	struct pci_sysdata	sysdata;
 	struct resource		resources[VMD_RES_COUNT];
 	struct irq_domain	*irq_domain;
-	struct pci_bus		*bus;
-	u8			busn_start;
+	struct pci_bus		*bus[VMD_BUS_COUNT];
+	u8			busn_start[VMD_BUS_COUNT];
 	u8			first_vec;
 	char			*name;
 	int			instance;
@@ -425,7 +430,7 @@ static unsigned int vmd_bus_to_ecam(struct vmd_dev *vmd, unsigned int busnr)
 	if (!!(vmd->features & VMD_FEAT_USE_BIOS_INFO))
 		return busnr;
 
-	return busnr - vmd->busn_start;
+	return busnr - vmd->busn_start[VMD_BUS_0];
 }
 
 static void __iomem *vmd_cfg_addr(struct vmd_dev *vmd, struct pci_bus *bus,
@@ -702,14 +707,14 @@ static int vmd_get_bus_number_start(struct vmd_dev *vmd)
 
 		switch (BUS_RESTRICT_CFG(reg)) {
 		case 0:
-			vmd->busn_start = VMD_RESTRICT_0_BUS_START;
+			vmd->busn_start[VMD_BUS_0] = VMD_RESTRICT_0_BUS_START;
 			break;
 		case 1:
-			vmd->busn_start = VMD_RESTRICT_1_BUS_START;
+			vmd->busn_start[VMD_BUS_0] = VMD_RESTRICT_1_BUS_START;
 			break;
 		case 3:
 		case 2:
-			vmd->busn_start = VMD_RESTRICT_2_BUS_START;
+			vmd->busn_start[VMD_BUS_0] = VMD_RESTRICT_2_BUS_START;
 			break;
 		default:
 			pci_err(dev, "Unknown Bus Offset Setting (%d)\n",
@@ -746,7 +751,7 @@ static int vmd_get_bus_info_from_bar4(struct vmd_dev *vmd,
 	bar4_2840 = readq(bar4 + BASE_ID_REG_28C1);
 	base_id = bar4_2840 & 0xFFFFFF;
 	base_bus = base_id >> 8;
-	vmd->busn_start = base_bus;
+	vmd->busn_start[VMD_BUS_0] = base_bus;
 
 	/* Calculate offsets like vmd_get_phys_offsets() does */
 	if (phys1)
@@ -923,11 +928,11 @@ static void vmd_configure_cfgbar(struct vmd_dev *vmd)
 	resource_size_t busn_end;
 
 	/* Do not let resource[0] end go out of bound.*/
-	busn_end = vmd->busn_start + (resource_size(res) >> 20) - 1;
+	busn_end = vmd->busn_start[VMD_BUS_0] + (resource_size(res) >> 20) - 1;
 	busn_end = min_t(resource_size_t, busn_end, 0xff);
 	vmd->resources[VMD_RES_CFGBAR] = (struct resource) {
 		.name  = "VMD CFGBAR",
-		.start = vmd->busn_start,
+		.start = vmd->busn_start[VMD_BUS_0],
 		.end   = busn_end,
 		.flags = IORESOURCE_BUS | IORESOURCE_PCI_FIXED,
 	};
@@ -1008,9 +1013,10 @@ static int vmd_create_bus(struct vmd_dev *vmd, struct pci_sysdata *sd,
 	pci_add_resource_offset(&resources, &vmd->resources[VMD_RES_MBAR_2],
 				offset[1]);
 
-	vmd->bus = pci_create_root_bus(&vmd->dev->dev, vmd->busn_start,
-				       &vmd_ops, sd, &resources);
-	if (!vmd->bus) {
+	vmd->bus[VMD_BUS_0] = pci_create_root_bus(&vmd->dev->dev,
+						  vmd->busn_start[VMD_BUS_0],
+						  &vmd_ops, sd, &resources);
+	if (!vmd->bus[VMD_BUS_0]) {
 		pci_free_resource_list(&resources);
 		return -ENODEV;
 	}
@@ -1021,13 +1027,13 @@ static int vmd_create_bus(struct vmd_dev *vmd, struct pci_sysdata *sd,
 	 */
 	if (!vmd_in_guest)
 		vmd_copy_host_bridge_flags(pci_find_host_bridge(vmd->dev->bus),
-					   to_pci_host_bridge(vmd->bus->bridge));
+					   to_pci_host_bridge(vmd->bus[VMD_BUS_0]->bridge));
 
 	vmd_attach_resources(vmd);
 	if (vmd->irq_domain)
-		dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
+		dev_set_msi_domain(&vmd->bus[VMD_BUS_0]->dev, vmd->irq_domain);
 	else
-		dev_set_msi_domain(&vmd->bus->dev,
+		dev_set_msi_domain(&vmd->bus[VMD_BUS_0]->dev,
 				   dev_get_msi_domain(&vmd->dev->dev));
 
 	return 0;
@@ -1162,10 +1168,11 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 		return ret;
 	}
 
-	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
-			       "domain"), "Can't create symlink to domain\n");
+	WARN(sysfs_create_link(&vmd->dev->dev.kobj,
+			       &vmd->bus[VMD_BUS_0]->dev.kobj, "domain"),
+	     "Can't create symlink to domain\n");
 
-	vmd_bus_enumeration(vmd->bus, features);
+	vmd_bus_enumeration(vmd->bus[VMD_BUS_0], features);
 
 	return 0;
 }
@@ -1262,9 +1269,9 @@ static void vmd_remove(struct pci_dev *dev)
 {
 	struct vmd_dev *vmd = pci_get_drvdata(dev);
 
-	pci_stop_root_bus(vmd->bus);
+	pci_stop_root_bus(vmd->bus[VMD_BUS_0]);
 	sysfs_remove_link(&vmd->dev->dev.kobj, "domain");
-	pci_remove_root_bus(vmd->bus);
+	pci_remove_root_bus(vmd->bus[VMD_BUS_0]);
 	vmd_cleanup_srcu(vmd);
 	vmd_detach_resources(vmd);
 	vmd_remove_irq_domain(vmd);
-- 
2.43.0

[PATCH v8 7/8] PCI: vmd: Add support for second rootbus under VMD

[Szymon Durawa]

Starting from Intel Arrow Lake VMD enhancement introduces second root bus
support with fixed root bus number (0x80). It means that all 3 MMIO BARs
exposed by VMD are shared now between both buses (current BUS0 and
new BUS1).

Add new BUS1 enumeration and divide MMIO space to be shared between
both root buses. Due to enumeration issues with root bus hardwired to a
fixed non-zero value, this patch will work with a workaround proposed
in next patch. Without workaround user won't see attached devices for BUS1
root bus.

Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>
---
 drivers/pci/controller/vmd.c | 314 +++++++++++++++++++++++++++++++----
 1 file changed, 278 insertions(+), 36 deletions(-)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index c6b5e693916c..51b8c2d9b7f0 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -4,6 +4,7 @@
  * Copyright (c) 2015, Intel Corporation.
  */
 
+#include <linux/bitfield.h>
 #include <linux/device.h>
 #include <linux/interrupt.h>
 #include <linux/irq.h>
@@ -30,6 +31,7 @@
 #define VMD_RESTRICT_0_BUS_START 0x0
 #define VMD_RESTRICT_1_BUS_START 0x80
 #define VMD_RESTRICT_2_BUS_START 0xE0
+#define VMD_RESTRICT_3_BUS_START 0xE1
 
 #define PCI_REG_VMCAP		0x40
 #define BUS_RESTRICT_CAP(vmcap)	(vmcap & 0x1)
@@ -48,15 +50,36 @@
 #define BASE_ID_REG_28C1		0x2840
 #define MEMBAR2_OFFSET_28C1		0x30d0
 
+/* Primary Bus Number for VMD devices on root bus 0 */
+#define VMD_PRIMARY_BUS0    0x00
+/* Primary Bus Number for VMD devices on root bus 1 */
+#define VMD_PRIMARY_BUS1    0x80
+
+#define VMD_BUSRANGE0       0xc8
+#define VMD_BUSRANGE1       0xcc
+#define VMD_MEMBAR1_OFFSET  0xd0
+#define VMD_MEMBAR2_OFFSET1 0xd8
+#define VMD_MEMBAR2_OFFSET2 0xdc
+#define VMD_BUS_END(busr) FIELD_GET(GENMASK(15, 8), busr)
+#define VMD_BUS_START(busr) FIELD_GET(GENMASK(7, 0), busr)
+
+/*
+ * Add VMD resources for BUS1, it will share the same MMIO space with
+ * previous VMD resources.
+ */
 enum vmd_resource {
-	VMD_RES_CFGBAR = 0, /* VMD Bus0 Config BAR */
-	VMD_RES_MBAR_1, /* VMD Bus0 Resource MemBAR 1 */
-	VMD_RES_MBAR_2, /* VMD Bus0 Resource MemBAR 2 */
+	VMD_RES_CFGBAR = 0,  /* VMD Bus0 Config BAR */
+	VMD_RES_MBAR_1,      /* VMD Bus0 Resource MemBAR 1 */
+	VMD_RES_MBAR_2,      /* VMD Bus0 Resource MemBAR 2 */
+	VMD_RES_BUS1_CFGBAR, /* VMD Bus1 Config BAR */
+	VMD_RES_BUS1_MBAR_1, /* VMD Bus1 Resource MemBAR 1 */
+	VMD_RES_BUS1_MBAR_2, /* VMD Bus1 Resource MemBAR 2 */
 	VMD_RES_COUNT
 };
 
 enum vmd_rootbus {
 	VMD_BUS_0 = 0,
+	VMD_BUS_1,
 	VMD_BUS_COUNT
 };
 
@@ -109,6 +132,12 @@ enum vmd_features {
 	 * referred to as MEMBAR2 or MSI-X BAR.
 	 */
 	VMD_FEAT_USE_BIOS_INFO		= (1 << 6),
+
+	/*
+	 * Starting from Intel Arrow Lake, VMD devices have their VMD root ports
+	 * on the additional BUS1 root bus.
+	 */
+	VMD_FEAT_HAS_BUS1_ROOTBUS	= (1 << 7)
 };
 
 #define VMD_BIOS_PM_QUIRK_LTR	0x1003	/* 3145728 ns */
@@ -116,7 +145,8 @@ enum vmd_features {
 #define VMD_FEATS_CLIENT	(VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP |	\
 				 VMD_FEAT_HAS_BUS_RESTRICTIONS |	\
 				 VMD_FEAT_OFFSET_FIRST_VECTOR |		\
-				 VMD_FEAT_BIOS_PM_QUIRK)
+				 VMD_FEAT_BIOS_PM_QUIRK |		\
+				 VMD_FEAT_HAS_BUS1_ROOTBUS)
 
 static DEFINE_IDA(vmd_instance_ida);
 
@@ -175,6 +205,7 @@ struct vmd_dev {
 	char			*name;
 	int			instance;
 	unsigned long		features;
+	bool			bus1_rootbus;
 };
 
 static inline struct vmd_dev *vmd_from_bus(struct pci_bus *bus)
@@ -579,7 +610,8 @@ static resource_size_t vmd_cfgbar_ecam_space(struct vmd_dev *vmd)
 }
 static void vmd_domain_reset(struct vmd_dev *vmd)
 {
-	u16 bus, max_buses = resource_size(&vmd->resources[VMD_RES_CFGBAR]);
+	/* One ECAM bus consumes 1MB of CFGBAR space. */
+	u16 bus, max_buses = resource_size(&vmd->dev->resource[VMD_CFGBAR]) >> 20;
 	u8 dev, functions, fn, hdr_type;
 	unsigned int ecam_bus;
 	char __iomem *base;
@@ -632,12 +664,24 @@ static void vmd_attach_resources(struct vmd_dev *vmd)
 {
 	vmd->dev->resource[VMD_MEMBAR1].child = &vmd->resources[VMD_RES_MBAR_1];
 	vmd->dev->resource[VMD_MEMBAR2].child = &vmd->resources[VMD_RES_MBAR_2];
+
+	if (vmd->bus1_rootbus) {
+		vmd->resources[VMD_RES_MBAR_1].sibling =
+			&vmd->resources[VMD_RES_BUS1_MBAR_1];
+		vmd->resources[VMD_RES_MBAR_2].sibling =
+			&vmd->resources[VMD_RES_BUS1_MBAR_2];
+	}
 }
 
 static void vmd_detach_resources(struct vmd_dev *vmd)
 {
 	vmd->dev->resource[VMD_MEMBAR1].child = NULL;
 	vmd->dev->resource[VMD_MEMBAR2].child = NULL;
+
+	if (vmd->bus1_rootbus) {
+		vmd->resources[VMD_RES_MBAR_1].sibling = NULL;
+		vmd->resources[VMD_RES_MBAR_2].sibling = NULL;
+	}
 }
 
 static int vmd_get_phys_offsets(struct vmd_dev *vmd, bool native_hint,
@@ -694,7 +738,7 @@ static int vmd_get_phys_offsets(struct vmd_dev *vmd, bool native_hint,
 	return 0;
 }
 
-static int vmd_get_bus_number_start(struct vmd_dev *vmd)
+static int vmd_get_bus_number_start(struct vmd_dev *vmd, unsigned long features)
 {
 	struct pci_dev *dev = vmd->dev;
 	u16 reg;
@@ -712,10 +756,27 @@ static int vmd_get_bus_number_start(struct vmd_dev *vmd)
 		case 1:
 			vmd->busn_start[VMD_BUS_0] = VMD_RESTRICT_1_BUS_START;
 			break;
-		case 3:
 		case 2:
 			vmd->busn_start[VMD_BUS_0] = VMD_RESTRICT_2_BUS_START;
 			break;
+		case 3:
+			if (!(features & VMD_FEAT_HAS_BUS1_ROOTBUS)) {
+				pci_err(dev, "VMD Bus Restriction detected type %d, but BUS1 root bus is not supported, aborting.\n",
+					BUS_RESTRICT_CFG(reg));
+				return -ENODEV;
+			}
+
+			/*
+			 * Per Intel VMD specification (e.g., Arrow Lake-S):
+			 * - VMD (on root bus 0) secondary bus start number: 0xE0
+			 *	(devices span 0xE2..0xF0 via VMD_BUSRANGE0).
+			 * - VMD (on root bus 1) secondary bus start number: 0xE1
+			 *	(devices span 0xF1..0xFF via VMD_BUSRANGE1).
+			 */
+			vmd->busn_start[VMD_BUS_0] = VMD_RESTRICT_2_BUS_START;
+			vmd->busn_start[VMD_BUS_1] = VMD_RESTRICT_3_BUS_START;
+			vmd->bus1_rootbus = true;
+			break;
 		default:
 			pci_err(dev, "Unknown Bus Offset Setting (%d)\n",
 				BUS_RESTRICT_CFG(reg));
@@ -855,7 +916,7 @@ static int vmd_prepare_offsets_and_bus(struct vmd_dev *vmd,
 	 * limits the bus range to between 0-127, 128-255, or 224-255.
 	 */
 	if (features & VMD_FEAT_HAS_BUS_RESTRICTIONS) {
-		ret = vmd_get_bus_number_start(vmd);
+		ret = vmd_get_bus_number_start(vmd, features);
 		if (ret)
 			return ret;
 	}
@@ -922,7 +983,7 @@ static int vmd_pm_enable_quirk(struct pci_dev *pdev, void *userdata)
 	return 0;
 }
 
-static void vmd_configure_cfgbar(struct vmd_dev *vmd)
+static int vmd_configure_cfgbar(struct vmd_dev *vmd)
 {
 	struct resource *res = &vmd->dev->resource[VMD_CFGBAR];
 	resource_size_t busn_end;
@@ -936,6 +997,62 @@ static void vmd_configure_cfgbar(struct vmd_dev *vmd)
 		.end   = busn_end,
 		.flags = IORESOURCE_BUS | IORESOURCE_PCI_FIXED,
 	};
+
+	if (vmd->bus1_rootbus) {
+		int ret;
+		u16 bus0_range = 0;
+		u16 bus1_range = 0;
+		u8 bus0_start, bus0_end;
+		u8 bus1_start, bus1_end;
+
+		ret = pci_read_config_word(vmd->dev, VMD_BUSRANGE0, &bus0_range);
+		if (ret) {
+			pci_err(vmd->dev, "Failed to read VMD_BUSRANGE0: %d\n", ret);
+			return -EIO;
+		}
+
+		ret = pci_read_config_word(vmd->dev, VMD_BUSRANGE1, &bus1_range);
+		if (ret) {
+			pci_err(vmd->dev, "Failed to read VMD_BUSRANGE1: %d\n", ret);
+			return -EIO;
+		}
+
+		bus0_start = VMD_BUS_START(bus0_range);
+		bus0_end = VMD_BUS_END(bus0_range);
+		bus1_start = VMD_BUS_START(bus1_range);
+		bus1_end = VMD_BUS_END(bus1_range);
+
+		if (bus0_start > bus0_end || bus1_start > bus1_end) {
+			pci_err(vmd->dev,
+				"Invalid bus range(s): BUS0 [%02x-%02x], BUS1 [%02x-%02x]\n",
+				bus0_start, bus0_end, bus1_start, bus1_end);
+			return -EINVAL;
+		}
+
+		if (!(bus0_end < bus1_start || bus1_end < bus0_start)) {
+			pci_err(vmd->dev,
+				"Overlapping bus ranges: BUS0 [%02x-%02x], BUS1 [%02x-%02x]\n",
+				bus0_start, bus0_end, bus1_start, bus1_end);
+			return -EINVAL;
+		}
+
+		/*
+		 * Resize BUS0 CFGBAR range to make space for BUS1
+		 * owned devices by adjusting range end with value stored in
+		 * VMD_BUSRANGE0 register.
+		 */
+		vmd->resources[VMD_RES_CFGBAR].start = bus0_start;
+		vmd->resources[VMD_RES_CFGBAR].end = bus0_end;
+
+		vmd->resources[VMD_RES_BUS1_CFGBAR] = (struct resource){
+			.name = "VMD CFGBAR BUS1",
+			.start = bus1_start,
+			.end = bus1_end,
+			.flags = IORESOURCE_BUS | IORESOURCE_PCI_FIXED,
+		};
+	}
+
+	return 0;
 }
 
 /*
@@ -966,8 +1083,9 @@ static int vmd_configure_membar(struct vmd_dev *vmd, u8 resource_number,
 	if (!upper_bits)
 		flags &= ~IORESOURCE_MEM_64;
 
-	name = devm_kasprintf(&vmd->dev->dev, GFP_KERNEL, "VMD MEMBAR%d",
-			      resource_number);
+	name = devm_kasprintf(&vmd->dev->dev, GFP_KERNEL, "VMD MEMBAR%d %s",
+			      membar_number / 2,
+			      resource_number > VMD_RES_MBAR_2 ? "BUS1" : "");
 	if (!name)
 		return -ENOMEM;
 
@@ -987,55 +1105,134 @@ static int vmd_configure_membar1_membar2(struct vmd_dev *vmd,
 {
 	int ret;
 
-	ret = vmd_configure_membar(vmd, VMD_RES_MBAR_1, VMD_MEMBAR1, 0, 0);
-	if (ret)
-		return ret;
+	if (vmd->bus1_rootbus) {
+		u32 reg = 0;
+		u32 bus1_mbar1_ofs = 0;
+		u64 bus1_mbar2_ofs = 0;
+		resource_size_t mbar1_sz, mbar2_sz;
 
-	ret = vmd_configure_membar(vmd, VMD_RES_MBAR_2, VMD_MEMBAR2, mbar2_ofs, 0);
-	if (ret) {
-		devm_kfree(&vmd->dev->dev, (void *)vmd->resources[VMD_RES_MBAR_1].name);
-		memset(&vmd->resources[VMD_RES_MBAR_1], 0,
-		       sizeof(vmd->resources[VMD_RES_MBAR_1]));
-		return ret;
+		mbar1_sz = resource_size(&vmd->dev->resource[VMD_MEMBAR1]);
+		mbar2_sz = resource_size(&vmd->dev->resource[VMD_MEMBAR2]);
+
+		ret = pci_read_config_dword(vmd->dev, VMD_MEMBAR1_OFFSET,
+					    &bus1_mbar1_ofs);
+		if (ret)
+			return -EIO;
+
+		ret = pci_read_config_dword(vmd->dev, VMD_MEMBAR2_OFFSET1, &reg);
+		if (ret)
+			return -EIO;
+		bus1_mbar2_ofs = reg;
+
+		ret = pci_read_config_dword(vmd->dev, VMD_MEMBAR2_OFFSET2, &reg);
+		if (ret)
+			return -EIO;
+		bus1_mbar2_ofs |= (u64)reg << 32;
+
+		if (!bus1_mbar1_ofs || bus1_mbar1_ofs >= mbar1_sz) {
+			pci_err(vmd->dev, "Invalid MEMBAR1 offset %#llx (BAR size %#llx)\n",
+				(unsigned long long)bus1_mbar1_ofs,
+				(unsigned long long)mbar1_sz);
+			return -EINVAL;
+		}
+
+		if (!bus1_mbar2_ofs || mbar2_ofs >= mbar2_sz ||
+		    bus1_mbar2_ofs <= mbar2_ofs || bus1_mbar2_ofs >= mbar2_sz) {
+			pci_err(vmd->dev,
+				"Invalid MEMBAR2 split %#llx (shadow offset %#llx, BAR size %#llx)\n",
+				(unsigned long long)bus1_mbar2_ofs,
+				(unsigned long long)mbar2_ofs,
+				(unsigned long long)mbar2_sz);
+			return -EINVAL;
+		}
+
+		/*
+		 * Resize BUS MEMBAR1 and MEMBAR2 ranges to make space
+		 * for BUS1 owned devices by adjusting range end with values
+		 * stored in VMD_MEMBAR1_OFFSET and VMD_MEMBAR2_OFFSET registers
+		 */
+		ret = vmd_configure_membar(vmd, VMD_RES_MBAR_1, VMD_MEMBAR1, 0,
+					   mbar1_sz - bus1_mbar1_ofs);
+		if (ret)
+			return ret;
+
+		ret = vmd_configure_membar(vmd, VMD_RES_MBAR_2, VMD_MEMBAR2,
+					   mbar2_ofs, mbar2_sz - bus1_mbar2_ofs);
+		if (ret)
+			return ret;
+
+		ret = vmd_configure_membar(vmd, VMD_RES_BUS1_MBAR_1, VMD_MEMBAR1,
+					   bus1_mbar1_ofs, 0);
+		if (ret)
+			return ret;
+
+		ret = vmd_configure_membar(vmd, VMD_RES_BUS1_MBAR_2, VMD_MEMBAR2,
+					   bus1_mbar2_ofs, 0);
+		if (ret)
+			return ret;
+	} else {
+		ret = vmd_configure_membar(vmd, VMD_RES_MBAR_1, VMD_MEMBAR1, 0, 0);
+		if (ret)
+			return ret;
+
+		ret = vmd_configure_membar(vmd, VMD_RES_MBAR_2, VMD_MEMBAR2,
+					   mbar2_ofs, 0);
+		if (ret)
+			return ret;
 	}
 
 	return 0;
 }
 
-static int vmd_create_bus(struct vmd_dev *vmd, struct pci_sysdata *sd,
-			  resource_size_t *offset, bool vmd_in_guest)
+static int vmd_create_bus(struct vmd_dev *vmd, enum vmd_rootbus bus_number,
+			  struct pci_sysdata *sd, resource_size_t *offset,
+			  bool vmd_in_guest, u8 primary)
 {
+	u8 cfgbar = bus_number * 3;
+	u8 membar1 = cfgbar + 1;
+	u8 membar2 = cfgbar + 2;
+	struct pci_bus *vmd_bus;
 	LIST_HEAD(resources);
 
-	pci_add_resource(&resources, &vmd->resources[VMD_RES_CFGBAR]);
-	pci_add_resource_offset(&resources, &vmd->resources[VMD_RES_MBAR_1],
+	pci_add_resource(&resources, &vmd->resources[cfgbar]);
+	pci_add_resource_offset(&resources, &vmd->resources[membar1],
 				offset[0]);
-	pci_add_resource_offset(&resources, &vmd->resources[VMD_RES_MBAR_2],
+	pci_add_resource_offset(&resources, &vmd->resources[membar2],
 				offset[1]);
 
-	vmd->bus[VMD_BUS_0] = pci_create_root_bus(&vmd->dev->dev,
-						  vmd->busn_start[VMD_BUS_0],
-						  &vmd_ops, sd, &resources);
-	if (!vmd->bus[VMD_BUS_0]) {
+	vmd_bus = pci_create_root_bus(&vmd->dev->dev,
+				      vmd->busn_start[bus_number], &vmd_ops, sd,
+				      &resources);
+
+	if (!vmd_bus) {
 		pci_free_resource_list(&resources);
 		return -ENODEV;
 	}
 
+	/*
+	 * pci_create_root_bus() does not initialise bus->primary.
+	 * Set it here before any scanning so bridge traversal logic
+	 * sees the correct upstream bus number from the start.
+	 */
+	vmd_bus->primary = primary;
+
 	/*
 	 * Don't copy _OSC control flags from root bridge if running in a VM, as
 	 * they don't reflect the physical root bridge capabilities.
 	 */
 	if (!vmd_in_guest)
 		vmd_copy_host_bridge_flags(pci_find_host_bridge(vmd->dev->bus),
-					   to_pci_host_bridge(vmd->bus[VMD_BUS_0]->bridge));
+					   to_pci_host_bridge(vmd_bus->bridge));
 
 	vmd_attach_resources(vmd);
 	if (vmd->irq_domain)
-		dev_set_msi_domain(&vmd->bus[VMD_BUS_0]->dev, vmd->irq_domain);
+		dev_set_msi_domain(&vmd_bus->dev, vmd->irq_domain);
 	else
-		dev_set_msi_domain(&vmd->bus[VMD_BUS_0]->dev,
+		dev_set_msi_domain(&vmd_bus->dev,
 				   dev_get_msi_domain(&vmd->dev->dev));
 
+	vmd->bus[bus_number] = vmd_bus;
+
 	return 0;
 }
 
@@ -1048,7 +1245,15 @@ static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features)
 	vmd_acpi_begin();
 
 	pci_scan_child_bus(bus);
-	vmd_domain_reset(vmd_from_bus(bus));
+
+	/*
+	 * vmd_domain_reset() walks the full VMD CFGBAR aperture, so a single
+	 * invocation from BUS0 resets bridge windows for the whole VMD domain,
+	 * including BUS1. Running it again during BUS1 enumeration would
+	 * re-clobber windows already assigned for BUS0.
+	 */
+	if (bus->primary == VMD_PRIMARY_BUS0)
+		vmd_domain_reset(vmd_from_bus(bus));
 
 	/*
 	 * When Intel VMD is enabled, the OS does not discover the Root Ports
@@ -1100,7 +1305,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	if (ret)
 		return ret;
 
-	vmd_configure_cfgbar(vmd);
+	ret = vmd_configure_cfgbar(vmd);
+	if (ret)
+		return ret;
 
 	/*
 	 * If the window is below 4GB, clear IORESOURCE_MEM_64 so we can
@@ -1160,7 +1367,8 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 
 	sd->node = pcibus_to_node(vmd->dev->bus);
 
-	ret = vmd_create_bus(vmd, sd, offset, vmd_in_guest);
+	ret = vmd_create_bus(vmd, VMD_BUS_0, sd, offset, vmd_in_guest,
+			     VMD_PRIMARY_BUS0);
 	if (ret) {
 		pci_err(vmd->dev, "Can't create bus: %d\n", ret);
 		pci_bus_release_emul_domain_nr(sd->domain);
@@ -1172,8 +1380,35 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 			       &vmd->bus[VMD_BUS_0]->dev.kobj, "domain"),
 	     "Can't create symlink to domain\n");
 
+	if (vmd->bus1_rootbus) {
+		ret = vmd_create_bus(vmd, VMD_BUS_1, sd, offset, vmd_in_guest,
+				     VMD_PRIMARY_BUS1);
+		if (ret) {
+			pci_warn(vmd->dev,
+				 "Can't create BUS1: %d, continuing with BUS0 only\n",
+				 ret);
+
+			vmd->resources[VMD_RES_MBAR_1].sibling = NULL;
+			vmd->resources[VMD_RES_MBAR_2].sibling = NULL;
+			vmd->resources[VMD_RES_BUS1_CFGBAR] = (struct resource){};
+			vmd->resources[VMD_RES_BUS1_MBAR_1] = (struct resource){};
+			vmd->resources[VMD_RES_BUS1_MBAR_2] = (struct resource){};
+			vmd->bus1_rootbus = false;
+			vmd->bus[VMD_BUS_1] = NULL;
+		}
+
+		if (vmd->bus1_rootbus)
+			WARN(sysfs_create_link(&vmd->dev->dev.kobj,
+					       &vmd->bus[VMD_BUS_1]->dev.kobj,
+					       "domain1"),
+			     "Can't create symlink to domain1\n");
+	}
+
 	vmd_bus_enumeration(vmd->bus[VMD_BUS_0], features);
 
+	if (vmd->bus1_rootbus)
+		vmd_bus_enumeration(vmd->bus[VMD_BUS_1], features);
+
 	return 0;
 }
 
@@ -1272,6 +1507,13 @@ static void vmd_remove(struct pci_dev *dev)
 	pci_stop_root_bus(vmd->bus[VMD_BUS_0]);
 	sysfs_remove_link(&vmd->dev->dev.kobj, "domain");
 	pci_remove_root_bus(vmd->bus[VMD_BUS_0]);
+
+	if (vmd->bus1_rootbus && vmd->bus[VMD_BUS_1]) {
+		pci_stop_root_bus(vmd->bus[VMD_BUS_1]);
+		sysfs_remove_link(&vmd->dev->dev.kobj, "domain1");
+		pci_remove_root_bus(vmd->bus[VMD_BUS_1]);
+	}
+
 	vmd_cleanup_srcu(vmd);
 	vmd_detach_resources(vmd);
 	vmd_remove_irq_domain(vmd);
@@ -1372,4 +1614,4 @@ module_pci_driver(vmd_drv);
 MODULE_AUTHOR("Intel Corporation");
 MODULE_DESCRIPTION("Volume Management Device driver");
 MODULE_LICENSE("GPL v2");
-MODULE_VERSION("0.6");
+MODULE_VERSION("0.7");
-- 
2.43.0

[PATCH v8 8/8] PCI: vmd: Workaround for hardwired BUS1 bus number

[Szymon Durawa]

The VMD BUS1 root bus number is fixed in hardware to 0x80. When the
PCI core sees a non-zero value (0x80) for the BUS1 primary bus number, it
treats the bridge configuration as invalid and attempts to reassign the
bus number.

This is especially problematic when a second root bus is present. In
the first pass of pci_scan_bridge(), the root bus number is temporarily
deconfigured so it can be reassigned to 0x00 in the second pass. As a
result, no subordinate bus behind the VMD BUS1 root bus is discovered.

To avoid that reconfiguration, the BUS1 bus number must match the BUS1
primary bus number. Keep the VMD BUS1 root bus number fixed instead of
reassigning it.

Log snippet without workaround:

  vmd 0000:00:0e.0: PCI host bridge to bus 10000:e1
  pci_bus 10000:e1: busn_res: can not insert [bus e1-ff] under domain [bus
   00-ff] (conflicts with (null) [bus e0-f0])
  pci_bus 10000:e1: root bus resource [bus f1-ff]
  pci_bus 10000:e1: root bus resource [mem 0x8c800000-0x8cffffff]
  pci_bus 10000:e1: root bus resource [mem 0x701b802000-0x701bffffff 64bit]
  pci_bus 10000:e1: scanning bus
  pci (null): Looking for ACPI companion (address 0x80e0ffff)
  pci 10000:e1:1c.0: [8086:7f38] type 01 class 0x060400 PCIe Root Port
  pci 10000:e1:1c.0: PCI bridge to [bus f1]
  pci 10000:e1:1c.0: scanning [bus f1-f1] behind bridge, pass 0
  pci 10000:e1:1c.0: primary 80, bus->number e1.
  pci 10000:e1:1c.0: bridge configuration invalid
   ([bus f1-f1]), reconfiguring
  pci 10000:e1:1c.0: scanning [bus 00-00] behind bridge, pass 1

Log snippet with workaround implemented:

  vmd 0000:00:0e.0: PCI host bridge to bus 10000:e0
  pci_bus 10000:e0: root bus resource [bus e2-f0]
  pci_bus 10000:e0: root bus resource [mem 0x8c000000-0x8c7fffff]
  pci_bus 10000:e0: root bus resource [mem 0x701b002000-0x701b801fff 64bit]
  vmd 0000:00:0e.0: PCI host bridge to bus 10000:80
  pci_bus 10000:80: busn_res: can not insert [bus 80-ff] under domain [bus
   00-ff] (conflicts with (null) [bus e0-f0])
  pci_bus 10000:80: root bus resource [bus f1-ff]
  pci_bus 10000:80: root bus resource [mem 0x8c800000-0x8cffffff]
  pci_bus 10000:80: root bus resource [mem 0x701b802000-0x701bffffff 64bit]
  pci 10000:e2:00.0: [144d:a80a] type 00 class 0x010802 PCIe Endpoint
  pci 10000:e2:00.0: BAR 0 [mem 0x8c000000-0x8c003fff 64bit]
  pci 10000:e0:01.0: PCI bridge to [bus e2]
  pci 10000:e0:01.0: Primary bus is hard wired to 0
  pci 10000:80:1d.0: [8086:09ab] type 00 class 0x088000
   conventional PCI endpoint
  pci 10000:80:1d.4: [8086:7f34] type 01 class 0x060400 PCIe Root Port
  pci 10000:80:1d.4: PCI bridge to [bus f1]
  pci 10000:80:1d.4: PME# supported from D0 D3hot D3cold
  pci 10000:f1:00.0: [144d:a810] type 00 class 0x010802 PCIe Endpoint
  pci 10000:f1:00.0: BAR 0 [mem 0x8c900000-0x8c903fff 64bit]
  pci 10000:80:1d.4: PCI bridge to [bus f1]
  vmd 0000:00:0e.0: Bound to PCI domain 10000

Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>
---
 drivers/pci/controller/vmd.c | 69 ++++++++++++++++++++++++++++++++----
 1 file changed, 63 insertions(+), 6 deletions(-)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index 51b8c2d9b7f0..91c5b02bf91d 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -467,10 +467,23 @@ static unsigned int vmd_bus_to_ecam(struct vmd_dev *vmd, unsigned int busnr)
 static void __iomem *vmd_cfg_addr(struct vmd_dev *vmd, struct pci_bus *bus,
 				  unsigned int devfn, int reg, int len)
 {
+	unsigned char bus_number;
 	unsigned int busnr_ecam;
 	u32 offset;
 
-	busnr_ecam = vmd_bus_to_ecam(vmd, bus->number);
+	/*
+	 * Remap ONLY the virtual BUS1 root bus number (0x80) to its physical
+	 * CFGBAR start aperture (0xE1). Downstream subordinate buses behind
+	 * the root port are assigned physical numbers (0xF1..0xFF per
+	 * VMD_BUSRANGE1) and must NOT be remapped, otherwise child endpoint
+	 * accesses would target the root port rather than their own ECAM space.
+	 */
+	if (vmd->bus1_rootbus && bus->number == VMD_PRIMARY_BUS1)
+		bus_number = vmd->busn_start[VMD_BUS_1];
+	else
+		bus_number = bus->number;
+
+	busnr_ecam = vmd_bus_to_ecam(vmd, bus_number);
 	offset = PCIE_ECAM_OFFSET(busnr_ecam, devfn, reg);
 
 	if (offset + len >= resource_size(&vmd->dev->resource[VMD_CFGBAR]))
@@ -550,18 +563,40 @@ static struct pci_ops vmd_ops = {
 static struct acpi_device *vmd_acpi_find_companion(struct pci_dev *pci_dev)
 {
 	struct pci_host_bridge *bridge;
-	u32 busnr, addr;
+	struct vmd_dev *vmd;
+	u32 addr;
+	int busnr;
+	u8 pci_bus_number;
+	u8 bridge_bus_number;
 
 	if (pci_dev->bus->ops != &vmd_ops)
 		return NULL;
 
+	vmd = vmd_from_bus(pci_dev->bus);
 	bridge = pci_find_host_bridge(pci_dev->bus);
-	busnr = pci_dev->bus->number - bridge->bus->number;
+	pci_bus_number = pci_dev->bus->number;
+	bridge_bus_number = bridge->bus->number;
+
+	/*
+	 * BUS1 is registered with logical root number 0x80. For ACPI companion
+	 * matching, map only the logical root bus (0x80) to the physical
+	 * BUS1 start base (0xE1). Downstream child buses (0xF1..0xFF) already
+	 * reflect their physical bus numbers and must remain untranslated to
+	 * produce the correct relative depth against bridge_bus_number.
+	 */
+	if (vmd->bus1_rootbus && bridge->bus == vmd->bus[VMD_BUS_1]) {
+		bridge_bus_number = vmd->busn_start[VMD_BUS_1];
+		if (pci_bus_number == VMD_PRIMARY_BUS1)
+			pci_bus_number = vmd->busn_start[VMD_BUS_1];
+	}
+
+	busnr = pci_bus_number - bridge_bus_number;
+
 	/*
 	 * The address computation below is only applicable to relative bus
 	 * numbers below 32.
 	 */
-	if (busnr > 31)
+	if (busnr < 0 || busnr > 31)
 		return NULL;
 
 	addr = (busnr << 24) | ((u32)pci_dev->devfn << 16) | 0x8000FFFFU;
@@ -1188,6 +1223,7 @@ static int vmd_create_bus(struct vmd_dev *vmd, enum vmd_rootbus bus_number,
 			  struct pci_sysdata *sd, resource_size_t *offset,
 			  bool vmd_in_guest, u8 primary)
 {
+	u8 root_busnr;
 	u8 cfgbar = bus_number * 3;
 	u8 membar1 = cfgbar + 1;
 	u8 membar2 = cfgbar + 2;
@@ -1200,8 +1236,29 @@ static int vmd_create_bus(struct vmd_dev *vmd, enum vmd_rootbus bus_number,
 	pci_add_resource_offset(&resources, &vmd->resources[membar2],
 				offset[1]);
 
-	vmd_bus = pci_create_root_bus(&vmd->dev->dev,
-				      vmd->busn_start[bus_number], &vmd_ops, sd,
+	/*
+	 * Register BUS1 with its logical root number (0x80) up front so PCI core
+	 * bridge scanning does not see a post-registration bus-number mutation.
+	 *
+	 * This is a workaround for pci_scan_bridge_extend(). That code marks a
+	 * bridge as misconfigured when it detects a non-zero root bus number for
+	 * VMD BUS1, even though the hardware hardwires that value to 0x80.
+	 *
+	 * In the first pass of pci_scan_bridge(), the Primary Bus Number of Root
+	 * Ports on BUS1 is therefore deconfigured and then reassigned to 0x0 in
+	 * the second pass. As a result, no subordinate bus number behind
+	 * VMD BUS1 is found.
+	 *
+	 * Workaround: set VMD_BUS_1's bus number to VMD_PRIMARY_BUS1 so it
+	 * matches vmd->bus[VMD_BUS_1]->primary and bypasses bus number
+	 * reconfiguration.
+	 */
+	if (bus_number == VMD_BUS_1 && vmd->bus1_rootbus)
+		root_busnr = VMD_PRIMARY_BUS1;
+	else
+		root_busnr = vmd->busn_start[bus_number];
+
+	vmd_bus = pci_create_root_bus(&vmd->dev->dev, root_busnr, &vmd_ops, sd,
 				      &resources);
 
 	if (!vmd_bus) {
-- 
2.43.0

Re: [PATCH v8 0/8] VMD add second root bus support

[Manivannan Sadhasivam]

On Mon, Sep 21, 2026 at 04:43:06PM +0000, Szymon Durawa wrote:
> This series adds support for a second VMD root bus.
> 
> Today the VMD driver can enumerate and manage devices only behind the
> first VMD root bus. Starting with Intel Arrow Lake, VMD may expose a
> second root bus (BUS1), and devices behind that bus are not discovered
> by the current driver.
> 
> The hardware model for Intel Arrow Lake VMD is slightly different
> from earlier VMD generations:
> 
> - CFGBAR, MEMBAR1, and MEMBAR2 are shared by both VMD root buses.
> - BUS1 uses a fixed primary bus number in hardware.
> - Interrupt handling remains based on the existing VMD MSI/MSI-X and
>   irq_domain model; this series does not introduce a separate interrupt
>   domain for BUS1.
> 
> To support that layout, the series first refactors the current code so
> resource setup, bus creation, and enumeration are split into helpers.
> It then partitions the shared CFGBAR/MEMBAR apertures between BUS0 and
> BUS1, creates a second root bus, and adds a VMD-specific workaround for
> the fixed BUS1 primary bus number so PCI core bridge scanning does not
> reassign BUS1 and lose subordinate device discovery.
> 
> Patch overview:
> 
> - Patches 1-4 extract helper functions for bus enumeration, CFGBAR and
>   MEMBAR configuration, and root-bus creation.
> - Patch 5 replaces hardcoded resource indices and constants with enums
>   and defines.
> - Patch 6 converts single-bus state into per-root-bus arrays.
> - Patch 7 adds BUS1 resource partitioning and second-root-bus support.
> - Patch 8 keeps the BUS1 root bus number aligned with the hardware-fixed
>   primary bus number so PCI core bridge scanning does not reconfigure it
>   and lose BUS1 subordinate enumeration.
> 
> Tested on an Arrow Lake platform with NVMe devices present behind both
> VMD root buses. With the series applied, devices behind BUS0 and BUS1
> are enumerated and become visible to the kernel. Without patch 8,
> devices behind BUS1 are not discovered.
> 

./scripts/get_maintainer.pl drivers/pci/controller/vmd.c
Nirmal Patel <nirmal.patel@linux.intel.com> (maintainer:PCI DRIVER FOR INTEL VOLUME MANAGEMENT DEVICE (...)
Jonathan Derrick <jonathan.derrick@linux.dev> (reviewer:PCI DRIVER FOR INTEL VOLUME MANAGEMENT DEVICE (...)
Lorenzo Pieralisi <lpieralisi@kernel.org> (maintainer:PCI NATIVE HOST BRIDGE AND ENDPOINT DRIVERS)
"Krzysztof Wilczyński" <kwilczynski@kernel.org> (maintainer:PCI NATIVE HOST BRIDGE AND ENDPOINT DRIVERS)
Manivannan Sadhasivam <mani@kernel.org> (maintainer:PCI NATIVE HOST BRIDGE AND ENDPOINT DRIVERS)
Rob Herring <robh@kernel.org> (reviewer:PCI NATIVE HOST BRIDGE AND ENDPOINT DRIVERS)
Bjorn Helgaas <bhelgaas@google.com> (maintainer:PCI SUBSYSTEM)
linux-pci@vger.kernel.org (open list:PCI DRIVER FOR INTEL VOLUME MANAGEMENT DEVICE (...)
linux-kernel@vger.kernel.org (open list)

For some reason, VMD patch submitters are keep forgetting to CC relevant
maintianers/reviewers.

Anyhow, I'd need a review from either Nirmal or Jonathan.

- Mani

-- 
மணிவண்ணன் சதாசிவம்