Re: [PATCH v10 1/7] PCI: vmd: Add vmd_bus_enumeration() helper function

Re: [PATCH v10 1/7] PCI: vmd: Add vmd_bus_enumeration() helper function

[sashiko-bot]

> Move the vmd bus enumeration code to a new helper vmd_bus_enumeration().
> No functional changes.
> 
> Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
> Reviewed-by: Asutosh Pathak <asutosh.pathak@linux.intel.com>
> Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106184256.4130-1-szymon.durawa@linux.intel.com?part=1

Re: [PATCH v10 2/7] PCI: vmd: Add vmd_configure_cfgbar() helper function

[sashiko-bot]

> Move the VMD CFGBAR initialization code to a new helper
> vmd_configure_cfgbar(). No functional changes.
> 
> Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
> Reviewed-by: Asutosh Pathak <asutosh.pathak@linux.intel.com>
> Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106184256.4130-1-szymon.durawa@linux.intel.com?part=2

Re: [PATCH v10 3/7] PCI: vmd: Add vmd_configure_membar() and vmd_configure_membar1_membar2()

[sashiko-bot]

> Move the MEMBAR1 and MEMBAR2 registry initialization code to new helpers
> vmd_configure_membar() and vmd_configure_membar1_membar2().
> The refactor preserves MEMBAR address/offset/flags programming, but
> resource name construction is now generated in the helper ("VMD MEMBAR%d")
> instead of using two separate string literals.
> 
> Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
> Reviewed-by: Asutosh Pathak <asutosh.pathak@linux.intel.com>
> Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106184256.4130-1-szymon.durawa@linux.intel.com?part=3

Re: [PATCH v10 4/7] PCI: vmd: Add vmd_create_bus()

[sashiko-bot]

> Move the VMD bus initialization code to a new helper vmd_create_bus().
> No functional changes.
> 
> Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
> Reviewed-by: Asutosh Pathak <asutosh.pathak@linux.intel.com>
> Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106184256.4130-1-szymon.durawa@linux.intel.com?part=4

Re: [PATCH v10 5/7] PCI: vmd: Replace hardcoded values with enum and defines

[sashiko-bot]

> Add enum vmd_resource type to replace hardcoded values. Add defines for
> vmd bus start number based on VMD restriction value. No functional
> changes.
> 
> Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
> Reviewed-by: Asutosh Pathak <asutosh.pathak@linux.intel.com>
> Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106184256.4130-1-szymon.durawa@linux.intel.com?part=5

Re: [PATCH v10 6/7] PCI: vmd: Convert bus, busn_start and sysdata to an array

[sashiko-bot]

> Convert bus, busn_start and sysdata in vmd_dev struct from scalar
> to an array to support multiple VMD buses in the future.
> No functional changes.
> 
> Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
> Reviewed-by: Asutosh Pathak <asutosh.pathak@linux.intel.com>
> Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106184256.4130-1-szymon.durawa@linux.intel.com?part=6

Re: [PATCH v10 7/7] PCI: vmd: Add support for second rootbus under VMD

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 4 potential
issues to consider:
- [High] PCI: vmd: overlapping busn_res and unisolated CFGBAR in
  vmd_create_bus()
- [Medium] PCI: vmd: missing VMD_BUSRANGE bounds check in
  vmd_configure_cfgbar()
- [Medium] PCI: vmd: wrong relative bus number for BUS1 in
  vmd_acpi_find_companion()
- [Medium] PCI: vmd: leak of srcu_struct on error paths in
  vmd_enable_domain()
--

commit e9682a58316a29ebd9d43933c4903d99f8a94e4b
Author: Szymon Durawa <szymon.durawa@linux.intel.com>

PCI: vmd: Add support for second rootbus under VMD

This commit adds support for a second VMD root bus (BUS1) exposed via Bus
Restriction type 3 on Intel Arrow Lake and later platforms, partitioning
CFGBAR, MEMBAR1, and MEMBAR2 across two emulated PCI domains.

> diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
> index 5733faf972d10..6831d0a5c443a 100644
> --- a/drivers/pci/controller/vmd.c
> +++ b/drivers/pci/controller/vmd.c

[ ... ]

> @@ -989,6 +1133,62 @@ static void vmd_configure_cfgbar(struct vmd_dev *vmd)
>  		.end   = busn_end,
>  		.flags = IORESOURCE_BUS | IORESOURCE_PCI_FIXED,
>  	};
> +
> +	if (vmd->bus1_rootbus) {
> +		int ret;
> +		u16 bus0_range = 0;
> +		u16 bus1_range = 0;
> +		u8 bus0_start, bus0_end;
> +		u8 bus1_start, bus1_end;
> +
> +		ret = pci_read_config_word(vmd->dev, VMD_BUSRANGE0, &bus0_range);
> +		if (ret) {
> +			pci_err(vmd->dev, "Failed to read VMD_BUSRANGE0: %d\n", ret);
> +			return -EIO;
> +		}
> +
> +		ret = pci_read_config_word(vmd->dev, VMD_BUSRANGE1, &bus1_range);
> +		if (ret) {
> +			pci_err(vmd->dev, "Failed to read VMD_BUSRANGE1: %d\n", ret);
> +			return -EIO;
> +		}
> +
> +		bus0_start = VMD_BUS_START(bus0_range);
> +		bus0_end = VMD_BUS_END(bus0_range);
> +		bus1_start = VMD_BUS_START(bus1_range);
> +		bus1_end = VMD_BUS_END(bus1_range);
> +
> +		if (bus0_start > bus0_end || bus1_start > bus1_end) {
> +			pci_err(vmd->dev,
> +				"Invalid bus range(s): BUS0 [%02x-%02x], BUS1 [%02x-%02x]\n",
> +				bus0_start, bus0_end, bus1_start, bus1_end);
> +			return -EINVAL;
> +		}
> +
> +		if (!(bus0_end < bus1_start || bus1_end < bus0_start)) {
> +			pci_err(vmd->dev,
> +				"Overlapping bus ranges: BUS0 [%02x-%02x], BUS1 [%02x-%02x]\n",
> +				bus0_start, bus0_end, bus1_start, bus1_end);
> +			return -EINVAL;
> +		}
> +
> +		/*
> +		 * Resize BUS0 CFGBAR range to make space for BUS1
> +		 * owned devices by adjusting range end with value stored in
> +		 * VMD_BUSRANGE0 register.
> +		 */
> +		vmd->resources[VMD_RES_CFGBAR].start = bus0_start;
> +		vmd->resources[VMD_RES_CFGBAR].end = bus0_end;
> +
> +		vmd->resources[VMD_RES_BUS1_CFGBAR] = (struct resource){
> +			.name = "VMD CFGBAR BUS1",
> +			.start = bus1_start,
> +			.end = bus1_end,
> +			.flags = IORESOURCE_BUS | IORESOURCE_PCI_FIXED,
> +		};
> +	}
> +
> +	return 0;
>  }

[Severity: Medium]
Does vmd_configure_cfgbar() need to validate that bus0_start and bus1_start
are above vmd->busn_start[VMD_BUS_1] (0xe1) and that bus0_end and bus1_end
do not exceed busn_end?

If firmware leaves VMD_BUSRANGE0 or VMD_BUSRANGE1 at 0x0000 when one root
bus has no subordinate devices, or reports an end bus beyond the VMD_CFGBAR
aperture, the checks above still pass and overwrite the bus resources with
an inverted range such as [0xe0..0x00] or an out-of-bounds bus number.

[ ... ]

> @@ -1040,55 +1241,132 @@ static int vmd_configure_membar1_membar2(struct vmd_dev *vmd,
>  static int vmd_create_bus(struct vmd_dev *vmd, enum vmd_rootbus bus_number,
>  			  struct pci_sysdata *sd, resource_size_t *offset,
>  			  bool vmd_in_guest, u8 primary)
>  {
> +	u8 cfgbar = bus_number * 3;
> +	u8 membar1 = cfgbar + 1;
> +	u8 membar2 = cfgbar + 2;
> +	struct pci_bus *vmd_bus;
>  	LIST_HEAD(resources);
>  
> -	pci_add_resource(&resources, &vmd->resources[VMD_RES_CFGBAR]);
> -	pci_add_resource_offset(&resources, &vmd->resources[VMD_RES_MBAR_1],
> +	pci_add_resource(&resources, &vmd->resources[cfgbar]);
> +	pci_add_resource_offset(&resources, &vmd->resources[membar1],
>  				offset[0]);
> -	pci_add_resource_offset(&resources, &vmd->resources[VMD_RES_MBAR_2],
> +	pci_add_resource_offset(&resources, &vmd->resources[membar2],
>  				offset[1]);
>  
> -	vmd->bus[VMD_BUS_0] = pci_create_root_bus(&vmd->dev->dev,
> -						  vmd->busn_start[VMD_BUS_0],
> -						  &vmd_ops, sd, &resources);
> -	if (!vmd->bus[VMD_BUS_0]) {
> +	vmd_bus = pci_create_root_bus(&vmd->dev->dev,
> +				      vmd->busn_start[bus_number], &vmd_ops, sd,
> +				      &resources);

[Severity: High]
Can VMD_BUS_0 and VMD_BUS_1 allocate and scan each other's ECAM bus numbers
when a root port bridge is unconfigured?

vmd_configure_cfgbar() sets vmd->resources[VMD_RES_CFGBAR] to
[bus0_start..bus0_end] (for example [0xe2..0xf0]) and
vmd->resources[VMD_RES_BUS1_CFGBAR] to [bus1_start..bus1_end] (for example
[0xf1..0xff]).  However, pci_create_root_bus() -> pci_register_host_bridge()
ignores res->start and inserts bus->busn_res starting at bus->number:

drivers/pci/probe.c:pci_register_host_bridge() {
    ...
	resource_list_for_each_entry(window, &bridge->windows) {
		res = window->res;
		if (resource_type(res) == IORESOURCE_BUS)
			pci_bus_insert_busn_res(bus, bus->number, res->end);
	}
    ...
}

As a result, VMD_BUS_0 gets busn_res [0xe0..0xf0] (which includes 0xe1, the
root bus of VMD_BUS_1) and VMD_BUS_1 gets busn_res [0xe1..0xff] (which
includes 0xe2..0xf0, the subordinate bus range of VMD_BUS_0), while
vmd_cfg_addr() maps bus->number directly into the shared vmd->cfgbar
without checking which domain owns that bus number.  When
pci_scan_child_bus_extend() encounters an unconfigured bridge on VMD_BUS_1
(or on VMD_BUS_0 before max passes 0xe0), Pass 1 assigns max + 1 (0xe2 on
VMD_BUS_1, or 0xe1 on VMD_BUS_0), scanning and reprogramming devices that
belong to the other VMD root bus.

[ ... ]

> @@ -1213,11 +1511,38 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
>  
>  	sd->node = pcibus_to_node(vmd->dev->bus);
>  
> -	ret = vmd_create_bus(vmd, sd, offset, vmd_in_guest);
> +	if (vmd->bus1_rootbus) {
> +		vmd->sysdata[VMD_BUS_1].vmd_dev = vmd->dev;
> +		vmd->sysdata[VMD_BUS_1].node = pcibus_to_node(vmd->dev->bus);
> +		vmd->sysdata[VMD_BUS_1].domain = pci_bus_find_emul_domain_nr(0, 0x10000, INT_MAX);
> +		if (vmd->sysdata[VMD_BUS_1].domain < 0) {
> +			pci_warn(vmd->dev,
> +				 "Can't allocate BUS1 domain: %d, continuing with BUS0 only\n",
> +				 vmd->sysdata[VMD_BUS_1].domain);
> +			vmd_disable_bus1(vmd);
> +		}
> +	}
> +
> +	ret = vmd_attach_resources(vmd);
> +	if (ret) {
> +		pci_bus_release_emul_domain_nr(sd->domain);
> +		if (vmd->bus1_rootbus && vmd->sysdata[VMD_BUS_1].domain >= 0)
> +			pci_bus_release_emul_domain_nr(vmd->sysdata[VMD_BUS_1].domain);
> +
> +		vmd_remove_irq_domain(vmd);
> +		return ret;
> +	}
> +
> +	ret = vmd_create_bus(vmd, VMD_BUS_0, sd, offset, vmd_in_guest,
> +			     VMD_PRIMARY_BUS0);
>  	if (ret) {
>  		pci_err(vmd->dev, "Can't create bus: %d\n", ret);
>  		pci_bus_release_emul_domain_nr(sd->domain);
> +		if (vmd->bus1_rootbus && vmd->sysdata[VMD_BUS_1].domain >= 0)
> +			pci_bus_release_emul_domain_nr(vmd->sysdata[VMD_BUS_1].domain);
> +
>  		vmd_remove_irq_domain(vmd);
> +		vmd_detach_resources(vmd);
>  		return ret;
>  	}

[Severity: Medium]
Do these error paths leak the per-vector srcu_struct instances initialized
by vmd_alloc_irqs()?

When MSI remapping is enabled, vmd_alloc_irqs() calls init_srcu_struct() on
each vmd->irqs[i].srcu, which allocates per-CPU memory not managed by
devres.  If vmd_attach_resources() or vmd_create_bus() fails, returning
without calling vmd_cleanup_srcu(vmd) leaks that memory and leaves active
srcu_struct objects in devm-allocated vmd->irqs memory.

>  	WARN(sysfs_create_link(&vmd->dev->dev.kobj,
>  			       &vmd->bus[VMD_BUS_0]->dev.kobj, "domain"),
>  	     "Can't create symlink to domain\n");
>  
> +	if (vmd->bus1_rootbus && vmd->sysdata[VMD_BUS_1].domain >= 0) {
> +		ret = vmd_create_bus(vmd, VMD_BUS_1, &vmd->sysdata[VMD_BUS_1], offset,
> +				     vmd_in_guest, VMD_PRIMARY_BUS1);
> +		if (ret) {
> +			pci_warn(vmd->dev,
> +				 "Can't create BUS1: %d, continuing with BUS0 only\n",
> +				 ret);
> +
> +			release_resource(&vmd->resources[VMD_RES_BUS1_MBAR_2]);
> +			release_resource(&vmd->resources[VMD_RES_BUS1_MBAR_1]);
> +			pci_bus_release_emul_domain_nr(vmd->sysdata[VMD_BUS_1].domain);
> +			vmd_disable_bus1(vmd);
> +		}
> +
> +		if (vmd->bus1_rootbus && vmd->sysdata[VMD_BUS_1].domain >= 0) {
> +			WARN(sysfs_create_link(&vmd->dev->dev.kobj,
> +					       &vmd->bus[VMD_BUS_1]->dev.kobj,
> +					       "domain1"),
> +			     "Can't create symlink to domain1\n");
> +		}
> +	}
> +
>  	vmd_bus_enumeration(vmd->bus[VMD_BUS_0], features);
>  
> +	if (vmd->bus1_rootbus && vmd->sysdata[VMD_BUS_1].domain >= 0)
> +		vmd_bus_enumeration(vmd->bus[VMD_BUS_1], features);

[Severity: Medium]
Does vmd_acpi_find_companion() compute the wrong relative bus number when
enumerating VMD_BUS_1?

drivers/pci/controller/vmd.c:vmd_acpi_find_companion() {
    ...
	bridge = pci_find_host_bridge(pci_dev->bus);
	busnr = pci_dev->bus->number - bridge->bus->number;
    ...
	addr = (busnr << 24) | ((u32)pci_dev->devfn << 16) | 0x8000FFFFU;

	return acpi_find_child_device(ACPI_COMPANION(bridge->dev.parent), addr,
				      false);
}

Both host bridges share &vmd->dev->dev as bridge->dev.parent, where the
relative bus number in the ACPI _ADR is the offset from VMD_BUS_0's start
bus (0xe0).  Because bridge->bus->number for VMD_BUS_1 is 0xe1, subtracting
bridge->bus->number yields busnr == 0 for root ports on 0xe1 (matching
VMD_BUS_0's ACPI companion objects) and shifts all VMD_BUS_1 subordinate bus
_ADR lookups down by one.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106184256.4130-1-szymon.durawa@linux.intel.com?part=7

[PATCH v10 0/7] VMD add second root bus support

[Szymon Durawa]

This series adds support for a second VMD root bus.

Today the VMD driver can enumerate and manage devices only behind the
first VMD root bus. Starting with Intel Arrow Lake, VMD may expose a
second root bus (BUS1), and devices behind that bus are not discovered
by the current driver.

The hardware model for Intel Arrow Lake VMD is slightly different
from earlier VMD generations:

- CFGBAR, MEMBAR1, and MEMBAR2 are shared by both VMD root buses.
- Root ports on BUS1 have PCI_PRIMARY_BUS hardwired to 0x80.
- Interrupt handling remains based on the existing VMD MSI/MSI-X and
  irq_domain model; this series does not introduce a separate interrupt
  domain for BUS1.

To support that layout, the series first refactors the current code so
resource setup, bus creation, and enumeration are split into helpers.
It then partitions the shared CFGBAR/MEMBAR apertures between BUS0 and
BUS1 and creates a second root bus in its own emulated PCI domain.
Reads of PCI_PRIMARY_BUS from BUS1 root port bridges are emulated to
return the actual root bus number, so PCI core bridge scanning does not
detect a mismatch and lose BUS1 subordinate device discovery. This
replaces the separate workaround patch used in v8.

Patch overview:

- Patches 1-4 extract helper functions for bus enumeration, CFGBAR and
  MEMBAR configuration, and root-bus creation.
- Patch 5 replaces hardcoded resource indices and constants with enums
  and defines.
- Patch 6 converts single-bus state (bus, busn_start, sysdata) into
  per-root-bus arrays.
- Patch 7 adds BUS1 resource partitioning, a separate PCI domain for
  BUS1, PCI_PRIMARY_BUS read emulation for BUS1 root ports, and
  second-root-bus support.

Tested on an Arrow Lake platform with NVMe devices present behind both
VMD root buses. With the series applied, devices behind BUS0 and BUS1
are enumerated and become visible to the kernel.

Changes from v1:
- splitting series into more commits, requested by Bjorn
- adding helper functions, suggested by Bjorn
- minor typos and unclear wording updated, suggested by Bjorn

Changes from v2:
- wording update in commit logs, suggested by Bjorn

Changes from v3:
- using GENMASK() instead of manual bits shifting, suggested by Bjorn
- converting decimal number to hex representation, suggested by Bjorn
- wording update in commit logs, suggested by Bjorn

Changes from v4:
- Update Dan's email address
- Resending the whole series to the correct mailing list

Changes from v5:
- Fixed correctness issues in BUS/resource layout in patches 7 and 8
- Fixed lifetime and use-after-free issues in patches 3 and 7
- Fixed resource tree integrity issues in patch 7
- Fixed error-path leaks across patches 3, 4, and 5
- Fixed allocation failure handling in patch 3
- Note on Sashiko feedback: Pre-existing warnings (unrelated to
  this feature series) were left untouched.

Changes from v6:
- Refactoring the series to align with recent changes in link below:
 https://patchwork.kernel.org/project/linux-pci/patch/20260629165025.268836-1-nirmal.patel@linux.intel.com/
- Fixed double free of the IRQ domain and emulated domain number
- Fixed a race with concurrent readers when modifying sibling pointers
- Fixed the MEMBAR2 split equations in
  vmd_configure_membar1_membar2()
- Fixed hardcoded resource indices in vmd_configure_membar()

Changes from v7:
- wording update in commit logs in patch 8, suggested by Bjorn

Changes from v8:
- Dropped patch 8 ("PCI: vmd: Workaround for hardwired BUS1 bus
  number") along with the issues Sashiko reported against it. The
  hardwired BUS1 primary bus number is now handled in patch 7 by
  emulating PCI_PRIMARY_BUS reads for BUS1 root port bridges in
  vmd_pci_read(), and BUS1 is assigned its own emulated PCI domain.
- Patch 6: sysdata is converted to a per-root-bus array as well.
- Patch 7: Fixed a data race reported by Sashiko where the MEMBAR
  .sibling pointers were assigned manually while the resource tree
  could be walked concurrently. BUS0/BUS1 MEMBAR resources are now
  inserted and removed with request_resource()/release_resource().
- Patch 7: Fixed an issue reported by Sashiko where resizing the CFGBAR
  start address for BUS1 caused vmd_domain_reset() to skip the root
  buses. The reset now starts from busn_start[VMD_BUS_0] and covers
  the whole CFGBAR aperture when BUS1 is present.
- Patch 7: BUS1 setup failures (domain allocation or bus creation) now
  release BUS1 resources and fall back to BUS0 only.

Changes from v9:
- Rebased series to v7.4-rc1, suggested by Mani
- Patch 7: VMD_FEAT_HAS_BUS1_ROOTBUS moved to bit 8, since bit 7 is now
  used by upstream VMD_FEAT_INTERRUPT_QUIRK

Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
Reviewed-by: Asutosh Pathak <asutosh.pathak@linux.intel.com>
Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>

Szymon Durawa (7):
  PCI: vmd: Add vmd_bus_enumeration() helper function
  PCI: vmd: Add vmd_configure_cfgbar() helper function
  PCI: vmd: Add vmd_configure_membar() and
    vmd_configure_membar1_membar2()
  PCI: vmd: Add vmd_create_bus()
  PCI: vmd: Replace hardcoded values with enum and defines
  PCI: vmd: Convert bus, busn_start and sysdata to an array
  PCI: vmd: Add support for second rootbus under VMD

 drivers/pci/controller/vmd.c | 727 ++++++++++++++++++++++++++++-------
 1 file changed, 592 insertions(+), 135 deletions(-)

-- 
2.43.0

[PATCH v10 1/7] PCI: vmd: Add vmd_bus_enumeration() helper function

[Szymon Durawa]

Move the vmd bus enumeration code to a new helper vmd_bus_enumeration().
No functional changes.

Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
Reviewed-by: Asutosh Pathak <asutosh.pathak@linux.intel.com>
Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>
---
 drivers/pci/controller/vmd.c | 89 ++++++++++++++++++++----------------
 1 file changed, 49 insertions(+), 40 deletions(-)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index 9d97a832785e..e4ac04936320 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -954,6 +954,54 @@ static int vmd_pm_enable_quirk(struct pci_dev *pdev, void *userdata)
 	return 0;
 }
 
+static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features)
+{
+	struct pci_bus *child;
+	struct pci_dev *dev;
+	int ret;
+
+	vmd_acpi_begin();
+
+	pci_scan_child_bus(bus);
+	vmd_domain_reset(vmd_from_bus(bus));
+
+	/*
+	 * When Intel VMD is enabled, the OS does not discover the Root Ports
+	 * owned by Intel VMD within the MMCFG space. pci_reset_bus() applies
+	 * a reset to the parent of the PCI device supplied as argument. This
+	 * is why we pass a child device, so the reset can be triggered at
+	 * the Intel bridge level and propagated to all the children in the
+	 * hierarchy.
+	 */
+	list_for_each_entry(child, &bus->children, node) {
+		if (!list_empty(&child->devices)) {
+			dev = list_first_entry(&child->devices, struct pci_dev,
+					       bus_list);
+			ret = pci_reset_bus(dev);
+			if (ret)
+				pci_warn(dev, "can't reset device: %d\n", ret);
+
+			break;
+		}
+	}
+
+	pci_assign_unassigned_bus_resources(bus);
+
+	pci_walk_bus(bus, vmd_pm_enable_quirk, &features);
+
+	/*
+	 * VMD root buses are virtual and don't return true on pci_is_pcie()
+	 * and will fail pcie_bus_configure_settings() early. It can instead be
+	 * run on each of the real root ports.
+	 */
+	list_for_each_entry(child, &bus->children, node)
+		pcie_bus_configure_settings(child);
+
+	pci_bus_add_devices(bus);
+
+	vmd_acpi_end();
+}
+
 static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 {
 	struct pci_sysdata *sd = &vmd->sysdata;
@@ -964,8 +1012,6 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	resource_size_t offset[2] = {0};
 	resource_size_t membar2_offset = 0x2000;
 	resource_size_t busn_end;
-	struct pci_bus *child;
-	struct pci_dev *dev;
 	bool vmd_in_guest;
 	int ret;
 
@@ -1096,45 +1142,8 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
 			       "domain"), "Can't create symlink to domain\n");
 
-	vmd_acpi_begin();
-
-	pci_scan_child_bus(vmd->bus);
-	vmd_domain_reset(vmd);
+	vmd_bus_enumeration(vmd->bus, features);
 
-	/* When Intel VMD is enabled, the OS does not discover the Root Ports
-	 * owned by Intel VMD within the MMCFG space. pci_reset_bus() applies
-	 * a reset to the parent of the PCI device supplied as argument. This
-	 * is why we pass a child device, so the reset can be triggered at
-	 * the Intel bridge level and propagated to all the children in the
-	 * hierarchy.
-	 */
-	list_for_each_entry(child, &vmd->bus->children, node) {
-		if (!list_empty(&child->devices)) {
-			dev = list_first_entry(&child->devices,
-					       struct pci_dev, bus_list);
-			ret = pci_reset_bus(dev);
-			if (ret)
-				pci_warn(dev, "can't reset device: %d\n", ret);
-
-			break;
-		}
-	}
-
-	pci_assign_unassigned_bus_resources(vmd->bus);
-
-	pci_walk_bus(vmd->bus, vmd_pm_enable_quirk, &features);
-
-	/*
-	 * VMD root buses are virtual and don't return true on pci_is_pcie()
-	 * and will fail pcie_bus_configure_settings() early. It can instead be
-	 * run on each of the real root ports.
-	 */
-	list_for_each_entry(child, &vmd->bus->children, node)
-		pcie_bus_configure_settings(child);
-
-	pci_bus_add_devices(vmd->bus);
-
-	vmd_acpi_end();
 	return 0;
 }
 
-- 
2.43.0

[PATCH v10 2/7] PCI: vmd: Add vmd_configure_cfgbar() helper function

[Szymon Durawa]

Move the VMD CFGBAR initialization code to a new helper
vmd_configure_cfgbar(). No functional changes.

Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
Reviewed-by: Asutosh Pathak <asutosh.pathak@linux.intel.com>
Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>
---
 drivers/pci/controller/vmd.c | 28 +++++++++++++++++-----------
 1 file changed, 17 insertions(+), 11 deletions(-)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index e4ac04936320..4992682a69b1 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -954,6 +954,22 @@ static int vmd_pm_enable_quirk(struct pci_dev *pdev, void *userdata)
 	return 0;
 }
 
+static void vmd_configure_cfgbar(struct vmd_dev *vmd)
+{
+	struct resource *res = &vmd->dev->resource[VMD_CFGBAR];
+	resource_size_t busn_end;
+
+	/* Do not let resource[0] end go out of bound.*/
+	busn_end = vmd->busn_start + (resource_size(res) >> 20) - 1;
+	busn_end = min_t(resource_size_t, busn_end, 0xff);
+	vmd->resources[0] = (struct resource) {
+		.name  = "VMD CFGBAR",
+		.start = vmd->busn_start,
+		.end   = busn_end,
+		.flags = IORESOURCE_BUS | IORESOURCE_PCI_FIXED,
+	};
+}
+
 static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features)
 {
 	struct pci_bus *child;
@@ -1011,7 +1027,6 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	LIST_HEAD(resources);
 	resource_size_t offset[2] = {0};
 	resource_size_t membar2_offset = 0x2000;
-	resource_size_t busn_end;
 	bool vmd_in_guest;
 	int ret;
 
@@ -1020,16 +1035,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	if (ret)
 		return ret;
 
-	/* Do not let resource[0] end go out of bounds */
-	res = &vmd->dev->resource[VMD_CFGBAR];
-	busn_end = vmd->busn_start + (resource_size(res) >> 20) - 1;
-	busn_end = min_t(resource_size_t, busn_end, 0xff);
-	vmd->resources[0] = (struct resource) {
-		.name  = "VMD CFGBAR",
-		.start = vmd->busn_start,
-		.end   = busn_end,
-		.flags = IORESOURCE_BUS | IORESOURCE_PCI_FIXED,
-	};
+	vmd_configure_cfgbar(vmd);
 
 	/*
 	 * If the window is below 4GB, clear IORESOURCE_MEM_64 so we can
-- 
2.43.0

[PATCH v10 3/7] PCI: vmd: Add vmd_configure_membar() and vmd_configure_membar1_membar2()

[Szymon Durawa]

Move the MEMBAR1 and MEMBAR2 registry initialization code to new helpers
vmd_configure_membar() and vmd_configure_membar1_membar2().
The refactor preserves MEMBAR address/offset/flags programming, but
resource name construction is now generated in the helper ("VMD MEMBAR%d")
instead of using two separate string literals.

Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
Reviewed-by: Asutosh Pathak <asutosh.pathak@linux.intel.com>
Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>
---
 drivers/pci/controller/vmd.c | 98 +++++++++++++++++++++++++-----------
 1 file changed, 68 insertions(+), 30 deletions(-)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index 4992682a69b1..f50286ae242e 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -970,6 +970,69 @@ static void vmd_configure_cfgbar(struct vmd_dev *vmd)
 	};
 }
 
+/*
+ * vmd_configure_membar - Configure VMD MemBAR register, which points
+ * to MMIO address assigned by the OS or BIOS.
+ * @vmd: the VMD device
+ * @resource_number: resource buffer number to be filled in
+ * @membar_number: number of the MemBAR
+ * @start_offset: 4K aligned offset applied to start of VMD’s MEMBAR MMIO space
+ * @end_offset: 4K aligned offset applied to end of VMD’s MEMBAR MMIO space
+ *
+ * Function fills resource buffer inside the VMD structure.
+ *
+ * Return: 0 on success, -ENOMEM on allocation failure.
+ */
+static int vmd_configure_membar(struct vmd_dev *vmd, u8 resource_number,
+				u8 membar_number, resource_size_t start_offset,
+				resource_size_t end_offset)
+{
+	char *name;
+	u32 upper_bits;
+	unsigned long flags;
+
+	struct resource *res = &vmd->dev->resource[membar_number];
+
+	upper_bits = upper_32_bits(res->end);
+	flags = res->flags & ~IORESOURCE_SIZEALIGN;
+	if (!upper_bits)
+		flags &= ~IORESOURCE_MEM_64;
+
+	name = devm_kasprintf(&vmd->dev->dev, GFP_KERNEL, "VMD MEMBAR%d",
+			      resource_number);
+	if (!name)
+		return -ENOMEM;
+
+	vmd->resources[resource_number] = (struct resource){
+		.name = name,
+		.start = res->start + start_offset,
+		.end = res->end - end_offset,
+		.flags = flags,
+		.parent = res,
+	};
+
+	return 0;
+}
+
+static int vmd_configure_membar1_membar2(struct vmd_dev *vmd,
+					 resource_size_t mbar2_ofs)
+{
+	int ret;
+
+	ret = vmd_configure_membar(vmd, 1, VMD_MEMBAR1, 0, 0);
+	if (ret)
+		return ret;
+
+	ret = vmd_configure_membar(vmd, 2, VMD_MEMBAR2, mbar2_ofs, 0);
+	if (ret) {
+		devm_kfree(&vmd->dev->dev, (void *)vmd->resources[1].name);
+		memset(&vmd->resources[1], 0, sizeof(vmd->resources[1]));
+		return ret;
+	}
+
+	return 0;
+}
+
 static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features)
 {
 	struct pci_bus *child;
@@ -1021,9 +1084,6 @@ static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features)
 static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 {
 	struct pci_sysdata *sd = &vmd->sysdata;
-	struct resource *res;
-	u32 upper_bits;
-	unsigned long flags;
 	LIST_HEAD(resources);
 	resource_size_t offset[2] = {0};
 	resource_size_t membar2_offset = 0x2000;
@@ -1049,36 +1109,14 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	 *
 	 * The only way we could use a 64-bit non-prefetchable MEMBAR is
 	 * if its address is <4GB so that we can convert it to a 32-bit
-	 * resource.  To be visible to the host OS, all VMD endpoints must
+	 * resource. To be visible to the host OS, all VMD endpoints must
 	 * be initially configured by platform BIOS, which includes setting
-	 * up these resources.  We can assume the device is configured
+	 * up these resources. We can assume the device is configured
 	 * according to the platform needs.
 	 */
-	res = &vmd->dev->resource[VMD_MEMBAR1];
-	upper_bits = upper_32_bits(res->end);
-	flags = res->flags & ~IORESOURCE_SIZEALIGN;
-	if (!upper_bits)
-		flags &= ~IORESOURCE_MEM_64;
-	vmd->resources[1] = (struct resource) {
-		.name  = "VMD MEMBAR1",
-		.start = res->start,
-		.end   = res->end,
-		.flags = flags,
-		.parent = res,
-	};
-
-	res = &vmd->dev->resource[VMD_MEMBAR2];
-	upper_bits = upper_32_bits(res->end);
-	flags = res->flags & ~IORESOURCE_SIZEALIGN;
-	if (!upper_bits)
-		flags &= ~IORESOURCE_MEM_64;
-	vmd->resources[2] = (struct resource) {
-		.name  = "VMD MEMBAR2",
-		.start = res->start + membar2_offset,
-		.end   = res->end,
-		.flags = flags,
-		.parent = res,
-	};
+	ret = vmd_configure_membar1_membar2(vmd, membar2_offset);
+	if (ret)
+		return ret;
 
 	/* Non-zero offset means guest/direct assign view. */
 	vmd_in_guest = offset[0] || offset[1];
-- 
2.43.0

[PATCH v10 4/7] PCI: vmd: Add vmd_create_bus()

[Szymon Durawa]

Move the VMD bus initialization code to a new helper vmd_create_bus().
No functional changes.

Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
Reviewed-by: Asutosh Pathak <asutosh.pathak@linux.intel.com>
Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>
---
 drivers/pci/controller/vmd.c | 63 ++++++++++++++++++++++--------------
 1 file changed, 38 insertions(+), 25 deletions(-)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index f50286ae242e..2acabe05e909 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -1033,6 +1033,40 @@ static int vmd_configure_membar1_membar2(struct vmd_dev *vmd,
 	return 0;
 }
 
+static int vmd_create_bus(struct vmd_dev *vmd, struct pci_sysdata *sd,
+			  resource_size_t *offset, bool vmd_in_guest)
+{
+	LIST_HEAD(resources);
+
+	pci_add_resource(&resources, &vmd->resources[0]);
+	pci_add_resource_offset(&resources, &vmd->resources[1], offset[0]);
+	pci_add_resource_offset(&resources, &vmd->resources[2], offset[1]);
+
+	vmd->bus = pci_create_root_bus(&vmd->dev->dev, vmd->busn_start,
+				       &vmd_ops, sd, &resources);
+	if (!vmd->bus) {
+		pci_free_resource_list(&resources);
+		return -ENODEV;
+	}
+
+	/*
+	 * Don't copy _OSC control flags from root bridge if running in a VM, as
+	 * they don't reflect the physical root bridge capabilities.
+	 */
+	if (!vmd_in_guest)
+		vmd_copy_host_bridge_flags(pci_find_host_bridge(vmd->dev->bus),
+					   to_pci_host_bridge(vmd->bus->bridge));
+
+	vmd_attach_resources(vmd);
+	if (vmd->irq_domain)
+		dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
+	else
+		dev_set_msi_domain(&vmd->bus->dev,
+				   dev_get_msi_domain(&vmd->dev->dev));
+
+	return 0;
+}
+
 static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features)
 {
 	struct pci_bus *child;
@@ -1084,7 +1118,6 @@ static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features)
 static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 {
 	struct pci_sysdata *sd = &vmd->sysdata;
-	LIST_HEAD(resources);
 	resource_size_t offset[2] = {0};
 	resource_size_t membar2_offset = 0x2000;
 	bool vmd_in_guest;
@@ -1141,10 +1174,6 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 		vmd_set_msi_remapping(vmd, false);
 	}
 
-	pci_add_resource(&resources, &vmd->resources[0]);
-	pci_add_resource_offset(&resources, &vmd->resources[1], offset[0]);
-	pci_add_resource_offset(&resources, &vmd->resources[2], offset[1]);
-
 	sd->vmd_dev = vmd->dev;
 
 	/*
@@ -1159,30 +1188,14 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 
 	sd->node = pcibus_to_node(vmd->dev->bus);
 
-	vmd->bus = pci_create_root_bus(&vmd->dev->dev, vmd->busn_start,
-				       &vmd_ops, sd, &resources);
-	if (!vmd->bus) {
+	ret = vmd_create_bus(vmd, sd, offset, vmd_in_guest);
+	if (ret) {
+		pci_err(vmd->dev, "Can't create bus: %d\n", ret);
 		pci_bus_release_emul_domain_nr(sd->domain);
-		pci_free_resource_list(&resources);
 		vmd_remove_irq_domain(vmd);
-		return -ENODEV;
+		return ret;
 	}
 
-	/*
-	 * Don't copy _OSC control flags from root bridge if running in a VM, as
-	 * they don't reflect the physical root bridge capabilities.
-	 */
-	if (!vmd_in_guest)
-		vmd_copy_host_bridge_flags(pci_find_host_bridge(vmd->dev->bus),
-					 to_pci_host_bridge(vmd->bus->bridge));
-
-	vmd_attach_resources(vmd);
-	if (vmd->irq_domain)
-		dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
-	else
-		dev_set_msi_domain(&vmd->bus->dev,
-				   dev_get_msi_domain(&vmd->dev->dev));
-
 	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
 			       "domain"), "Can't create symlink to domain\n");
 
-- 
2.43.0

[PATCH v10 5/7] PCI: vmd: Replace hardcoded values with enum and defines

[Szymon Durawa]

Add enum vmd_resource type to replace hardcoded values. Add defines for
vmd bus start number based on VMD restriction value. No functional
changes.

Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
Reviewed-by: Asutosh Pathak <asutosh.pathak@linux.intel.com>
Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>
---
 drivers/pci/controller/vmd.c | 52 +++++++++++++++++++++++-------------
 1 file changed, 34 insertions(+), 18 deletions(-)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index 2acabe05e909..09d3bf35c934 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -26,6 +26,11 @@
 #define VMD_MEMBAR1	2
 #define VMD_MEMBAR2	4
 
+/* VMD restriction value determines secondary start bus number */
+#define VMD_RESTRICT_0_BUS_START 0x0
+#define VMD_RESTRICT_1_BUS_START 0x80
+#define VMD_RESTRICT_2_BUS_START 0xE0
+
 #define PCI_REG_VMCAP		0x40
 #define BUS_RESTRICT_CAP(vmcap)	(vmcap & 0x1)
 #define PCI_REG_VMCONFIG	0x44
@@ -43,6 +48,13 @@
 #define BASE_ID_REG_28C1		0x2840
 #define MEMBAR2_OFFSET_28C1		0x30d0
 
+enum vmd_resource {
+	VMD_RES_CFGBAR = 0, /* VMD Bus0 Config BAR */
+	VMD_RES_MBAR_1, /* VMD Bus0 Resource MemBAR 1 */
+	VMD_RES_MBAR_2, /* VMD Bus0 Resource MemBAR 2 */
+	VMD_RES_COUNT
+};
+
 enum vmd_features {
 	/*
 	 * Device may contain registers which hint the physical location of the
@@ -179,7 +191,7 @@ struct vmd_dev {
 	struct vmd_irq_list	*irqs;
 
 	struct pci_sysdata	sysdata;
-	struct resource		resources[3];
+	struct resource		resources[VMD_RES_COUNT];
 	struct irq_domain	*irq_domain;
 	struct pci_bus		*bus;
 	u8			busn_start;
@@ -590,7 +602,7 @@ static resource_size_t vmd_cfgbar_ecam_space(struct vmd_dev *vmd)
 	unsigned int ecam_start;
 
 	cfgbar_buses = resource_size(&vmd->dev->resource[VMD_CFGBAR]) >> 20;
-	ecam_start = vmd_bus_to_ecam(vmd, vmd->resources[0].start);
+	ecam_start = vmd_bus_to_ecam(vmd, vmd->resources[VMD_RES_CFGBAR].start);
 	if (ecam_start >= cfgbar_buses)
 		return 0;
 
@@ -598,14 +610,15 @@ static resource_size_t vmd_cfgbar_ecam_space(struct vmd_dev *vmd)
 }
 static void vmd_domain_reset(struct vmd_dev *vmd)
 {
-	u16 bus, max_buses = resource_size(&vmd->resources[0]);
+	u16 bus, max_buses = resource_size(&vmd->resources[VMD_RES_CFGBAR]);
 	u8 dev, functions, fn, hdr_type;
 	unsigned int ecam_bus;
 	char __iomem *base;
 
 	max_buses = min_t(u16, max_buses, vmd_cfgbar_ecam_space(vmd));
 	for (bus = 0; bus < max_buses; bus++) {
-		ecam_bus = vmd_bus_to_ecam(vmd, vmd->resources[0].start + bus);
+		ecam_bus = vmd_bus_to_ecam(vmd, vmd->resources[VMD_RES_CFGBAR].start +
+					   bus);
 		for (dev = 0; dev < 32; dev++) {
 			base = vmd->cfgbar + PCIE_ECAM_OFFSET(ecam_bus,
 						PCI_DEVFN(dev, 0), 0);
@@ -649,8 +662,8 @@ static void vmd_domain_reset(struct vmd_dev *vmd)
 
 static void vmd_attach_resources(struct vmd_dev *vmd)
 {
-	vmd->dev->resource[VMD_MEMBAR1].child = &vmd->resources[1];
-	vmd->dev->resource[VMD_MEMBAR2].child = &vmd->resources[2];
+	vmd->dev->resource[VMD_MEMBAR1].child = &vmd->resources[VMD_RES_MBAR_1];
+	vmd->dev->resource[VMD_MEMBAR2].child = &vmd->resources[VMD_RES_MBAR_2];
 }
 
 static void vmd_detach_resources(struct vmd_dev *vmd)
@@ -726,14 +739,14 @@ static int vmd_get_bus_number_start(struct vmd_dev *vmd)
 
 		switch (BUS_RESTRICT_CFG(reg)) {
 		case 0:
-			vmd->busn_start = 0;
+			vmd->busn_start = VMD_RESTRICT_0_BUS_START;
 			break;
 		case 1:
-			vmd->busn_start = 128;
+			vmd->busn_start = VMD_RESTRICT_1_BUS_START;
 			break;
 		case 3:
 		case 2:
-			vmd->busn_start = 224;
+			vmd->busn_start = VMD_RESTRICT_2_BUS_START;
 			break;
 		default:
 			pci_err(dev, "Unknown Bus Offset Setting (%d)\n",
@@ -959,10 +972,10 @@ static void vmd_configure_cfgbar(struct vmd_dev *vmd)
 	struct resource *res = &vmd->dev->resource[VMD_CFGBAR];
 	resource_size_t busn_end;
 
-	/* Do not let resource[0] end go out of bound.*/
+	/* Do not let resource[VMD_RES_CFGBAR] end go out of bound.*/
 	busn_end = vmd->busn_start + (resource_size(res) >> 20) - 1;
 	busn_end = min_t(resource_size_t, busn_end, 0xff);
-	vmd->resources[0] = (struct resource) {
+	vmd->resources[VMD_RES_CFGBAR] = (struct resource) {
 		.name  = "VMD CFGBAR",
 		.start = vmd->busn_start,
 		.end   = busn_end,
@@ -1019,14 +1032,15 @@ static int vmd_configure_membar1_membar2(struct vmd_dev *vmd,
 {
 	int ret;
 
-	ret = vmd_configure_membar(vmd, 1, VMD_MEMBAR1, 0, 0);
+	ret = vmd_configure_membar(vmd, VMD_RES_MBAR_1, VMD_MEMBAR1, 0, 0);
 	if (ret)
 		return ret;
 
-	ret = vmd_configure_membar(vmd, 2, VMD_MEMBAR2, mbar2_ofs, 0);
+	ret = vmd_configure_membar(vmd, VMD_RES_MBAR_2, VMD_MEMBAR2, mbar2_ofs, 0);
 	if (ret) {
-		devm_kfree(&vmd->dev->dev, (void *)vmd->resources[1].name);
-		memset(&vmd->resources[1], 0, sizeof(vmd->resources[1]));
+		devm_kfree(&vmd->dev->dev, (void *)vmd->resources[VMD_RES_MBAR_1].name);
+		memset(&vmd->resources[VMD_RES_MBAR_1], 0,
+		       sizeof(vmd->resources[VMD_RES_MBAR_1]));
 		return ret;
 	}
 
@@ -1038,9 +1052,11 @@ static int vmd_create_bus(struct vmd_dev *vmd, struct pci_sysdata *sd,
 {
 	LIST_HEAD(resources);
 
-	pci_add_resource(&resources, &vmd->resources[0]);
-	pci_add_resource_offset(&resources, &vmd->resources[1], offset[0]);
-	pci_add_resource_offset(&resources, &vmd->resources[2], offset[1]);
+	pci_add_resource(&resources, &vmd->resources[VMD_RES_CFGBAR]);
+	pci_add_resource_offset(&resources, &vmd->resources[VMD_RES_MBAR_1],
+				offset[0]);
+	pci_add_resource_offset(&resources, &vmd->resources[VMD_RES_MBAR_2],
+				offset[1]);
 
 	vmd->bus = pci_create_root_bus(&vmd->dev->dev, vmd->busn_start,
 				       &vmd_ops, sd, &resources);
-- 
2.43.0

[PATCH v10 6/7] PCI: vmd: Convert bus, busn_start and sysdata to an array

[Szymon Durawa]

Convert bus, busn_start and sysdata in vmd_dev struct from scalar
to an array to support multiple VMD buses in the future.
No functional changes.

Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
Reviewed-by: Asutosh Pathak <asutosh.pathak@linux.intel.com>
Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>
---
 drivers/pci/controller/vmd.c | 64 +++++++++++++++++++++---------------
 1 file changed, 37 insertions(+), 27 deletions(-)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index 09d3bf35c934..5733faf972d1 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -55,6 +55,11 @@ enum vmd_resource {
 	VMD_RES_COUNT
 };
 
+enum vmd_rootbus {
+	VMD_BUS_0 = 0,
+	VMD_BUS_COUNT
+};
+
 enum vmd_features {
 	/*
 	 * Device may contain registers which hint the physical location of the
@@ -190,11 +195,11 @@ struct vmd_dev {
 	int msix_count;
 	struct vmd_irq_list	*irqs;
 
-	struct pci_sysdata	sysdata;
+	struct pci_sysdata	sysdata[VMD_BUS_COUNT];
 	struct resource		resources[VMD_RES_COUNT];
 	struct irq_domain	*irq_domain;
-	struct pci_bus		*bus;
-	u8			busn_start;
+	struct pci_bus		*bus[VMD_BUS_COUNT];
+	u8			busn_start[VMD_BUS_COUNT];
 	u8			first_vec;
 	char			*name;
 	int			instance;
@@ -203,7 +208,10 @@ struct vmd_dev {
 
 static inline struct vmd_dev *vmd_from_bus(struct pci_bus *bus)
 {
-	return container_of(bus->sysdata, struct vmd_dev, sysdata);
+	struct pci_sysdata *sd = bus->sysdata;
+
+	return pci_get_drvdata(sd->vmd_dev);
+
 }
 
 static inline unsigned int index_from_irqs(struct vmd_dev *vmd,
@@ -412,7 +420,7 @@ static int vmd_create_irq_domain(struct vmd_dev *vmd)
 	};
 
 	info.fwnode = irq_domain_alloc_named_id_fwnode("VMD-MSI",
-						       vmd->sysdata.domain);
+						       vmd->sysdata[VMD_BUS_0].domain);
 	if (!info.fwnode)
 		return -ENODEV;
 
@@ -461,7 +469,7 @@ static unsigned int vmd_bus_to_ecam(struct vmd_dev *vmd, unsigned int busnr)
 	if (!!(vmd->features & VMD_FEAT_USE_BIOS_INFO))
 		return busnr;
 
-	return busnr - vmd->busn_start;
+	return busnr - vmd->busn_start[VMD_BUS_0];
 }
 
 static void __iomem *vmd_cfg_addr(struct vmd_dev *vmd, struct pci_bus *bus,
@@ -739,14 +747,14 @@ static int vmd_get_bus_number_start(struct vmd_dev *vmd)
 
 		switch (BUS_RESTRICT_CFG(reg)) {
 		case 0:
-			vmd->busn_start = VMD_RESTRICT_0_BUS_START;
+			vmd->busn_start[VMD_BUS_0] = VMD_RESTRICT_0_BUS_START;
 			break;
 		case 1:
-			vmd->busn_start = VMD_RESTRICT_1_BUS_START;
+			vmd->busn_start[VMD_BUS_0] = VMD_RESTRICT_1_BUS_START;
 			break;
 		case 3:
 		case 2:
-			vmd->busn_start = VMD_RESTRICT_2_BUS_START;
+			vmd->busn_start[VMD_BUS_0] = VMD_RESTRICT_2_BUS_START;
 			break;
 		default:
 			pci_err(dev, "Unknown Bus Offset Setting (%d)\n",
@@ -783,7 +791,7 @@ static int vmd_get_bus_info_from_bar4(struct vmd_dev *vmd,
 	bar4_2840 = readq(bar4 + BASE_ID_REG_28C1);
 	base_id = bar4_2840 & 0xFFFFFF;
 	base_bus = base_id >> 8;
-	vmd->busn_start = base_bus;
+	vmd->busn_start[VMD_BUS_0] = base_bus;
 
 	/* Calculate offsets like vmd_get_phys_offsets() does */
 	if (phys1)
@@ -973,11 +981,11 @@ static void vmd_configure_cfgbar(struct vmd_dev *vmd)
 	resource_size_t busn_end;
 
 	/* Do not let resource[VMD_RES_CFGBAR] end go out of bound.*/
-	busn_end = vmd->busn_start + (resource_size(res) >> 20) - 1;
+	busn_end = vmd->busn_start[VMD_BUS_0] + (resource_size(res) >> 20) - 1;
 	busn_end = min_t(resource_size_t, busn_end, 0xff);
 	vmd->resources[VMD_RES_CFGBAR] = (struct resource) {
 		.name  = "VMD CFGBAR",
-		.start = vmd->busn_start,
+		.start = vmd->busn_start[VMD_BUS_0],
 		.end   = busn_end,
 		.flags = IORESOURCE_BUS | IORESOURCE_PCI_FIXED,
 	};
@@ -1058,9 +1066,10 @@ static int vmd_create_bus(struct vmd_dev *vmd, struct pci_sysdata *sd,
 	pci_add_resource_offset(&resources, &vmd->resources[VMD_RES_MBAR_2],
 				offset[1]);
 
-	vmd->bus = pci_create_root_bus(&vmd->dev->dev, vmd->busn_start,
-				       &vmd_ops, sd, &resources);
-	if (!vmd->bus) {
+	vmd->bus[VMD_BUS_0] = pci_create_root_bus(&vmd->dev->dev,
+						  vmd->busn_start[VMD_BUS_0],
+						  &vmd_ops, sd, &resources);
+	if (!vmd->bus[VMD_BUS_0]) {
 		pci_free_resource_list(&resources);
 		return -ENODEV;
 	}
@@ -1071,13 +1080,13 @@ static int vmd_create_bus(struct vmd_dev *vmd, struct pci_sysdata *sd,
 	 */
 	if (!vmd_in_guest)
 		vmd_copy_host_bridge_flags(pci_find_host_bridge(vmd->dev->bus),
-					   to_pci_host_bridge(vmd->bus->bridge));
+					   to_pci_host_bridge(vmd->bus[VMD_BUS_0]->bridge));
 
 	vmd_attach_resources(vmd);
 	if (vmd->irq_domain)
-		dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
+		dev_set_msi_domain(&vmd->bus[VMD_BUS_0]->dev, vmd->irq_domain);
 	else
-		dev_set_msi_domain(&vmd->bus->dev,
+		dev_set_msi_domain(&vmd->bus[VMD_BUS_0]->dev,
 				   dev_get_msi_domain(&vmd->dev->dev));
 
 	return 0;
@@ -1133,7 +1142,7 @@ static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features)
 
 static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 {
-	struct pci_sysdata *sd = &vmd->sysdata;
+	struct pci_sysdata *sd = &vmd->sysdata[VMD_BUS_0];
 	resource_size_t offset[2] = {0};
 	resource_size_t membar2_offset = 0x2000;
 	bool vmd_in_guest;
@@ -1212,10 +1221,11 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 		return ret;
 	}
 
-	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
-			       "domain"), "Can't create symlink to domain\n");
+	WARN(sysfs_create_link(&vmd->dev->dev.kobj,
+			       &vmd->bus[VMD_BUS_0]->dev.kobj, "domain"),
+	     "Can't create symlink to domain\n");
 
-	vmd_bus_enumeration(vmd->bus, features);
+	vmd_bus_enumeration(vmd->bus[VMD_BUS_0], features);
 
 	return 0;
 }
@@ -1252,7 +1262,7 @@ static int vmd_probe(struct pci_dev *dev, const struct pci_device_id *id)
 		return -ENOMEM;
 
 	vmd->dev = dev;
-	vmd->sysdata.domain = PCI_DOMAIN_NR_NOT_SET;
+	vmd->sysdata[VMD_BUS_0].domain = PCI_DOMAIN_NR_NOT_SET;
 	vmd->features = features;
 	vmd->instance = ida_alloc(&vmd_instance_ida, GFP_KERNEL);
 	if (vmd->instance < 0)
@@ -1292,7 +1302,7 @@ static int vmd_probe(struct pci_dev *dev, const struct pci_device_id *id)
 		goto out_release_instance;
 
 	dev_info(&vmd->dev->dev, "Bound to PCI domain %04x\n",
-		 vmd->sysdata.domain);
+		 vmd->sysdata[VMD_BUS_0].domain);
 	return 0;
 
  out_release_instance:
@@ -1312,14 +1322,14 @@ static void vmd_remove(struct pci_dev *dev)
 {
 	struct vmd_dev *vmd = pci_get_drvdata(dev);
 
-	pci_stop_root_bus(vmd->bus);
+	pci_stop_root_bus(vmd->bus[VMD_BUS_0]);
 	sysfs_remove_link(&vmd->dev->dev.kobj, "domain");
-	pci_remove_root_bus(vmd->bus);
+	pci_remove_root_bus(vmd->bus[VMD_BUS_0]);
 	vmd_cleanup_srcu(vmd);
 	vmd_detach_resources(vmd);
 	vmd_remove_irq_domain(vmd);
 	ida_free(&vmd_instance_ida, vmd->instance);
-	pci_bus_release_emul_domain_nr(vmd->sysdata.domain);
+	pci_bus_release_emul_domain_nr(vmd->sysdata[VMD_BUS_0].domain);
 }
 
 static void vmd_shutdown(struct pci_dev *dev)
-- 
2.43.0

[PATCH v10 7/7] PCI: vmd: Add support for second rootbus under VMD

[Szymon Durawa]

Starting from Intel Arrow Lake, VMD exposes a second root bus (BUS1),
reported via Bus Restriction type 3. BUS0 starts at bus 0xE0 and BUS1
at 0xE1; the subordinate bus ranges for each are provided by firmware
in VMD_BUSRANGE0/1. All three VMD BARs are shared between BUS0 and
BUS1: CFGBAR is split by those bus ranges, and MEMBAR1/MEMBAR2 are
split at the offsets in VMD_MEMBAR1_OFFSET and VMD_MEMBAR2_OFFSET1/2.

Add VMD_FEAT_HAS_BUS1_ROOTBUS for client VMDs and, when restriction
type 3 is reported, create and enumerate a second root bus in its own
emulated PCI domain so that its bus numbers and resources stay
separate from BUS0. A "domain1" sysfs link is added alongside the
existing "domain" link. If the BUS1 domain or root bus cannot be
created, the driver warns and continues with BUS0 only. Restriction
type 3 on a VMD without the feature is now rejected instead of being
treated as type 2.

The root ports on BUS1 have PCI_PRIMARY_BUS hardwired to 0x80. Emulate
reads of that register for Type 1 devices on the BUS1 root bus so it
reports the actual root bus number, and set bus->primary on both root
buses before scanning.

Suggested-by: Nirmal Patel <nirmal.patel@linux.intel.com>
Reviewed-by: Asutosh Pathak <asutosh.pathak@linux.intel.com>
Signed-off-by: Szymon Durawa <szymon.durawa@linux.intel.com>
---
 drivers/pci/controller/vmd.c | 467 +++++++++++++++++++++++++++++++----
 1 file changed, 416 insertions(+), 51 deletions(-)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index 5733faf972d1..6831d0a5c443 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -4,6 +4,7 @@
  * Copyright (c) 2015, Intel Corporation.
  */
 
+#include <linux/bitfield.h>
 #include <linux/device.h>
 #include <linux/interrupt.h>
 #include <linux/irq.h>
@@ -30,6 +31,7 @@
 #define VMD_RESTRICT_0_BUS_START 0x0
 #define VMD_RESTRICT_1_BUS_START 0x80
 #define VMD_RESTRICT_2_BUS_START 0xE0
+#define VMD_RESTRICT_3_BUS_START 0xE1
 
 #define PCI_REG_VMCAP		0x40
 #define BUS_RESTRICT_CAP(vmcap)	(vmcap & 0x1)
@@ -48,15 +50,36 @@
 #define BASE_ID_REG_28C1		0x2840
 #define MEMBAR2_OFFSET_28C1		0x30d0
 
+/* Primary Bus Number for VMD devices on root bus 0 */
+#define VMD_PRIMARY_BUS0    0x00
+/* Primary Bus Number for VMD devices on root bus 1 */
+#define VMD_PRIMARY_BUS1    0x80
+
+#define VMD_BUSRANGE0       0xc8
+#define VMD_BUSRANGE1       0xcc
+#define VMD_MEMBAR1_OFFSET  0xd0
+#define VMD_MEMBAR2_OFFSET1 0xd8
+#define VMD_MEMBAR2_OFFSET2 0xdc
+#define VMD_BUS_END(busr) FIELD_GET(GENMASK(15, 8), busr)
+#define VMD_BUS_START(busr) FIELD_GET(GENMASK(7, 0), busr)
+
+/*
+ * Add VMD resources for BUS1, it will share the same MMIO space with
+ * previous VMD resources.
+ */
 enum vmd_resource {
-	VMD_RES_CFGBAR = 0, /* VMD Bus0 Config BAR */
-	VMD_RES_MBAR_1, /* VMD Bus0 Resource MemBAR 1 */
-	VMD_RES_MBAR_2, /* VMD Bus0 Resource MemBAR 2 */
+	VMD_RES_CFGBAR = 0,  /* VMD Bus0 Config BAR */
+	VMD_RES_MBAR_1,      /* VMD Bus0 Resource MemBAR 1 */
+	VMD_RES_MBAR_2,      /* VMD Bus0 Resource MemBAR 2 */
+	VMD_RES_BUS1_CFGBAR, /* VMD Bus1 Config BAR */
+	VMD_RES_BUS1_MBAR_1, /* VMD Bus1 Resource MemBAR 1 */
+	VMD_RES_BUS1_MBAR_2, /* VMD Bus1 Resource MemBAR 2 */
 	VMD_RES_COUNT
 };
 
 enum vmd_rootbus {
 	VMD_BUS_0 = 0,
+	VMD_BUS_1,
 	VMD_BUS_COUNT
 };
 
@@ -126,6 +149,12 @@ enum vmd_features {
 	 * against the child's writes and is not sufficient (measured).
 	 */
 	VMD_FEAT_INTERRUPT_QUIRK	= (1 << 7),
+
+	/*
+	 * Starting from Intel Arrow Lake, VMD devices have their VMD root ports
+	 * on the additional BUS1 root bus.
+	 */
+	VMD_FEAT_HAS_BUS1_ROOTBUS	= (1 << 8),
 };
 
 #define VMD_BIOS_PM_QUIRK_LTR	0x1003	/* 3145728 ns */
@@ -133,7 +162,8 @@ enum vmd_features {
 #define VMD_FEATS_CLIENT	(VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP |	\
 				 VMD_FEAT_HAS_BUS_RESTRICTIONS |	\
 				 VMD_FEAT_OFFSET_FIRST_VECTOR |		\
-				 VMD_FEAT_BIOS_PM_QUIRK)
+				 VMD_FEAT_BIOS_PM_QUIRK |		\
+				 VMD_FEAT_HAS_BUS1_ROOTBUS)
 
 static DEFINE_IDA(vmd_instance_ida);
 
@@ -204,6 +234,7 @@ struct vmd_dev {
 	char			*name;
 	int			instance;
 	unsigned long		features;
+	bool			bus1_rootbus;
 };
 
 static inline struct vmd_dev *vmd_from_bus(struct pci_bus *bus)
@@ -487,6 +518,36 @@ static void __iomem *vmd_cfg_addr(struct vmd_dev *vmd, struct pci_bus *bus,
 	return vmd->cfgbar + offset;
 }
 
+static bool vmd_is_bus1_root_bridge(struct vmd_dev *vmd, struct pci_bus *bus,
+				    unsigned int devfn)
+{
+	void __iomem *hdr_addr;
+	u8 hdr_type;
+
+	lockdep_assert_held(&vmd->cfg_lock);
+
+	if (!vmd->bus1_rootbus)
+		return false;
+
+	/* Limit strictly to BUS1 */
+	if (bus->sysdata != &vmd->sysdata[VMD_BUS_1])
+		return false;
+
+	if (!pci_is_root_bus(bus))
+		return false;
+
+	/*
+	 * Offset 0x18 is PCI_PRIMARY_BUS only for Type 1 bridge headers.
+	 * On Type 0 endpoints, 0x18 is BAR2. Only apply the quirk to bridges.
+	 */
+	hdr_addr = vmd_cfg_addr(vmd, bus, devfn, PCI_HEADER_TYPE, 1);
+	if (!hdr_addr)
+		return false;
+
+	hdr_type = readb(hdr_addr) & PCI_HEADER_TYPE_MASK;
+	return hdr_type == PCI_HEADER_TYPE_BRIDGE;
+}
+
 /*
  * CPU may deadlock if config space is not serialized on some versions of this
  * hardware, so all config space access is done under a spinlock.
@@ -496,6 +557,7 @@ static int vmd_pci_read(struct pci_bus *bus, unsigned int devfn, int reg,
 {
 	struct vmd_dev *vmd = vmd_from_bus(bus);
 	void __iomem *addr = vmd_cfg_addr(vmd, bus, devfn, reg, len);
+	int ret = 0;
 
 	if (!addr)
 		return -EFAULT;
@@ -504,16 +566,34 @@ static int vmd_pci_read(struct pci_bus *bus, unsigned int devfn, int reg,
 	switch (len) {
 	case 1:
 		*value = readb(addr);
-		return 0;
+		break;
 	case 2:
 		*value = readw(addr);
-		return 0;
+		break;
 	case 4:
 		*value = readl(addr);
-		return 0;
+		break;
 	default:
-		return -EINVAL;
+		ret = -EINVAL;
+		break;
 	}
+
+	/*
+	 * Quirk: Hardware PCIe root port bridges on BUS1 have
+	 * PCI_PRIMARY_BUS hardwired to 0x80 (VMD_PRIMARY_BUS1).
+	 *
+	 * Emulate the primary bus number to match the actual allocated
+	 * root bus number. This prevents the PCI core from detecting a
+	 * bus mismatch without needing synthetic bus ranges.
+	 */
+	if (!ret && reg <= PCI_PRIMARY_BUS && reg + len > PCI_PRIMARY_BUS &&
+	    vmd_is_bus1_root_bridge(vmd, bus, devfn)) {
+		u8 *buf = (u8 *)value;
+
+		buf[PCI_PRIMARY_BUS - reg] = bus->number;
+	}
+
+	return ret;
 }
 
 /*
@@ -610,12 +690,19 @@ static resource_size_t vmd_cfgbar_ecam_space(struct vmd_dev *vmd)
 	unsigned int ecam_start;
 
 	cfgbar_buses = resource_size(&vmd->dev->resource[VMD_CFGBAR]) >> 20;
-	ecam_start = vmd_bus_to_ecam(vmd, vmd->resources[VMD_RES_CFGBAR].start);
+
+	if (vmd->bus1_rootbus)
+		/* BUS1 splits the aperture, so reset scans start from BUS0. */
+		ecam_start = vmd_bus_to_ecam(vmd, vmd->busn_start[VMD_BUS_0]);
+	else
+		ecam_start = vmd_bus_to_ecam(vmd, vmd->resources[VMD_RES_CFGBAR].start);
+
 	if (ecam_start >= cfgbar_buses)
 		return 0;
 
 	return cfgbar_buses - ecam_start;
 }
+
 static void vmd_domain_reset(struct vmd_dev *vmd)
 {
 	u16 bus, max_buses = resource_size(&vmd->resources[VMD_RES_CFGBAR]);
@@ -623,10 +710,13 @@ static void vmd_domain_reset(struct vmd_dev *vmd)
 	unsigned int ecam_bus;
 	char __iomem *base;
 
+	if (vmd->bus1_rootbus)
+		max_buses = resource_size(&vmd->dev->resource[VMD_CFGBAR]) >> 20;
+
 	max_buses = min_t(u16, max_buses, vmd_cfgbar_ecam_space(vmd));
 	for (bus = 0; bus < max_buses; bus++) {
-		ecam_bus = vmd_bus_to_ecam(vmd, vmd->resources[VMD_RES_CFGBAR].start +
-					   bus);
+		/* Calculate the ECAM bus nr starting from the first bus of VMD_BUS_0 */
+		ecam_bus = vmd_bus_to_ecam(vmd, vmd->busn_start[VMD_BUS_0] + bus);
 		for (dev = 0; dev < 32; dev++) {
 			base = vmd->cfgbar + PCIE_ECAM_OFFSET(ecam_bus,
 						PCI_DEVFN(dev, 0), 0);
@@ -668,16 +758,53 @@ static void vmd_domain_reset(struct vmd_dev *vmd)
 	}
 }
 
-static void vmd_attach_resources(struct vmd_dev *vmd)
+static int vmd_attach_resources(struct vmd_dev *vmd)
 {
-	vmd->dev->resource[VMD_MEMBAR1].child = &vmd->resources[VMD_RES_MBAR_1];
-	vmd->dev->resource[VMD_MEMBAR2].child = &vmd->resources[VMD_RES_MBAR_2];
+	int ret;
+
+	ret = request_resource(&vmd->dev->resource[VMD_MEMBAR1],
+			       &vmd->resources[VMD_RES_MBAR_1]);
+	if (ret)
+		return ret;
+
+	ret = request_resource(&vmd->dev->resource[VMD_MEMBAR2],
+			       &vmd->resources[VMD_RES_MBAR_2]);
+	if (ret) {
+		release_resource(&vmd->resources[VMD_RES_MBAR_1]);
+		return ret;
+	}
+
+	if (vmd->bus1_rootbus && vmd->sysdata[VMD_BUS_1].domain >= 0) {
+		ret = request_resource(&vmd->dev->resource[VMD_MEMBAR1],
+				       &vmd->resources[VMD_RES_BUS1_MBAR_1]);
+		if (ret) {
+			release_resource(&vmd->resources[VMD_RES_MBAR_1]);
+			release_resource(&vmd->resources[VMD_RES_MBAR_2]);
+			return ret;
+		}
+
+		ret = request_resource(&vmd->dev->resource[VMD_MEMBAR2],
+				       &vmd->resources[VMD_RES_BUS1_MBAR_2]);
+		if (ret) {
+			release_resource(&vmd->resources[VMD_RES_MBAR_1]);
+			release_resource(&vmd->resources[VMD_RES_MBAR_2]);
+			release_resource(&vmd->resources[VMD_RES_BUS1_MBAR_1]);
+			return ret;
+		}
+	}
+
+	return 0;
 }
 
 static void vmd_detach_resources(struct vmd_dev *vmd)
 {
-	vmd->dev->resource[VMD_MEMBAR1].child = NULL;
-	vmd->dev->resource[VMD_MEMBAR2].child = NULL;
+	if (vmd->bus1_rootbus && vmd->sysdata[VMD_BUS_1].domain >= 0) {
+		release_resource(&vmd->resources[VMD_RES_BUS1_MBAR_2]);
+		release_resource(&vmd->resources[VMD_RES_BUS1_MBAR_1]);
+	}
+
+	release_resource(&vmd->resources[VMD_RES_MBAR_2]);
+	release_resource(&vmd->resources[VMD_RES_MBAR_1]);
 }
 
 static int vmd_get_phys_offsets(struct vmd_dev *vmd, bool native_hint,
@@ -734,7 +861,7 @@ static int vmd_get_phys_offsets(struct vmd_dev *vmd, bool native_hint,
 	return 0;
 }
 
-static int vmd_get_bus_number_start(struct vmd_dev *vmd)
+static int vmd_get_bus_number_start(struct vmd_dev *vmd, unsigned long features)
 {
 	struct pci_dev *dev = vmd->dev;
 	u16 reg;
@@ -752,10 +879,27 @@ static int vmd_get_bus_number_start(struct vmd_dev *vmd)
 		case 1:
 			vmd->busn_start[VMD_BUS_0] = VMD_RESTRICT_1_BUS_START;
 			break;
-		case 3:
 		case 2:
 			vmd->busn_start[VMD_BUS_0] = VMD_RESTRICT_2_BUS_START;
 			break;
+		case 3:
+			if (!(features & VMD_FEAT_HAS_BUS1_ROOTBUS)) {
+				pci_err(dev, "VMD Bus Restriction detected type %d, but BUS1 root bus is not supported, aborting.\n",
+					BUS_RESTRICT_CFG(reg));
+				return -ENODEV;
+			}
+
+			/*
+			 * Per Intel VMD specification (e.g., Arrow Lake-S):
+			 * - VMD (on root bus 0) secondary bus start number: 0xE0
+			 *	(devices span 0xE2..0xF0 via VMD_BUSRANGE0).
+			 * - VMD (on root bus 1) secondary bus start number: 0xE1
+			 *	(devices span 0xF1..0xFF via VMD_BUSRANGE1).
+			 */
+			vmd->busn_start[VMD_BUS_0] = VMD_RESTRICT_2_BUS_START;
+			vmd->busn_start[VMD_BUS_1] = VMD_RESTRICT_3_BUS_START;
+			vmd->bus1_rootbus = true;
+			break;
 		default:
 			pci_err(dev, "Unknown Bus Offset Setting (%d)\n",
 				BUS_RESTRICT_CFG(reg));
@@ -908,7 +1052,7 @@ static int vmd_prepare_offsets_and_bus(struct vmd_dev *vmd,
 	 * limits the bus range to between 0-127, 128-255, or 224-255.
 	 */
 	if (features & VMD_FEAT_HAS_BUS_RESTRICTIONS) {
-		ret = vmd_get_bus_number_start(vmd);
+		ret = vmd_get_bus_number_start(vmd, features);
 		if (ret)
 			return ret;
 	}
@@ -975,7 +1119,7 @@ static int vmd_pm_enable_quirk(struct pci_dev *pdev, void *userdata)
 	return 0;
 }
 
-static void vmd_configure_cfgbar(struct vmd_dev *vmd)
+static int vmd_configure_cfgbar(struct vmd_dev *vmd)
 {
 	struct resource *res = &vmd->dev->resource[VMD_CFGBAR];
 	resource_size_t busn_end;
@@ -989,6 +1133,62 @@ static void vmd_configure_cfgbar(struct vmd_dev *vmd)
 		.end   = busn_end,
 		.flags = IORESOURCE_BUS | IORESOURCE_PCI_FIXED,
 	};
+
+	if (vmd->bus1_rootbus) {
+		int ret;
+		u16 bus0_range = 0;
+		u16 bus1_range = 0;
+		u8 bus0_start, bus0_end;
+		u8 bus1_start, bus1_end;
+
+		ret = pci_read_config_word(vmd->dev, VMD_BUSRANGE0, &bus0_range);
+		if (ret) {
+			pci_err(vmd->dev, "Failed to read VMD_BUSRANGE0: %d\n", ret);
+			return -EIO;
+		}
+
+		ret = pci_read_config_word(vmd->dev, VMD_BUSRANGE1, &bus1_range);
+		if (ret) {
+			pci_err(vmd->dev, "Failed to read VMD_BUSRANGE1: %d\n", ret);
+			return -EIO;
+		}
+
+		bus0_start = VMD_BUS_START(bus0_range);
+		bus0_end = VMD_BUS_END(bus0_range);
+		bus1_start = VMD_BUS_START(bus1_range);
+		bus1_end = VMD_BUS_END(bus1_range);
+
+		if (bus0_start > bus0_end || bus1_start > bus1_end) {
+			pci_err(vmd->dev,
+				"Invalid bus range(s): BUS0 [%02x-%02x], BUS1 [%02x-%02x]\n",
+				bus0_start, bus0_end, bus1_start, bus1_end);
+			return -EINVAL;
+		}
+
+		if (!(bus0_end < bus1_start || bus1_end < bus0_start)) {
+			pci_err(vmd->dev,
+				"Overlapping bus ranges: BUS0 [%02x-%02x], BUS1 [%02x-%02x]\n",
+				bus0_start, bus0_end, bus1_start, bus1_end);
+			return -EINVAL;
+		}
+
+		/*
+		 * Resize BUS0 CFGBAR range to make space for BUS1
+		 * owned devices by adjusting range end with value stored in
+		 * VMD_BUSRANGE0 register.
+		 */
+		vmd->resources[VMD_RES_CFGBAR].start = bus0_start;
+		vmd->resources[VMD_RES_CFGBAR].end = bus0_end;
+
+		vmd->resources[VMD_RES_BUS1_CFGBAR] = (struct resource){
+			.name = "VMD CFGBAR BUS1",
+			.start = bus1_start,
+			.end = bus1_end,
+			.flags = IORESOURCE_BUS | IORESOURCE_PCI_FIXED,
+		};
+	}
+
+	return 0;
 }
 
 /*
@@ -1019,8 +1219,9 @@ static int vmd_configure_membar(struct vmd_dev *vmd, u8 resource_number,
 	if (!upper_bits)
 		flags &= ~IORESOURCE_MEM_64;
 
-	name = devm_kasprintf(&vmd->dev->dev, GFP_KERNEL, "VMD MEMBAR%d",
-			      resource_number);
+	name = devm_kasprintf(&vmd->dev->dev, GFP_KERNEL, "VMD MEMBAR%d %s",
+			      membar_number / 2,
+			      resource_number > VMD_RES_MBAR_2 ? "BUS1" : "");
 	if (!name)
 		return -ENOMEM;
 
@@ -1040,55 +1241,132 @@ static int vmd_configure_membar1_membar2(struct vmd_dev *vmd,
 {
 	int ret;
 
-	ret = vmd_configure_membar(vmd, VMD_RES_MBAR_1, VMD_MEMBAR1, 0, 0);
-	if (ret)
-		return ret;
+	if (vmd->bus1_rootbus) {
+		u32 reg = 0;
+		u32 bus1_mbar1_ofs = 0;
+		u64 bus1_mbar2_ofs = 0;
+		resource_size_t mbar1_sz, mbar2_sz;
 
-	ret = vmd_configure_membar(vmd, VMD_RES_MBAR_2, VMD_MEMBAR2, mbar2_ofs, 0);
-	if (ret) {
-		devm_kfree(&vmd->dev->dev, (void *)vmd->resources[VMD_RES_MBAR_1].name);
-		memset(&vmd->resources[VMD_RES_MBAR_1], 0,
-		       sizeof(vmd->resources[VMD_RES_MBAR_1]));
-		return ret;
+		mbar1_sz = resource_size(&vmd->dev->resource[VMD_MEMBAR1]);
+		mbar2_sz = resource_size(&vmd->dev->resource[VMD_MEMBAR2]);
+
+		ret = pci_read_config_dword(vmd->dev, VMD_MEMBAR1_OFFSET,
+					    &bus1_mbar1_ofs);
+		if (ret)
+			return -EIO;
+
+		ret = pci_read_config_dword(vmd->dev, VMD_MEMBAR2_OFFSET1, &reg);
+		if (ret)
+			return -EIO;
+		bus1_mbar2_ofs = reg;
+
+		ret = pci_read_config_dword(vmd->dev, VMD_MEMBAR2_OFFSET2, &reg);
+		if (ret)
+			return -EIO;
+		bus1_mbar2_ofs |= (u64)reg << 32;
+
+		if (!bus1_mbar1_ofs || bus1_mbar1_ofs >= mbar1_sz) {
+			pci_err(vmd->dev, "Invalid MEMBAR1 offset %#llx (BAR size %#llx)\n",
+				(unsigned long long)bus1_mbar1_ofs,
+				(unsigned long long)mbar1_sz);
+			return -EINVAL;
+		}
+
+		if (!bus1_mbar2_ofs || mbar2_ofs >= mbar2_sz ||
+		    bus1_mbar2_ofs <= mbar2_ofs || bus1_mbar2_ofs >= mbar2_sz) {
+			pci_err(vmd->dev,
+				"Invalid MEMBAR2 split %#llx (shadow offset %#llx, BAR size %#llx)\n",
+				(unsigned long long)bus1_mbar2_ofs,
+				(unsigned long long)mbar2_ofs,
+				(unsigned long long)mbar2_sz);
+			return -EINVAL;
+		}
+
+		/*
+		 * Resize BUS MEMBAR1 and MEMBAR2 ranges to make space
+		 * for BUS1 owned devices by adjusting range end with values
+		 * stored in VMD_MEMBAR1_OFFSET and VMD_MEMBAR2_OFFSET registers
+		 */
+		ret = vmd_configure_membar(vmd, VMD_RES_MBAR_1, VMD_MEMBAR1, 0,
+					   mbar1_sz - bus1_mbar1_ofs);
+		if (ret)
+			return ret;
+
+		ret = vmd_configure_membar(vmd, VMD_RES_MBAR_2, VMD_MEMBAR2,
+					   mbar2_ofs, mbar2_sz - bus1_mbar2_ofs);
+		if (ret)
+			return ret;
+
+		ret = vmd_configure_membar(vmd, VMD_RES_BUS1_MBAR_1, VMD_MEMBAR1,
+					   bus1_mbar1_ofs, 0);
+		if (ret)
+			return ret;
+
+		ret = vmd_configure_membar(vmd, VMD_RES_BUS1_MBAR_2, VMD_MEMBAR2,
+					   bus1_mbar2_ofs, 0);
+		if (ret)
+			return ret;
+	} else {
+		ret = vmd_configure_membar(vmd, VMD_RES_MBAR_1, VMD_MEMBAR1, 0, 0);
+		if (ret)
+			return ret;
+
+		ret = vmd_configure_membar(vmd, VMD_RES_MBAR_2, VMD_MEMBAR2,
+					   mbar2_ofs, 0);
+		if (ret)
+			return ret;
 	}
 
 	return 0;
 }
 
-static int vmd_create_bus(struct vmd_dev *vmd, struct pci_sysdata *sd,
-			  resource_size_t *offset, bool vmd_in_guest)
+static int vmd_create_bus(struct vmd_dev *vmd, enum vmd_rootbus bus_number,
+			  struct pci_sysdata *sd, resource_size_t *offset,
+			  bool vmd_in_guest, u8 primary)
 {
+	u8 cfgbar = bus_number * 3;
+	u8 membar1 = cfgbar + 1;
+	u8 membar2 = cfgbar + 2;
+	struct pci_bus *vmd_bus;
 	LIST_HEAD(resources);
 
-	pci_add_resource(&resources, &vmd->resources[VMD_RES_CFGBAR]);
-	pci_add_resource_offset(&resources, &vmd->resources[VMD_RES_MBAR_1],
+	pci_add_resource(&resources, &vmd->resources[cfgbar]);
+	pci_add_resource_offset(&resources, &vmd->resources[membar1],
 				offset[0]);
-	pci_add_resource_offset(&resources, &vmd->resources[VMD_RES_MBAR_2],
+	pci_add_resource_offset(&resources, &vmd->resources[membar2],
 				offset[1]);
 
-	vmd->bus[VMD_BUS_0] = pci_create_root_bus(&vmd->dev->dev,
-						  vmd->busn_start[VMD_BUS_0],
-						  &vmd_ops, sd, &resources);
-	if (!vmd->bus[VMD_BUS_0]) {
+	vmd_bus = pci_create_root_bus(&vmd->dev->dev,
+				      vmd->busn_start[bus_number], &vmd_ops, sd,
+				      &resources);
+
+	if (!vmd_bus) {
 		pci_free_resource_list(&resources);
 		return -ENODEV;
 	}
 
+	/*
+	 * pci_create_root_bus() does not initialise bus->primary.
+	 * Set it here before any scanning so bridge traversal logic
+	 * sees the correct upstream bus number from the start.
+	 */
+	vmd_bus->primary = primary;
+
 	/*
 	 * Don't copy _OSC control flags from root bridge if running in a VM, as
 	 * they don't reflect the physical root bridge capabilities.
 	 */
 	if (!vmd_in_guest)
 		vmd_copy_host_bridge_flags(pci_find_host_bridge(vmd->dev->bus),
-					   to_pci_host_bridge(vmd->bus[VMD_BUS_0]->bridge));
-
-	vmd_attach_resources(vmd);
+					   to_pci_host_bridge(vmd_bus->bridge));
 	if (vmd->irq_domain)
-		dev_set_msi_domain(&vmd->bus[VMD_BUS_0]->dev, vmd->irq_domain);
+		dev_set_msi_domain(&vmd_bus->dev, vmd->irq_domain);
 	else
-		dev_set_msi_domain(&vmd->bus[VMD_BUS_0]->dev,
+		dev_set_msi_domain(&vmd_bus->dev,
 				   dev_get_msi_domain(&vmd->dev->dev));
 
+	vmd->bus[bus_number] = vmd_bus;
+
 	return 0;
 }
 
@@ -1101,7 +1379,15 @@ static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features)
 	vmd_acpi_begin();
 
 	pci_scan_child_bus(bus);
-	vmd_domain_reset(vmd_from_bus(bus));
+
+	/*
+	 * vmd_domain_reset() walks the full VMD CFGBAR aperture, so a single
+	 * invocation from BUS0 resets bridge windows for the whole VMD domain,
+	 * including BUS1. Running it again during BUS1 enumeration would
+	 * re-clobber windows already assigned for BUS0.
+	 */
+	if (bus->primary == VMD_PRIMARY_BUS0)
+		vmd_domain_reset(vmd_from_bus(bus));
 
 	/*
 	 * When Intel VMD is enabled, the OS does not discover the Root Ports
@@ -1140,6 +1426,16 @@ static void vmd_bus_enumeration(struct pci_bus *bus, unsigned long features)
 	vmd_acpi_end();
 }
 
+/* Drop BUS1 configuration and continue with BUS0 only. */
+static void vmd_disable_bus1(struct vmd_dev *vmd)
+{
+	vmd->resources[VMD_RES_BUS1_CFGBAR] = (struct resource){};
+	vmd->resources[VMD_RES_BUS1_MBAR_1] = (struct resource){};
+	vmd->resources[VMD_RES_BUS1_MBAR_2] = (struct resource){};
+	vmd->sysdata[VMD_BUS_1].domain = PCI_DOMAIN_NR_NOT_SET;
+	vmd->bus[VMD_BUS_1] = NULL;
+}
+
 static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 {
 	struct pci_sysdata *sd = &vmd->sysdata[VMD_BUS_0];
@@ -1153,7 +1449,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	if (ret)
 		return ret;
 
-	vmd_configure_cfgbar(vmd);
+	ret = vmd_configure_cfgbar(vmd);
+	if (ret)
+		return ret;
 
 	/*
 	 * If the window is below 4GB, clear IORESOURCE_MEM_64 so we can
@@ -1213,11 +1511,38 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 
 	sd->node = pcibus_to_node(vmd->dev->bus);
 
-	ret = vmd_create_bus(vmd, sd, offset, vmd_in_guest);
+	if (vmd->bus1_rootbus) {
+		vmd->sysdata[VMD_BUS_1].vmd_dev = vmd->dev;
+		vmd->sysdata[VMD_BUS_1].node = pcibus_to_node(vmd->dev->bus);
+		vmd->sysdata[VMD_BUS_1].domain = pci_bus_find_emul_domain_nr(0, 0x10000, INT_MAX);
+		if (vmd->sysdata[VMD_BUS_1].domain < 0) {
+			pci_warn(vmd->dev,
+				 "Can't allocate BUS1 domain: %d, continuing with BUS0 only\n",
+				 vmd->sysdata[VMD_BUS_1].domain);
+			vmd_disable_bus1(vmd);
+		}
+	}
+
+	ret = vmd_attach_resources(vmd);
+	if (ret) {
+		pci_bus_release_emul_domain_nr(sd->domain);
+		if (vmd->bus1_rootbus && vmd->sysdata[VMD_BUS_1].domain >= 0)
+			pci_bus_release_emul_domain_nr(vmd->sysdata[VMD_BUS_1].domain);
+
+		vmd_remove_irq_domain(vmd);
+		return ret;
+	}
+
+	ret = vmd_create_bus(vmd, VMD_BUS_0, sd, offset, vmd_in_guest,
+			     VMD_PRIMARY_BUS0);
 	if (ret) {
 		pci_err(vmd->dev, "Can't create bus: %d\n", ret);
 		pci_bus_release_emul_domain_nr(sd->domain);
+		if (vmd->bus1_rootbus && vmd->sysdata[VMD_BUS_1].domain >= 0)
+			pci_bus_release_emul_domain_nr(vmd->sysdata[VMD_BUS_1].domain);
+
 		vmd_remove_irq_domain(vmd);
+		vmd_detach_resources(vmd);
 		return ret;
 	}
 
@@ -1225,8 +1550,33 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 			       &vmd->bus[VMD_BUS_0]->dev.kobj, "domain"),
 	     "Can't create symlink to domain\n");
 
+	if (vmd->bus1_rootbus && vmd->sysdata[VMD_BUS_1].domain >= 0) {
+		ret = vmd_create_bus(vmd, VMD_BUS_1, &vmd->sysdata[VMD_BUS_1], offset,
+				     vmd_in_guest, VMD_PRIMARY_BUS1);
+		if (ret) {
+			pci_warn(vmd->dev,
+				 "Can't create BUS1: %d, continuing with BUS0 only\n",
+				 ret);
+
+			release_resource(&vmd->resources[VMD_RES_BUS1_MBAR_2]);
+			release_resource(&vmd->resources[VMD_RES_BUS1_MBAR_1]);
+			pci_bus_release_emul_domain_nr(vmd->sysdata[VMD_BUS_1].domain);
+			vmd_disable_bus1(vmd);
+		}
+
+		if (vmd->bus1_rootbus && vmd->sysdata[VMD_BUS_1].domain >= 0) {
+			WARN(sysfs_create_link(&vmd->dev->dev.kobj,
+					       &vmd->bus[VMD_BUS_1]->dev.kobj,
+					       "domain1"),
+			     "Can't create symlink to domain1\n");
+		}
+	}
+
 	vmd_bus_enumeration(vmd->bus[VMD_BUS_0], features);
 
+	if (vmd->bus1_rootbus && vmd->sysdata[VMD_BUS_1].domain >= 0)
+		vmd_bus_enumeration(vmd->bus[VMD_BUS_1], features);
+
 	return 0;
 }
 
@@ -1263,6 +1613,7 @@ static int vmd_probe(struct pci_dev *dev, const struct pci_device_id *id)
 
 	vmd->dev = dev;
 	vmd->sysdata[VMD_BUS_0].domain = PCI_DOMAIN_NR_NOT_SET;
+	vmd->sysdata[VMD_BUS_1].domain = PCI_DOMAIN_NR_NOT_SET;
 	vmd->features = features;
 	vmd->instance = ida_alloc(&vmd_instance_ida, GFP_KERNEL);
 	if (vmd->instance < 0)
@@ -1301,8 +1652,14 @@ static int vmd_probe(struct pci_dev *dev, const struct pci_device_id *id)
 	if (err)
 		goto out_release_instance;
 
-	dev_info(&vmd->dev->dev, "Bound to PCI domain %04x\n",
-		 vmd->sysdata[VMD_BUS_0].domain);
+	if (vmd->bus1_rootbus && vmd->sysdata[VMD_BUS_1].domain >= 0)
+		dev_info(&vmd->dev->dev, "Bound to PCI domains %04x and %04x\n",
+			 vmd->sysdata[VMD_BUS_0].domain,
+			 vmd->sysdata[VMD_BUS_1].domain);
+	else
+		dev_info(&vmd->dev->dev, "Bound to PCI domain %04x\n",
+			 vmd->sysdata[VMD_BUS_0].domain);
+
 	return 0;
 
  out_release_instance:
@@ -1325,6 +1682,14 @@ static void vmd_remove(struct pci_dev *dev)
 	pci_stop_root_bus(vmd->bus[VMD_BUS_0]);
 	sysfs_remove_link(&vmd->dev->dev.kobj, "domain");
 	pci_remove_root_bus(vmd->bus[VMD_BUS_0]);
+
+	if (vmd->bus1_rootbus && vmd->bus[VMD_BUS_1]) {
+		pci_stop_root_bus(vmd->bus[VMD_BUS_1]);
+		sysfs_remove_link(&vmd->dev->dev.kobj, "domain1");
+		pci_remove_root_bus(vmd->bus[VMD_BUS_1]);
+		pci_bus_release_emul_domain_nr(vmd->sysdata[VMD_BUS_1].domain);
+	}
+
 	vmd_cleanup_srcu(vmd);
 	vmd_detach_resources(vmd);
 	vmd_remove_irq_domain(vmd);
@@ -1425,4 +1790,4 @@ module_pci_driver(vmd_drv);
 MODULE_AUTHOR("Intel Corporation");
 MODULE_DESCRIPTION("Volume Management Device driver");
 MODULE_LICENSE("GPL v2");
-MODULE_VERSION("0.6");
+MODULE_VERSION("0.7");
-- 
2.43.0

Re: [PATCH v10 0/7] VMD add second root bus support

[Manivannan Sadhasivam]

On Fri, 06 Nov 2026 18:42:48 +0000, Szymon Durawa wrote:
> This series adds support for a second VMD root bus.
> 
> Today the VMD driver can enumerate and manage devices only behind the
> first VMD root bus. Starting with Intel Arrow Lake, VMD may expose a
> second root bus (BUS1), and devices behind that bus are not discovered
> by the current driver.
> 
> [...]

Applied, thanks!

[1/7] PCI: vmd: Add vmd_bus_enumeration() helper function
      commit: 9ec146e311d1bb69e1e76e523d5a2631838e9f2d
[2/7] PCI: vmd: Add vmd_configure_cfgbar() helper function
      commit: 2f3f85305a90be0b1f71636a55c286db49a49cb8
[3/7] PCI: vmd: Add vmd_configure_membar() and vmd_configure_membar1_membar2()
      commit: ed34664daffa63d455fc3a614d0722e57183a9ce
[4/7] PCI: vmd: Add vmd_create_bus()
      commit: b294e381f1de731dfbf8f7b150818fbb73120622
[5/7] PCI: vmd: Replace hardcoded values with enum and defines
      commit: 7aa850769255689870bd19080da2a398ac583d93
[6/7] PCI: vmd: Convert bus, busn_start and sysdata to an array
      commit: 99b617df5c36773206c492790863985137fb189a
[7/7] PCI: vmd: Add support for second rootbus under VMD
      commit: 4d790512e44ae235166faddf7cf10ade0c86c4a8

Best regards,
-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH v10 0/7] VMD add second root bus support

[Vladimir Nedoshivin]

Hello Szymon, Bjorn, and PCI maintainers,

I have successfully tested the complete VMD Second Root Bus v10
patch series on my MSI Vector 17 HX AI laptop with Intel Arrow Lake
and VMD controller 8086:ad0b.

I would also like to provide some background on the original problems,
the previously developed workaround, and the relationship between this
series and the ARL004 interrupt-ordering fix.

The original investigation is documented here:

https://bugs.launchpad.net/bugs/2166325

Hardware:
- Laptop: MSI Vector 17 HX AI A2XWIG
- CPU: Intel Core Ultra 9 275HX
- VMD controller: Intel 8086:ad0b
- VMD PCI address: 0000:00:0e.0
- NVMe: Phison 1TB ESR01TBYCCA4-EDJ-2MS
- Operating system: Ubuntu 26.04
- Secure Boot: enabled

1. Original problem: NVMe device not detected

With the stock Ubuntu 7.0.0-30/31 VMD driver, the internal NVMe SSD
was not detected.

The kernel reported:

  vmd 0000:00:0e.0: Unknown Bus Offset Setting (3)

As a result, the NVMe device behind Intel VMD was not enumerated.

The laptop firmware does not provide a usable option to disable
Intel VMD/RST.

I initially integrated an earlier revision of your Second Root Bus
patch series into the Ubuntu VMD driver.

This successfully restored NVMe enumeration, but exposed a second
hardware-related problem.

2. Second problem: ARL004 interrupt-ordering erratum

After restoring NVMe enumeration, storage operations experienced
repeated stalls of approximately 30 seconds.

The kernel reported:

  nvme nvme0: I/O tag ... timeout, completion polled

A diagnostic sequential read of 1 GiB required approximately
121.339 seconds, corresponding to about 8.8 MB/s.

The investigation identified the Intel Arrow Lake ARL004 erratum:

  MSI from VMD-Owned Device May Pass Memory Write

An MSI may reach the processor before preceding posted writes from
a VMD-owned device become visible in memory.

This can cause the NVMe interrupt handler to observe stale
completion-queue data.

An experimental workaround was implemented by performing a PCI
configuration-space read from the MSI-initiating device before
dispatching the corresponding interrupt.

With this workaround, the same diagnostic read completed in
approximately 0.882 seconds.

A subsequent 10 GiB read also completed normally, without the
previously observed NVMe completion timeouts.

3. Upstream ARL004 fix

Following the initial investigation, I submitted an upstream patch
enabling the existing VMD interrupt-ordering quirk for Intel
Arrow Lake VMD device 0xad0b.

The patch was accepted by Bjorn Helgaas under the title:

  PCI: vmd: Flush DMA writes before handling MSI on Arrow Lake

Upstream commit:

  2c88c2dc1852

The accepted implementation uses the existing
VMD_FEAT_INTERRUPT_QUIRK mechanism.

This is separate from the Second Root Bus feature.

The original Ubuntu investigation and test results are available
in Launchpad bug #2166325.

4. Testing the Second Root Bus v10 series

I have now tested all seven patches from your v10 series,
based on Linux 7.4-rc1.

To preserve my working Ubuntu installation, I compiled the patched
vmd.c as an external module against the Ubuntu 7.0.0-43-generic
kernel headers.

The only source compatibility adjustment was replacing
PCI_DEVICE_ID_INTEL_VMD_28C1 with its numeric value, 0x28c1,
because the identifier was unavailable in the Ubuntu headers.

The resulting module was signed using an enrolled MOK and loaded
through a separate test initramfs.

The test therefore used:
- Ubuntu kernel 7.0.0-43-generic
- VMD source from Linux 7.4-rc1
- All seven Second Root Bus v10 patches
- The upstream ARL004 interrupt-ordering quirk already present
  in the 7.4-rc1 base

No additional local ARL004 workaround was applied to the v10 source.

The loaded module was verified using srcversion:

  45EF68AF5A6E01AAA2882CF

5. Test results

The system booted successfully.

Both VMD PCI domains were created:

  vmd 0000:00:0e.0: PCI host bridge to bus 10000:e0
  vmd 0000:00:0e.0: PCI host bridge to bus 10001:e1
  vmd 0000:00:0e.0: Bound to PCI domains 10000 and 10001

The NVMe controller was successfully detected behind BUS1:

  nvme nvme0: pci function 10001:f1:00.0

The NVMe controller initialized successfully, and the Ubuntu
root filesystem mounted normally.

Sequential read performance:

  Command:
  dd if=/dev/nvme0n1 of=/dev/null bs=4M count=1024 iflag=direct

  Data read: 4 GiB
  Duration: 4.16695 seconds
  Throughput: approximately 983 MiB/s

This is comparable to the approximately 984 MiB/s previously
observed with my working ARL004-specific driver.

Hardware MSI-X interrupts were active.

During a separate 1 GiB direct-read test, the aggregate VMD
interrupt count increased:

  Before: 134377
  After:  138680
  Difference: +4303

One observation worth mentioning:

The virtual NVMe interrupt counters (nvme0q*) remained at zero
in /proc/interrupts, while the underlying physical VMD MSI-X
counters were active.

Despite this, NVMe I/O completed successfully, and no VMD/NVMe
errors or timeouts were found in the checked kernel logs.

6. Test limitations

This was a test of the v10 VMD source built as an external module
against Ubuntu 7.0.0-43, not a boot test of the complete
Linux 7.4-rc1 kernel.

The completed testing covered boot, device enumeration,
read-only NVMe I/O, and interrupt activity.

Dedicated write stress testing, suspend/resume, and long-term
stability testing have not yet been performed.

7. Conclusion

The Second Root Bus v10 series successfully resolves the NVMe
enumeration problem on this Intel Arrow Lake VMD 8086:ad0b platform.

Combined with the previously accepted upstream ARL004
interrupt-ordering fix, the system boots successfully, detects
the NVMe controller behind BUS1, and achieves normal sequential
read performance.

No functional regressions were observed during the completed tests.

Thank you for your continued work on Second Root Bus support.

Tested-by: Vladimir Nedoshivin <gvozd188@mail.ru>

Best regards,
Vladimir Nedoshivin