[PATCH v2 1/1] PCI: vmd: Avoid acceidental enablement of window when zeroing config space of VMD root ports

[Adrian Huang <adrianhuang0701@gmail.com>]

From: Adrian Huang <ahuang12@lenovo.com>

Commit 6aab5622296b ("PCI: vmd: Clean up domain before enumeration")
clears PCI configuration space of VMD root ports. However, the host OS
cannot boot successfully with the following error message:

  vmd 0000:64:05.5: PCI host bridge to bus 10000:00
  ...
  vmd 0000:64:05.5: Bound to PCI domain 10000
  ...
  DMAR: VT-d detected Invalidation Queue Error: Reason f
  DMAR: VT-d detected Invalidation Time-out Error: SID ffff
  DMAR: VT-d detected Invalidation Completion Error: SID ffff
  DMAR: QI HEAD: UNKNOWN qw0 = 0x0, qw1 = 0x0
  DMAR: QI PRIOR: UNKNOWN qw0 = 0x0, qw1 = 0x0
  DMAR: Invalidation Time-out Error (ITE) cleared

The root cause is that memset_io() clears prefetchable memory base/limit
registers and prefetchable base/limit 32 bits registers sequentially. This
might enable prefetchable memory if the device disables prefetchable memory
originally. Here is an example (before memset_io()):

  PCI configuration space for 10000:00:00.0:
  86 80 30 20 06 00 10 00 04 00 04 06 00 00 01 00
  00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 20
  00 00 00 00 01 00 01 00 ff ff ff ff 75 05 00 00
  00 00 00 00 40 00 00 00 00 00 00 00 00 01 02 00

So, prefetchable memory is ffffffff00000000-575000fffff, which is disabled.
Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:

  The Prefetchable Memory Limit register must be programmed to a smaller
  value than the Prefetchable Memory Base register if there is no
  prefetchable memory on the secondary side of the bridge.

When memset_io() clears prefetchable base 32 bits register, the
prefetchable memory becomes 0000000000000000-575000fffff, which is enabled.
This behavior (accidental enablement of window) causes that config accesses
get routed to the wrong place, and the access content of PCI configuration
space of VMD root ports is 0xff after invoking memset_io() in
vmd_domain_reset():

  10000:00:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev ff) (prog-if ff)
          !!! Unknown header type 7f
  00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
  ...
  f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

  10000:00:01.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port B (rev ff) (prog-if ff)
          !!! Unknown header type 7f
  00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
  ...
  f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

To fix the issue, prefetchable limit upper 32 bits register needs to be
cleared firstly. This also adheres to the implementation of
pci_setup_bridge_mmio_pref(). Please see the function for detail.

Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=216644
Fixes: 6aab5622296b ("PCI: vmd: Clean up domain before enumeration")
Cc: Nirmal Patel <nirmal.patel@linux.intel.com>
Signed-off-by: Adrian Huang <ahuang12@lenovo.com>
Reviewed-by: Jon Derrick <jonathan.derrick@linux.dev>
---
Changes since v1:
  - Changed subject per Bjorn's suggestion

 drivers/pci/controller/vmd.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
index 769eedeb8802..e520aec55b68 100644
--- a/drivers/pci/controller/vmd.c
+++ b/drivers/pci/controller/vmd.c
@@ -526,6 +526,9 @@ static void vmd_domain_reset(struct vmd_dev *vmd)
 				     PCI_CLASS_BRIDGE_PCI))
 					continue;
 
+				/* Clear the upper 32 bits of PREF limit. */
+				memset_io(base + PCI_PREF_LIMIT_UPPER32, 0, 4);
+
 				memset_io(base + PCI_IO_BASE, 0,
 					  PCI_ROM_ADDRESS1 - PCI_IO_BASE);
 			}
-- 
2.31.1