Re: [PATCH v4] PCI: vmd: Disable bridge window for domain reset

["Patel, Nirmal" <nirmal.patel@linux.intel.com>]

On 8/9/2023 8:24 PM, Bjorn Helgaas wrote:
> On Wed, Aug 09, 2023 at 04:14:06PM -0700, Patel, Nirmal wrote:
>> On 8/9/2023 3:00 PM, Bjorn Helgaas wrote:
>>> On Wed, Aug 09, 2023 at 05:14:54PM -0400, Nirmal Patel wrote:
>>>> During domain reset process vmd_domain_reset() clears PCI
>>>> configuration space of VMD root ports. But certain platform
>>>> has observed following errors and failed to boot.
>>>>   ...
>>>>   DMAR: VT-d detected Invalidation Queue Error: Reason f
>>>>   DMAR: VT-d detected Invalidation Time-out Error: SID ffff
>>>>   DMAR: VT-d detected Invalidation Completion Error: SID ffff
>>>>   DMAR: QI HEAD: UNKNOWN qw0 = 0x0, qw1 = 0x0
>>>>   DMAR: QI PRIOR: UNKNOWN qw0 = 0x0, qw1 = 0x0
>>>>   DMAR: Invalidation Time-out Error (ITE) cleared
>>>>
>>>> The root cause is that memset_io() clears prefetchable memory base/limit
>>>> registers and prefetchable base/limit 32 bits registers sequentially.
>>>> This seems to be enabling prefetchable memory if the device disabled
>>>> prefetchable memory originally.
>>>>
>>>> Here is an example (before memset_io()):
>>>>
>>>>   PCI configuration space for 10000:00:00.0:
>>>>   86 80 30 20 06 00 10 00 04 00 04 06 00 00 01 00
>>>>   00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 20
>>>>   00 00 00 00 01 00 01 00 ff ff ff ff 75 05 00 00
>>>>   ...
>>>>
>>>> So, prefetchable memory is ffffffff00000000-575000fffff, which is
>>>> disabled. When memset_io() clears prefetchable base 32 bits register,
>>>> the prefetchable memory becomes 0000000000000000-575000fffff, which is
>>>> enabled and incorrect.
>>> It's not clear to me how this window config causes the VT-d errors.
>>> But empirically it seems to be related, and maybe that's enough.
>>>
>>>> Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:
>>>>
>>>>   The Prefetchable Memory Limit register must be programmed to a smaller
>>>>   value than the Prefetchable Memory Base register if there is no
>>>>   prefetchable memory on the secondary side of the bridge.
>>>>
>>>> This is believed to be the reason for the failure and in addition the
>>>> sequence of operation in vmd_domain_reset() is not following the PCIe
>>>> specs.
>>>>
>>>> Disable the bridge window by executing a sequence of operations
>>>> borrowed from pci_disable_bridge_window() and pci_setup_bridge_io(),
>>>> that comply with the PCI specifications.
>>>>
>>>> Signed-off-by: Nirmal Patel <nirmal.patel@linux.intel.com>
>>>> ---
>>>> v3->v4: Following same operation as pci_setup_bridge_io.
>>>> v2->v3: Add more information to commit description.
>>>> v1->v2: Follow same chain of operation as pci_disable_bridge_window
>>>>         and update commit log.
>>>> ---
>>>>  drivers/pci/controller/vmd.c | 17 +++++++++++++++--
>>>>  1 file changed, 15 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
>>>> index 769eedeb8802..ae5b4c1704e4 100644
>>>> --- a/drivers/pci/controller/vmd.c
>>>> +++ b/drivers/pci/controller/vmd.c
>>>> @@ -526,8 +526,21 @@ static void vmd_domain_reset(struct vmd_dev *vmd)
>>>>  				     PCI_CLASS_BRIDGE_PCI))
>>>>  					continue;
>>>>  
>>>> -				memset_io(base + PCI_IO_BASE, 0,
>>>> -					  PCI_ROM_ADDRESS1 - PCI_IO_BASE);
>>>> +				/* Temporarily disable the I/O range before updating PCI_IO_BASE */
>>>> +				writel(0x0000ffff, base + PCI_IO_BASE_UPPER16);
>>>> +				/* Update lower 16 bits of I/O base/limit */
>>>> +				writew(0x00f0, base + PCI_IO_BASE);
>>>> +				/* Update upper 16 bits of I/O base/limit */
>>>> +				writel(0, base + PCI_IO_BASE_UPPER16);
>>>> +
>>>> +				/* MMIO Base/Limit */
>>>> +				writel(0x0000fff0, base + PCI_MEMORY_BASE);
>>>> +
>>>> +				/* Prefetchable MMIO Base/Limit */
>>>> +				writel(0, base + PCI_PREF_LIMIT_UPPER32);
>>>> +				writel(0x0000fff0, base + PCI_PREF_MEMORY_BASE);
>>>> +				writel(0xffffffff, base + PCI_PREF_BASE_UPPER32);
>>>> +				writeb(0, base + PCI_CAPABILITY_LIST);
>>> What's the purpose of this PCI_CAPABILITY_LIST write?  I guess you
>>> don't want to find PM, MSI, MSI-X, PCIe, etc. capabilities?
>>>
>>> It's been there since the v1 patch, but the commit log only mentions
>>> disabling bridge windows.
>> I added it since it was part of original memset_io range. However
>> from your previous comment, I checked the lspci output for
>> PCI_CAPABILITY_LIST with and without the change and it doesn't seem
>> to make any difference.
> Ah, I see.  My guess is that was a mistake in the original memset_io()
> because I don't see a reason to clear PCI_CAPABILITY_LIST.
>
> PCI_CAPABILITY_LIST is HwInit, so should be read-only in terms of
> config accesses, and if lspci sees the same capability list before and
> after writing a zero to it, it sounds like it *is* read-only.
>
> Bjorn

Yes, I will remove it. Thanks.