Re: uefi secureboot vm and IO window overlap

[Bjorn Helgaas <helgaas@kernel.org>]

On Mon, Jun 12, 2023 at 05:46:44PM +0000, Kallol Biswas [C] wrote:
> -----Original Message-----
> From: Bjorn Helgaas <bjorn.helgaas@gmail.com> 
> Sent: Wednesday, June 7, 2023 3:15 PM
> To: Kallol Biswas [C] <kallol.biswas@nutanix.com>
> Cc: Bjorn Helgaas <helgaas@kernel.org>; linux-pci@vger.kernel.org
> Subject: Re: uefi secureboot vm and IO window overlap
> 
> > [FYI, I'm not sure why, but your email didn't seem to make it to the list; maybe there's a clue at https://urldefense.proofpoint.com/v2/url?u=http-3A__vger.kernel.org_majordomo-2Dinfo.html&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=HSy9eEcg9NrbI7YGUiURvZ2SD5_lmb_abr4v5sda9bk&m=VHASNjeAs866UrvKqCdEbB1HjTvZTYFLidSIbWc0malEoRTcGFfCjQi729H0C69-&s=irSta89Vw2NwFRSAc4FiIRvh0Dant62MNDdEvOteqP0&e= ]
> 
> On Wed, Jun 7, 2023 at 4:42 PM Kallol Biswas [C] <kallol.biswas@nutanix.com> wrote:
> >
> > Hello Bjorn,
> >                             I have reproduced the problem in the 6.3.6 kernel and debugged the source of the conflict.
> >
> > Here is the OVMF log:
> > PciBus: Resource Map for Root Bridge PciRoot(0x0)^M
> > Type =   Io16; Base = 0x6000;   Length = 0x3000;        Alignment = 0xFFF^M
> >    Base = 0x6000;       Length = 0x200; Alignment = 0xFFF;      Owner = PPB [00|02|02:**]^M
> >    Base = 0x7000;       Length = 0x200; Alignment = 0xFFF;      Owner = PPB [00|02|01:**]^M
> >    Base = 0x8000;       Length = 0x200; Alignment = 0xFFF;      Owner = PPB [00|02|00:**]^M
> >    Base = 0x8200;       Length = 0x40;  Alignment = 0x3F;       Owner = PCI [00|1F|03:20]^M
> >    Base = 0x8240;       Length = 0x20;  Alignment = 0x1F;       Owner = PCI [00|1F|02:20]^M
> >    Base = 0x8260;       Length = 0x20;  Alignment = 0x1F;       Owner = PCI [00|1D|02:20]^M
> >    Base = 0x8280;       Length = 0x20;  Alignment = 0x1F;       Owner = PCI [00|1D|01:20]^M
> >    Base = 0x82A0;       Length = 0x20;  Alignment = 0x1F;       Owner = PCI [00|1D|00:20]^M
> >    Base = 0x82C0;       Length = 0x20;  Alignment = 0x1F;       Owner = PCI [00|03|00:10]^M
> >
> > [nutanix@localhost ~]$ lspci -s 0:2.0 -vvv
> > 00:02.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode])
> >         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
> >         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> >         Latency: 0
> >         Interrupt: pin A routed to IRQ 22
> >         Region 0: Memory at c1645000 (32-bit, non-prefetchable) [size=4K]
> >         Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
> >         I/O behind bridge: 00008000-00008fff [size=4K]
> >         Memory behind bridge: c1400000-c15fffff [size=2M]
> >         Prefetchable memory behind bridge: 0000084000000000-00000840000fffff [size=1M]
> >         Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
> >         BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
> >                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> >         Capabilities: <access denied>
> >         Kernel driver in use: pcieport Dmesg log:
> > [    2.232081] pci 0000:00:02.0: PCI bridge to [bus 01]
> > [    2.232098] pci_read_bridge_io:base 0x8000 limit 0x8000 io_granulatiry 0x1000
> > [    2.232099] pci 0000:00:02.0:   bridge window [io  0x8000-0x8fff]
> > [    2.233005] pci 0000:00:02.0:   bridge window [mem 0xc1400000-0xc15fffff]
> > [    2.233034] pci 0000:00:02.0:   bridge window [mem 0x84000000000-0x840000fffff 64bit
> >
> > Kernel code:
> > static void pci_read_bridge_io(struct pci_bus *child) {
> >         struct pci_dev *dev = child->self;
> >         u8 io_base_lo, io_limit_lo;
> >         unsigned long io_mask, io_granularity, base, limit;
> >         struct pci_bus_region region;
> >         struct resource *res;
> >
> >         io_mask = PCI_IO_RANGE_MASK;
> >         io_granularity = 0x1000;
> >         if (dev->io_window_1k) {
> >                 /* Support 1K I/O space granularity */
> >                 io_mask = PCI_IO_1K_RANGE_MASK;
> >                 io_granularity = 0x400;
> >         }
> >
> >
> >         printk("pci_read_bridge_io:base 0x%x limit 0x%x io_granulatiry 0x%x\n", base, limit, io_granularity);  <= my print
> >         if (base <= limit) {
> >                 res->flags = (io_base_lo & PCI_IO_RANGE_TYPE_MASK) | IORESOURCE_IO;
> >                 region.start = base;
> >                 region.end = limit + io_granularity - 1;
> >                 pcibios_bus_to_resource(dev->bus, res, &region);
> >                 pci_info(dev, "  bridge window %pR\n", res);
> >         }
> >
> >
> > OVMF sets the base for 0:2.0 as 0x8000 and length as 0x200 but kernel 
> > io_granularity is 0x1000 So, the bridge window becomes 0x8000 to 
> > 0x8fff, which overlaps the OVMF programmed IO base registers for other endpoints.
> >
> > [    2.996029] pci 0000:00:03.0: can't claim BAR 0 [io  0x82c0-0x82df]: address conflict with PCI Bus 0000:01 [io  0x8000-0x8fff]
> > [    2.996049] pci 0000:00:1d.0: can't claim BAR 4 [io  0x82a0-0x82bf]: address conflict with PCI Bus 0000:01 [io  0x8000-0x8fff]
> > [    2.996058] pci 0000:00:1d.1: can't claim BAR 4 [io  0x8280-0x829f]: address conflict with PCI Bus 0000:01 [io  0x8000-0x8fff]
> > [    2.996068] pci 0000:00:1d.2: can't claim BAR 4 [io  0x8260-0x827f]: address conflict with PCI Bus 0000:01 [io  0x8000-0x8fff]
> > [    2.996093] pci 0000:00:1f.2: can't claim BAR 4 [io  0x8240-0x825f]: address conflict with PCI Bus 0000:01 [io  0x8000-0x8fff]
> > [    2.996997] pci 0000:00:1f.3: can't claim BAR 4 [io  0x8200-0x823f]: address conflict with PCI Bus 0000:01 [io  0x8000-0x8fff]
> >
> > Sorry, did not get time to debug this before.
> >
> > Question, why does the kernel set IO granularity to 4k?
> 
> > The "io_granularity = 0x1000" in pci_read_bridge_io() comes from the fact that PCIe r6.0, sec 7.5.1.3.6, says bridges assume the lower 12 address bits of I/O Base registers (the bridge I/O window) are zero.
> 
> I think you have answered my question.   The lower 12 bits of the
> limit register are assumed to be 0xfff. Granularity is derived from
> this, right?
>
> Spec:
> "Thus, the bottom of the defined I/O address range will be aligned
> to a 4 KB boundary and the top of the defined I/O address range will
> be one less than a 4 KB boundary".

Right.  There are few bridges that support 1K granularity instead of
4K; see dev->io_window_1k.

> > -----Original Message-----
> > From: Bjorn Helgaas <helgaas@kernel.org>
> > Sent: Tuesday, December 13, 2022 1:31 PM
> > To: Kallol Biswas [C] <kallol.biswas@nutanix.com>
> > Cc: linux-pci@vger.kernel.org
> > Subject: Re: uefi secureboot vm and IO window overlap
> >
> > On Sat, Dec 10, 2022 at 05:45:50PM +0000, Kallol Biswas [C] wrote:
> > > The part1 of the dmesg:
> > >
> > > [    0.000000] Initializing cgroup subsys cpuset
> > > [    0.000000] Initializing cgroup subsys cpu
> > > [    0.000000] Initializing cgroup subsys cpuacct
> > > [    0.000000] Linux version 3.10.0-957.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Thu Nov 8 23:39:32 UTC 2018
> >
> > Is there any chance you can reproduce the problem on a current kernel?
> > If it's been fixed by now, maybe we could identify the fix and you could backport it?
> >
> > Bjorn