Re: [PATCH v9 00/26] PCI: Allow BAR movement during boot and hotplug

["mika.westerberg@linux.intel.com" <mika.westerberg@linux.intel.com>]

Hi,

On Wed, Feb 10, 2021 at 07:40:06PM +0000, Sergei Miroshnichenko wrote:
> On Thu, 2021-02-04 at 12:49 +0200, Mika Westerberg
> wrote:
> > On Wed, Feb 03, 2021 at 08:17:14PM +0000, Sergei Miroshnichenko
> > wrote:
> > > On Mon, 2021-02-01 at 14:55 +0200, Mika Westerberg wrote:
> > > > On Thu, Jan 28, 2021 at 09:39:29PM +0100, Lukas Wunner wrote:
> > > > > On Thu, Jan 28, 2021 at 08:53:16AM -0600, Bjorn Helgaas wrote:
> > > > > > On Fri, Dec 18, 2020 at 08:39:45PM +0300, Sergei
> > > > > > Miroshnichenko
> > > > > > wrote:
> > > > > > > ...
> > > > > 
> > > > > I intended to review and test this iteration of the series more
> > > > > closely, but haven't been able to carve out the required time.
> > > > > I'm adding some Thunderbolt folks to cc in the hope that they
> > > > > can at least test the series on their development branch.
> > > > > Getting this upstreamed should really be in the best interest
> > > > > of Intel and other promulgators of Thunderbolt.
> > > > 
> > > > Sure. It seems that this series was submitted in December so
> > > > probably
> > > > not applicable to the pci.git/next anymore. Anyways, I can give
> > > > it a
> > > > try
> > > > on a TBT capable system if someone tells me what exactly to test
> > > > ;-)
> > > > Probably at least that the existing functionality still works but
> > > > something else maybe too?
> > > 
> > > For setups that worked fine, the only expected change is a possible
> > > little different BAR layout (in /proc/iomem), and there should the
> > > same
> > > quantity (or more) of BARs assigned than before.
> > > 
> > > But if there are any problematic setups, which weren't able to
> > > arrange
> > > new BARs, this patchset may push a bit further.
> > 
> > Got it.
> > 
> > > In a few days I'll provide an updated branch for our mirror of the
> > > kernel on Github, with a complete and bumped set of patches,
> > > reducing
> > > the steps required to test them.
> > 
> > Sounds good, thanks!
> 
> Hi Mika,
> 
> The branch is finally ready, so if you still have time for that, please
> take a look:
> 
> https://github.com/YADRO-KNS/linux/tree/yadro/pcie_hotplug/movable_bars_v9.1

Thanks for sharing!

I tried this series on top of v5.11-rc7 on a Dell XPS 9380 so that I
have two TBT3 devices connected. Each device includes PCIe switch and a
xHCI endpoint.

What I see that the hotplug downstream port does not have enough bus
numbers (and other resources allocated) so when attaching the second
device it does not fit there anymore. The resulting 'lspci -t' output
looks like below:

-[0000:00]-+-00.0
           +-02.0
           +-04.0
           +-08.0
           +-12.0
           +-14.0
           +-14.2
           +-15.0
           +-15.1
           +-16.0
           +-1c.0-[01]----00.0
           +-1c.6-[02]----00.0
           +-1d.0-[03-3b]----00.0-[04-3b]--+-00.0-[05]----00.0
           |                               +-01.0-[06-1f]----00.0-[07-09]--+-02.0-[08]----00.0
           |                               |                               \-04.0-[09]----00.0-[0a]--
           |                               +-02.0-[20]----00.0
           |                               \-04.0-[21-3b]--
           +-1d.4-[3c]----00.0
           +-1f.0
           +-1f.3
           +-1f.4
           \-1f.5

So the last PCIE switch is not visible anymore, and the xHCI on the
second TBT device is not functional either.

On the mainline kernel I get this:

-[0000:00]-+-00.0
           +-02.0
           +-04.0
           +-08.0
           +-12.0
           +-14.0
           +-14.2
           +-15.0
           +-15.1
           +-16.0
           +-1c.0-[01]----00.0
           +-1c.6-[02]----00.0
           +-1d.0-[03-3b]----00.0-[04-3b]--+-00.0-[05]----00.0
           |                               +-01.0-[06-1f]----00.0-[07-1f]--+-02.0-[08]----00.0
           |                               |                               \-04.0-[09-1f]----00.0-[0a-1f]--+-02.0-[0b]----00.0
           |                               |                                                               \-04.0-[0c-1f]--
           |                               +-02.0-[20]----00.0
           |                               \-04.0-[21-3b]--
           +-1d.4-[3c]----00.0
           +-1f.0
           +-1f.3
           +-1f.4
           \-1f.5

In this topology I can add yet another TBT device and there are still
resources available and all the endpoints are functional too.

I can send you the full dmesg and lspci -vv output if needed.