Re: [Qemu-devel] Fwd: [RFC PATCH 0/2] Allow RedHat PCI bridges reserve more buses than necessary during init

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Wed, Jul 19, 2017 at 09:38:50PM +0300, Alexander Bezzubikov wrote:
> 2017-07-19 21:18 GMT+03:00 Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>:
> 
> > On Wed, Jul 19, 2017 at 05:14:41PM +0000, Alexander Bezzubikov wrote:
> > > ср, 19 июля 2017 г. в 16:57, Konrad Rzeszutek Wilk <
> > konrad.wilk@oracle.com>:
> > >
> > > > On Wed, Jul 19, 2017 at 04:20:12PM +0300, Aleksandr Bezzubikov wrote:
> > > > > Now PCI bridges (and PCIE root port too) get a bus range number in
> > > > system init,
> > > > > basing on currently plugged devices. That's why when one wants to
> > > > hotplug another bridge,
> > > > > it needs his child bus, which the parent is unable to provide.
> > > >
> > > > Could you explain how you trigger this?
> > >
> > >
> > > I'm trying to hot plug pcie-pci bridge into pcie root port, and Linux
> > says
> > > 'cannot allocate bus number for device bla-bla'. This obviously does not
> > > allow me to use the bridge at all.
> > >
> > > >
> > > >
> > > > > The suggested workaround is to have vendor-specific capability in
> > RedHat
> > > > generic pcie-root-port
> > > > > that contains number of additional bus to reserve on BIOS PCI init.
> > > >
> > > > But wouldn't the proper fix be for the PCI bridge to have the
> > subordinate
> > > > value be extended to fit more bus ranges?
> > >
> > >
> > > What do you mean? This is what I'm trying to do. Do you suppose to get
> > rid
> > > of vendor-specific cap and use original register value instead of it?
> >
> > I would suggest a simple fix - each bridge has a a number of bus devices
> > it can use. You have up to 255 - so you split the number of northbridge
> > numbers by the amount of NUMA nodes (if that is used) - so for example
> > if you have 4 NUMA nodes, each bridge would cover 63 bus numbers.
> >
> > Meaning the root bridge would cover 0->63 bus, 64->128, and so on.
> > That gives you enough space to plug in your plugged in devices
> > (up to 63).
> >
> > And if you need sub-briges then carve out a specific range.
> >
> 
> The problem is that we don't know at the init moment how many subbridges we
> may need,

                                                                                                          
And the explanation above does not either. It just setups at init time                                                              
an range where you can plug in your new devices in. But in a more uniform                                                           
way such that you can also utilize this with NUMA and _PXM topology                                                                 
in the future.               

> and how deep the whole device tree will be. The key moment - PCI bridge
> hotplugging
> needs either rescan all buses on each bridge device addition, or reserve
> space in advance during BIOS init.

can all buses on each bridge device addition, or reserve                                                          
                                                                                                                                    
It is more complex than that - you may need to move devices that are                                                                
below you. And Linux kernel (nor any other OS) can handle that.                                                                     
(They can during bootup)   

> In this series the second way was chosen.
> 
> 
> >
> >
> > >
> > > >
> > > > >
> > > > > Aleksandr Bezzubikov (2):
> > > > >   pci: add support for direct usage of bdf for capability lookup
> > > > >   pci: enable RedHat pci bridges to reserve more buses
> > > > >
> > > > >  src/fw/pciinit.c   | 12 ++++++++++--
> > > > >  src/hw/pcidevice.c | 24 ++++++++++++++++++++++++
> > > > >  src/hw/pcidevice.h |  1 +
> > > > >  3 files changed, 35 insertions(+), 2 deletions(-)
> > > > >
> > > > > --
> > > > > 2.7.4
> > > > >
> > > > >
> > > >
> > > --
> > > Alexander Bezzubikov
> >
> 
> 
> 
> -- 
> Alexander Bezzubikov