Re: [Qemu-devel] [RFC 3/3] acpi-build: allocate mcfg for multiple host bridges

[Alex Williamson <alex.williamson@redhat.com>]

On Wed, 23 May 2018 02:38:52 +0300
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Tue, May 22, 2018 at 03:47:41PM -0600, Alex Williamson wrote:
> > On Wed, 23 May 2018 00:44:22 +0300
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >   
> > > On Tue, May 22, 2018 at 03:36:59PM -0600, Alex Williamson wrote:  
> > > > On Tue, 22 May 2018 23:58:30 +0300
> > > > "Michael S. Tsirkin" <mst@redhat.com> wrote:     
> > > > >
> > > > > It's not hard to think of a use-case where >256 devices
> > > > > are helpful, for example a nested virt scenario where
> > > > > each device is passed on to a different nested guest.
> > > > >
> > > > > But I think the main feature this is needed for is numa modeling.
> > > > > Guests seem to assume a numa node per PCI root, ergo we need more PCI
> > > > > roots.    
> > > > 
> > > > But even if we have NUMA affinity per PCI host bridge, a PCI host
> > > > bridge does not necessarily imply a new PCIe domain.    
> > > 
> > > What are you calling a PCIe domain?  
> > 
> > Domain/segment
> > 
> > 0000:00:00.0
> > ^^^^ This  
> 
> Right. So we can thinkably have PCIe root complexes share an ACPI segment.
> I don't see what this buys us by itself.

The ability to define NUMA locality for a PCI sub-hierarchy while
maintaining compatibility with non-segment aware OSes (and firmware).

> > Isn't that the only reason we'd need a new MCFG section and the reason
> > we're limited to 256 buses?  Thanks,
> > 
> > Alex  
> 
> I don't know whether a single MCFG section can describe multiple roots.
> I think it would be certainly unusual.

I'm not sure here if you're referring to the actual MCFG ACPI table or
the MMCONFIG range, aka the ECAM.  Neither of these describe PCI host
bridges.  The MCFG table can describe one or more ECAM ranges, which
provides the ECAM base address, the PCI segment associated with that
ECAM and the start and end bus numbers to know the offset and extent of
the ECAM range.  PCI host bridges would then theoretically be separate
ACPI objects with _SEG and _BBN methods to associate them to the
correct ECAM range by segment number and base bus number.  So it seems
that tooling exists that an ECAM/MMCONFIG range could be provided per
PCI host bridge, even if they exist within the same domain, but in
practice what I see on systems I have access to is a single MMCONFIG
range supporting all of the host bridges.  It also seems there are
numerous ways to describe the MMCONFIG range and I haven't actually
found an example that seems to use the MCFG table.  Two have MCFG
tables (that don't seem terribly complete) and the kernel claims to
find the MMCONFIG via e820, another doesn't even have an MCFG table and
the kernel claims to find MMCONFIG via an ACPI motherboard resource.
I'm not sure if I can enable PCI segments on anything to see how the
firmware changes.  Thanks,

Alex