Re: KVM call minutes 2013-01-29 - Port I/O

["Michael S. Tsirkin" <mst@redhat.com>]

On Thu, Jan 31, 2013 at 09:34:03AM -0700, Alex Williamson wrote:
> 
> On Thu, 2013-01-31 at 12:49 +0200, Michael S. Tsirkin wrote:
> > On Wed, Jan 30, 2013 at 04:28:30PM -0700, Alex Williamson wrote:
> > > On Thu, 2013-01-31 at 10:02 +1100, Benjamin Herrenschmidt wrote:
> > > > On Thu, 2013-01-31 at 00:49 +0200, Michael S. Tsirkin wrote:
> > > > > > In practice they do (VGA at least)
> > > > > > 
> > > > > > >From a SW modelling standpoint, I don't think it's worth
> > > > > differentiating
> > > > > > PCI and PCIE.
> > > > > > 
> > > > > > Cheers,
> > > > > > Ben.
> > > > > 
> > > > > Interesting.
> > > > > Do you have such hardware? Could you please dump
> > > > > the output of lspci -vv?
> > > > 
> > > > Any ATI or nVidia card still supports hard decoding of VGA regions for
> > > > the sake of legacy operating systems and BIOSes :-) I don't know about
> > > > Intel but I suppose it's the same.
> > > 
> > > For example:
> > > 
> > > -[0000:00]-+-00.0  Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (external gfx0 p
> > >            +-04.0-[02]--+-00.0  Advanced Micro Devices [AMD] nee ATI Cedar PRO [Radeon HD 5450/6350]
> > > 
> > > 00:04.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (PCI express gpp port D) (prog-if 00 [Normal decode])
> > > 	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
> > > 	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> > > 	Latency: 0, Cache Line Size: 64 bytes
> > > 	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
> > > 	I/O behind bridge: 0000c000-0000cfff
> > > 	Memory behind bridge: fd100000-fd1fffff
> > > 	Prefetchable memory behind bridge: 00000000d0000000-00000000dfffffff
> > > 	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
> > > 	BridgeCtl: Parity- SERR- NoISA- VGA+ MAbort- >Reset- FastB2B-
> > >                                       ^^^^
> > > VGA+ (VGA Enable) indicates positive decode of 0x3b0 - 0x3bb, 0x3c0 -
> > > 0x3df, and 0xa0000 - 0xbfff.  Device 2:00.0 of course doesn't report
> > > these "ISA" ranges as they're implicit in the VGA class code.
> > 
> > OK but this appears behind a bridge.  So the bridge configuration tells
> > the root complex where to send accesses to the VGA.
> > 
> > But qemu currently puts devices directly on root bus.
> > 
> > And as far as I can tell when we present devices directly on bus 0, we
> > pretend these are integrated in the root complex. The spec seems to
> > say explicitly that root complex integrated devices should not use legacy
> > addresses or support hotplug. So I would be surprised if such one
> > appears in real world.
> > 
> > Luckily guests do not seem to be worried as long as we use ACPI.
> 
> Yes, in fact I just figured out last night that Windows is unhappy with
> assigned PCI devices on bus 0 that claim to be an endpoint in their PCIe
> capability rather than an integrated endpoint.  We'll need to do extra
> mangling of the PCIe capability to massage it into the guest visible
> topology.

For now, just put you device behind an express bridge. This breaks acpi
hotplug for now, but I'm looking into hotplug with bridges anyway.

If you really need it I can give you a hack for hotplug too.

Of course express  does not allow hotplug of root complex parts
but happens to work because we use ACPI.

> Section 1.3.2.3 of the 3.0 spec says integrated endpoints must not
> require I/O resources claimed through BAR(s).  VGA skirts around this by
> not having the legacy resources claimed by BARs, but instead being
> implicit.

Aha. I missed this point.

>  Are there other sections restricting legacy I/O?

One other interesting things is that VGA enable bit (for bridge control
register) does not appear in express spec at all.

> It's common that a plugin VGA card sits behind a root port where the
> bridge registers tell us about VGA routing,
> but integrated VGA devices
> are often on bus 0 though, here's an example:
> 
> -[0000:00]-+-00.0  Intel Corporation 2nd Generation Core Processor Family DRAM Controller
>            +-02.0  Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller
> 
> Often these systems will disable the integrated graphics when a plugin
> graphics is installed below a root port.  I'm not sure how the system
> knows to route VGA to the integrated device vs the root port otherwise.

I am guessing it disables the integrated graphics?

> Here's a more interesting example:
> 
> -+-[0000:01]-+-00.0  NVIDIA Corporation GT218 [GeForce G210M]
>  |           \-00.1  NVIDIA Corporation High Definition Audio Controller
>  \-[0000:00]-+-00.0  Intel Corporation Mobile 4 Series Chipset Memory Controller Hub
>              +-01.0  Intel Corporation Mobile 4 Series Chipset PCI Express Graphics Port
> 
> This system seems to have two host bridges with VGA behind each of them.
> There's no bridge to control VGA routing, so I don't know how the
> selection is done.

Is IO space disabled for the inactive card? Maybe that is how.

>  It's possible the g210m never sees legacy VGA
> accesses in this mode.  This bios has another mode which makes the g210m
> the primary graphics and hides the integrated graphics, essentially the
> same as I mention above with hiding integrated endpoint graphics when
> plugin graphics are used.  Thanks,
> 
> Alex