Re: KVM call minutes 2013-01-29 - Port I/O

["Michael S. Tsirkin" <mst@redhat.com>]

On Thu, Jan 31, 2013 at 02:21:50PM -0700, Alex Williamson wrote:
> On Thu, 2013-01-31 at 23:11 +0200, Michael S. Tsirkin wrote:
> > On Thu, Jan 31, 2013 at 09:34:03AM -0700, Alex Williamson wrote:
> > > 
> > > On Thu, 2013-01-31 at 12:49 +0200, Michael S. Tsirkin wrote:
> > > > On Wed, Jan 30, 2013 at 04:28:30PM -0700, Alex Williamson wrote:
> > > > > On Thu, 2013-01-31 at 10:02 +1100, Benjamin Herrenschmidt wrote:
> > > > > > On Thu, 2013-01-31 at 00:49 +0200, Michael S. Tsirkin wrote:
> > > > > > > > In practice they do (VGA at least)
> > > > > > > > 
> > > > > > > > >From a SW modelling standpoint, I don't think it's worth
> > > > > > > differentiating
> > > > > > > > PCI and PCIE.
> > > > > > > > 
> > > > > > > > Cheers,
> > > > > > > > Ben.
> > > > > > > 
> > > > > > > Interesting.
> > > > > > > Do you have such hardware? Could you please dump
> > > > > > > the output of lspci -vv?
> > > > > > 
> > > > > > Any ATI or nVidia card still supports hard decoding of VGA regions for
> > > > > > the sake of legacy operating systems and BIOSes :-) I don't know about
> > > > > > Intel but I suppose it's the same.
> > > > > 
> > > > > For example:
> > > > > 
> > > > > -[0000:00]-+-00.0  Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (external gfx0 p
> > > > >            +-04.0-[02]--+-00.0  Advanced Micro Devices [AMD] nee ATI Cedar PRO [Radeon HD 5450/6350]
> > > > > 
> > > > > 00:04.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (PCI express gpp port D) (prog-if 00 [Normal decode])
> > > > > 	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
> > > > > 	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> > > > > 	Latency: 0, Cache Line Size: 64 bytes
> > > > > 	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
> > > > > 	I/O behind bridge: 0000c000-0000cfff
> > > > > 	Memory behind bridge: fd100000-fd1fffff
> > > > > 	Prefetchable memory behind bridge: 00000000d0000000-00000000dfffffff
> > > > > 	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
> > > > > 	BridgeCtl: Parity- SERR- NoISA- VGA+ MAbort- >Reset- FastB2B-
> > > > >                                       ^^^^
> > > > > VGA+ (VGA Enable) indicates positive decode of 0x3b0 - 0x3bb, 0x3c0 -
> > > > > 0x3df, and 0xa0000 - 0xbfff.  Device 2:00.0 of course doesn't report
> > > > > these "ISA" ranges as they're implicit in the VGA class code.
> > > > 
> > > > OK but this appears behind a bridge.  So the bridge configuration tells
> > > > the root complex where to send accesses to the VGA.
> > > > 
> > > > But qemu currently puts devices directly on root bus.
> > > > 
> > > > And as far as I can tell when we present devices directly on bus 0, we
> > > > pretend these are integrated in the root complex. The spec seems to
> > > > say explicitly that root complex integrated devices should not use legacy
> > > > addresses or support hotplug. So I would be surprised if such one
> > > > appears in real world.
> > > > 
> > > > Luckily guests do not seem to be worried as long as we use ACPI.
> > > 
> > > Yes, in fact I just figured out last night that Windows is unhappy with
> > > assigned PCI devices on bus 0 that claim to be an endpoint in their PCIe
> > > capability rather than an integrated endpoint.  We'll need to do extra
> > > mangling of the PCIe capability to massage it into the guest visible
> > > topology.
> > 
> > For now, just put you device behind an express bridge. This breaks acpi
> > hotplug for now, but I'm looking into hotplug with bridges anyway.
> 
> We have the problem in both directions though, Endpoints that should be
> Integrated Endpoints and Integrated Endpoints that should be Endpoints.
> So I think we need to mangle the type.
> 
> > If you really need it I can give you a hack for hotplug too.
> > 
> > Of course express  does not allow hotplug of root complex parts
> > but happens to work because we use ACPI.
> 
> That's a little odd.
> 
> > > Section 1.3.2.3 of the 3.0 spec says integrated endpoints must not
> > > require I/O resources claimed through BAR(s).  VGA skirts around this by
> > > not having the legacy resources claimed by BARs, but instead being
> > > implicit.
> > 
> > Aha. I missed this point.
> > 
> > >  Are there other sections restricting legacy I/O?
> > 
> > One other interesting things is that VGA enable bit (for bridge control
> > register) does not appear in express spec at all.
> 
> Yep, but it appears on hardware.
> 
> > > It's common that a plugin VGA card sits behind a root port where the
> > > bridge registers tell us about VGA routing,
> > > but integrated VGA devices
> > > are often on bus 0 though, here's an example:
> > > 
> > > -[0000:00]-+-00.0  Intel Corporation 2nd Generation Core Processor Family DRAM Controller
> > >            +-02.0  Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller
> > > 
> > > Often these systems will disable the integrated graphics when a plugin
> > > graphics is installed below a root port.  I'm not sure how the system
> > > knows to route VGA to the integrated device vs the root port otherwise.
> > 
> > I am guessing it disables the integrated graphics?
> > 
> > > Here's a more interesting example:
> > > 
> > > -+-[0000:01]-+-00.0  NVIDIA Corporation GT218 [GeForce G210M]
> > >  |           \-00.1  NVIDIA Corporation High Definition Audio Controller
> > >  \-[0000:00]-+-00.0  Intel Corporation Mobile 4 Series Chipset Memory Controller Hub
> > >              +-01.0  Intel Corporation Mobile 4 Series Chipset PCI Express Graphics Port
> > > 
> > > This system seems to have two host bridges with VGA behind each of them.
> > > There's no bridge to control VGA routing, so I don't know how the
> > > selection is done.
> > 
> > Is IO space disabled for the inactive card? Maybe that is how.
> 
> The card has BAR defined I/O space resources.  My guess is that VGA is
> just statically routed to the integrated device and the secondary works
> only in non-legacy mode until the BIOS switch is flipped, the integrated
> device is hidden and VGA is switched to static routing for the nvidia
> device.  I suppose that means I'll never be able to assign the nvidia to
> a guest, at least not with any kind of legacy VGA support.  Thanks,
> 
> Alex

Can you check device control for both before and after the switch.

-- 
MST