PCI passthru issue

PCI passthru issue

[Anirban Chakraborty]

Hi All,

I noticed that the PCI registers are not all mapped to the guest for a PCI pass through device. For example, I was trying to pass through a PCI function (NIC) to a guest and in the guest the MSI-X table offset register is not mapped properly. It was showing all 0s. This is in RHEL 6.0 beta and the guest was a SLES11 64 bit. 
In the RHEL 6.0 host I see following for the PCI function:
---------
05:00.0 Ethernet controller: QLogic Corp. cLOM8214 1/10GbE Controller (rev 54)
	Subsystem: QLogic Corp. 8200 Series Dual Port 10GbE Converged Network Adapter (TCP/IP Networking)
	Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 74
	Region 0: Memory at df200000 (64-bit, non-prefetchable) [size=2M]
	Region 4: Memory at df1b0000 (64-bit, non-prefetchable) [size=64K]
	Expansion ROM at df1c0000 [disabled] [size=256K]
	Capabilities: [40] MSI-X: Enable- Count=32 Masked-
		Vector table: BAR=0 offset=00090000
		PBA: BAR=0 offset=00090800
...
	Kernel driver in use: pci-stub
00: 77 10 20 80 42 05 10 00 54 00 00 02 40 00 80 00
10: 04 00 20 df 00 00 00 00 00 00 00 00 00 00 00 00
20: 04 00 1b df 00 00 00 00 00 00 00 00 77 10 07 02
30: 00 00 1c df 40 00 00 00 00 00 00 00 0a 01 00 00
40: 11 80 1f 00 00 00 09 00 00 08 09 00 00 00 00 00
----------
And in the SLES11 guest, I see following for the same function:
------------
00:07.0 Ethernet controller: QLogic Corp. Device 8020 (rev 54)
	Subsystem: QLogic Corp. Device 0207
	Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at f2200000 (32-bit, non-prefetchable) [size=2M]
	Region 4: Memory at f2400000 (32-bit, non-prefetchable) [size=64K]
	Expansion ROM at f2440000 [disabled] [size=256K]
	Capabilities: [40] Message Signalled Interrupts: Mask- 64bit- Count=1/1 Enable-
		Address: 00000000  Data: 0000
	Capabilities: [50] MSI-X: Enable- Mask- TabSize=32
		Vector table: BAR=0 offset=00090000
		PBA: BAR=0 offset=00000000
00: 77 10 20 80 42 05 10 00 54 00 00 02 40 00 80 00
10: 00 00 20 f2 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 40 f2 00 00 00 00 00 00 00 00 77 10 07 02
30: 08 00 44 f2 40 00 00 00 00 00 00 00 0b 01 00 00
40: 05 50 00 00 00 00 00 00 00 00 00 00 00 00 00 00
------------

Note that the data at byte offset 40h is different for the host and the guest. Is this expected or is it a bug in KVM?

Is there any spec out there that specifies what PCI registers are accessible from guest and how much of the PCI config space is mapped to the guest? This is helpful to know, especially in the context of SR-IOV. If a VF is pass throughed to a guest, are all the VF registers visible in host system will be mapped (and accessible) to the guest OS?

thanks much,
Anirban

Re: PCI passthru issue

[Hidetoshi Seto]

(2010/09/17 8:01), Anirban Chakraborty wrote:
> Hi All,
> 
> I noticed that the PCI registers are not all mapped to the
> guest for a PCI pass through device. For example, I was trying
> to pass through a PCI function (NIC) to a guest and in the
> guest the MSI-X table offset register is not mapped properly.
> It was showing all 0s. This is in RHEL 6.0 beta and the guest
> was a SLES11 64 bit. 
>
> In the RHEL 6.0 host I see following for the PCI function:
> ---------
> 05:00.0 Ethernet controller: QLogic Corp. cLOM8214 1/10GbE Controller (rev 54)
> 	Subsystem: QLogic Corp. 8200 Series Dual Port 10GbE Converged Network Adapter (TCP/IP Networking)
> 	Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
> 	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> 	Interrupt: pin A routed to IRQ 74
> 	Region 0: Memory at df200000 (64-bit, non-prefetchable) [size=2M]
> 	Region 4: Memory at df1b0000 (64-bit, non-prefetchable) [size=64K]
> 	Expansion ROM at df1c0000 [disabled] [size=256K]
> 	Capabilities: [40] MSI-X: Enable- Count=32 Masked-
> 		Vector table: BAR=0 offset=00090000
> 		PBA: BAR=0 offset=00090800
> ...
> 	Kernel driver in use: pci-stub
> 00: 77 10 20 80 42 05 10 00 54 00 00 02 40 00 80 00
> 10: 04 00 20 df 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 04 00 1b df 00 00 00 00 00 00 00 00 77 10 07 02
> 30: 00 00 1c df 40 00 00 00 00 00 00 00 0a 01 00 00
> 40: 11 80 1f 00 00 00 09 00 00 08 09 00 00 00 00 00
> ----------
> And in the SLES11 guest, I see following for the same function:
> ------------
> 00:07.0 Ethernet controller: QLogic Corp. Device 8020 (rev 54)
> 	Subsystem: QLogic Corp. Device 0207
> 	Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
> 	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> 	Interrupt: pin A routed to IRQ 11
> 	Region 0: Memory at f2200000 (32-bit, non-prefetchable) [size=2M]
> 	Region 4: Memory at f2400000 (32-bit, non-prefetchable) [size=64K]
> 	Expansion ROM at f2440000 [disabled] [size=256K]
> 	Capabilities: [40] Message Signalled Interrupts: Mask- 64bit- Count=1/1 Enable-
> 		Address: 00000000  Data: 0000
> 	Capabilities: [50] MSI-X: Enable- Mask- TabSize=32
> 		Vector table: BAR=0 offset=00090000
> 		PBA: BAR=0 offset=00000000
> 00: 77 10 20 80 42 05 10 00 54 00 00 02 40 00 80 00
> 10: 00 00 20 f2 00 00 00 00 00 00 00 00 00 00 00 00
> 20: 00 00 40 f2 00 00 00 00 00 00 00 00 77 10 07 02
> 30: 08 00 44 f2 40 00 00 00 00 00 00 00 0b 01 00 00
> 40: 05 50 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> ------------
> 
> Note that the data at byte offset 40h is different for the
> host and the guest. Is this expected or is it a bug in KVM?

This is expected, since MSIs of PCI pass through device need
some emulation to route them to the guest instead of the host.

So what we can see on the guest's PCI pass through device is
modified capability list that contains emulated capabilities.

> 
> Is there any spec out there that specifies what PCI registers
> are accessible from guest and how much of the PCI config space
> is mapped to the guest? This is helpful to know, especially
> in the context of SR-IOV. If a VF is pass throughed to a guest,
> are all the VF registers visible in host system will be mapped
> (and accessible) to the guest OS?

AFAIK the QEMU source code, hw/device-assignment.c, will help
you to know what are going on there.

I'm not sure what interesting registers can the VF have.
But I think you should be careful because the register in the
PCI config space might not work like that on the host even the
register is mapped and visible from the guest.


Thanks,
H.Seto