Bad ROM contents on pass-through card

Bad ROM contents on pass-through card

[Gordon Messmer]

I started trying to pass through a PCI-E card to a KVM guest early this 
year.  Other hardware can be passed through successfully, but the guest 
sees the wrong option ROM for this card.

My system is an AMD CPU and GA-970A-D3 motherboard.  There are currently 
two Radeon HD 3470 cards installed, partially so that I could read their 
ROMs into a file.

The host OS is CentOS 6.3, and I've updated seabios and qemu to current 
versions.

My guest is started with:

/usr/local/bin/qemu-system-x86_64 ... -vga cirrus \
-device pci-assign,host=05:00.0,id=hostdev1,configfd=27,\
bus=pci.0,addr=0x5,romfile=/var/lib/libvirt/images/Radeon3470-5.rom

However, when the guest tries to read the ROM for the ATI card, it's 
seeing the ROM for the emulated cirrus card instead:

# strings /sys/devices/pci0000:00/0000:00:05.0/rom | head -4
Plex86/Bochs VGABios (PCI)
current-cvs 23 Sep 2012
(C) 2008 the LGPL VGABios developers Team

If I don't pass an emulated VGA device (and modify seabios to not run 
the VGA ROM), the guess won't read anything from the rom at all.

Can anyone point me at which parts of the code I might review to see 
where things are getting mapped and what's going wrong?



On the host system, lspci describes the device:

05:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI 
RV620 PRO [Radeon HD 3470] (prog-if 00 [VGA controller])
	Subsystem: Dell Device 3243
	Flags: fast devsel, IRQ 26
	Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Memory at fd9f0000 (64-bit, non-prefetchable) [size=64K]
	I/O ports at ae00 [size=256]
	Expansion ROM at fd900000 [disabled] [size=128K]
	Capabilities: [50] Power Management version 3
	Capabilities: [58] Express Legacy Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information <?>
	Kernel driver in use: pci-stub
	Kernel modules: radeon

and in the guest (Fedora 17):

00:05.0 VGA compatible controller: ATI Technologies Inc Mobility Radeon 
HD 3470 (prog-if 00 [VGA controller])
	Subsystem: Dell Device 3243
	Physical Slot: 5
	Flags: fast devsel, IRQ 11
	Memory at e0000000 (32-bit, non-prefetchable) [size=256M]
	Memory at f2030000 (32-bit, non-prefetchable) [size=64K]
	I/O ports at c100 [size=256]
	Expansion ROM at f2040000 [disabled] [size=64K]
	Capabilities: [58] Express Legacy Endpoint, MSI 00
	Capabilities: [50] Power Management version 3
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit-

Re: Bad ROM contents on pass-through card

[Alex Williamson]

On Sat, 2012-10-06 at 18:23 -0700, Gordon Messmer wrote:
> I started trying to pass through a PCI-E card to a KVM guest early this 
> year.  Other hardware can be passed through successfully, but the guest 
> sees the wrong option ROM for this card.
> 
> My system is an AMD CPU and GA-970A-D3 motherboard.  There are currently 
> two Radeon HD 3470 cards installed, partially so that I could read their 
> ROMs into a file.
> 
> The host OS is CentOS 6.3, and I've updated seabios and qemu to current 
> versions.
> 
> My guest is started with:
> 
> /usr/local/bin/qemu-system-x86_64 ... -vga cirrus \
> -device pci-assign,host=05:00.0,id=hostdev1,configfd=27,\
> bus=pci.0,addr=0x5,romfile=/var/lib/libvirt/images/Radeon3470-5.rom
> 
> However, when the guest tries to read the ROM for the ATI card, it's 
> seeing the ROM for the emulated cirrus card instead:
> 
> # strings /sys/devices/pci0000:00/0000:00:05.0/rom | head -4
> Plex86/Bochs VGABios (PCI)
> current-cvs 23 Sep 2012
> (C) 2008 the LGPL VGABios developers Team
> 
> If I don't pass an emulated VGA device (and modify seabios to not run 
> the VGA ROM), the guess won't read anything from the rom at all.
> 
> Can anyone point me at which parts of the code I might review to see 
> where things are getting mapped and what's going wrong?

The comments in linux:/drivers/pci/rom.c:pci_map_rom are probably
relevant.  Linux decides based on the VGA enable bit whether to return
the PCI option ROM or legacy VGA ROM.  In your case, it's giving you the
legacy ROM from the cirrus device.  The next step is probably looking at
generating a PCI topology in the guest similar to a physical system when
multiple VGA devices are present.  That means bridges and/or root ports
with VGA routing control.

> 
> On the host system, lspci describes the device:
> 
> 05:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI 
> RV620 PRO [Radeon HD 3470] (prog-if 00 [VGA controller])
> 	Subsystem: Dell Device 3243
> 	Flags: fast devsel, IRQ 26
> 	Memory at c0000000 (64-bit, prefetchable) [size=256M]
> 	Memory at fd9f0000 (64-bit, non-prefetchable) [size=64K]
> 	I/O ports at ae00 [size=256]
> 	Expansion ROM at fd900000 [disabled] [size=128K]

> and in the guest (Fedora 17):
> 
> 00:05.0 VGA compatible controller: ATI Technologies Inc Mobility Radeon 
> HD 3470 (prog-if 00 [VGA controller])
> 	Subsystem: Dell Device 3243
> 	Physical Slot: 5
> 	Flags: fast devsel, IRQ 11
> 	Memory at e0000000 (32-bit, non-prefetchable) [size=256M]
> 	Memory at f2030000 (32-bit, non-prefetchable) [size=64K]
> 	I/O ports at c100 [size=256]
> 	Expansion ROM at f2040000 [disabled] [size=64K]

What's the size of Radeon3470-5.rom?  When using romfile the size of the
option ROM BAR is made to be big enough to contain the ROM.  Without
that, we match the physical device and expose the BAR as potentially
being bigger than the ROM it contains.  Thanks,

Alex

Re: Bad ROM contents on pass-through card

[Gordon Messmer]

On 10/07/2012 06:43 AM, Alex Williamson wrote:
> On Sat, 2012-10-06 at 18:23 -0700, Gordon Messmer wrote:
>>
>> If I don't pass an emulated VGA device (and modify seabios to not run
>> the VGA ROM), the guess won't read anything from the rom at all.
>>
>> Can anyone point me at which parts of the code I might review to see
>> where things are getting mapped and what's going wrong?
>
> The comments in linux:/drivers/pci/rom.c:pci_map_rom are probably
> relevant.  Linux decides based on the VGA enable bit whether to return
> the PCI option ROM or legacy VGA ROM.  In your case, it's giving you the
> legacy ROM from the cirrus device.

Interesting.  There are two things that seem odd about that.  First, the 
Linux guest doesn't seem to read the correct ROM either.  During boot, 
dmesg prints:
[    1.047559] [drm] radeon defaulting to kernel modesetting.
[    1.047561] [drm] radeon kernel modesetting enabled.
[    1.047595] checking generic (f0000000 160000) vs hw (e0000000 10000000)
[    1.047832] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11
[    1.048270] [drm] initializing kernel modesetting (RV620 
0x1002:0x95C0 0x1028:0x3243).
[    1.048288] [drm] register mmio base: 0xF2030000
[    1.048289] [drm] register mmio size: 65536
[    1.048395] radeon 0000:00:05.0: Expecting atombios for R600 GPU
[    1.048924] radeon 0000:00:05.0: Fatal error during GPU init
[    1.048925] [drm] radeon: finishing device.
[    1.048927] [TTM] Memory type 2 has not been initialized

Second, if I don't provide a -vga argument, the guest will hang during 
boot.  If I modify seabios not to run the VGA ROM, the guest boots but 
the radeon driver output is the same and I can't read any ROM at all 
from /sys

> The next step is probably looking at
> generating a PCI topology in the guest similar to a physical system when
> multiple VGA devices are present.  That means bridges and/or root ports
> with VGA routing control.

Can you point me toward the documentation?  I don't see anything that 
looks relevant in the man page.

> What's the size of Radeon3470-5.rom?  When using romfile the size of the
> option ROM BAR is made to be big enough to contain the ROM.  Without
> that, we match the physical device and expose the BAR as potentially
> being bigger than the ROM it contains.  Thanks,

It's 64512 bytes.

Thanks, Alex.  I really appreciate your time.