Re: qemu-xen-dir + PCI passthrough = BOOM

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Fri, Jan 10, 2014 at 06:38:18PM +0100, Sander Eikelenboom wrote:
> 
> Friday, January 10, 2014, 5:35:44 PM, you wrote:
> 
> >> >> (d1) [2014-01-10 03:20:29] Running option rom at ca00:0003
> >> >> 
> >> >> (d1) [2014-01-10 03:20:47] Booting from DVD/CD...
> >> >> (d1) [2014-01-10 03:20:47] Booting from 0000:7c00
> >> >> ..
> >> >> and I did see the PXE boot menu in my guest - so even
> >> >> better!
> >> 
> >> > Perfect, look like it is the fix for PCI passthrough.
> >> 
> >> Hi Konrad,
> >> 
> >> Are you sure it's the rom of the NIC, and not the iPXE rom from the emulated device that
> >> gets run ?
> 
> > Yes. I double checked that the MAC address that was given an DHCP
> > address was indeed for the physical hardware. And it was.
> 
> OK
> 
> >> 
> >> With this patch and VGA devices it still points to another rom for me.
> >> (it looks like it is pointing to the rom of the device with a BDF just one lower than the passed through one)
> 
> > That all sounds to me like a bug in QEMU which constructs
> > the 'world'. Then 'hvmloader' and the kernel ingest this to
> > create their view of what the PCI configuration/slots/etc should look like.
> > They use 'inb' and 'outb' instructions on the 0xcf8/0xcfc port.
> 
> > Perhaps the 'vga=none' is having an hard time dealing with 'no-VGA-but-wait-there
> > is-and-PT-VGA'?
> 
> > Just to make sure I am not forgetting a crucial fact - if you don't
> > have vga=none, does the lspci output look sane?
> 
> It does .. except when using an PV NIC, that one doesn't show in lspci, but it does occupy a slot (the numbers are not
> consecutive anymore, one is "hidden"), but the PV nic itself is working ok.
Right, because your Intel emulated gets 'unplugged' (disappears)
when the Xen PV one kicks in.

> This doesn't seem to happen for disk so that seems a bit strange to me ..

That might be due to the CD-ROM option. The disk should
nonetless disappear (/dev/hda).

> 
> Apart from that it looks sane.
> 
> But I'm rebuilding everything now from scratch and will try again .. just too damned many parameters :-)
> 
> Will see if i can make a complete post again with all data, although the previous time i tried that,
> it was a little bit to intimidating i guess.
> 
> >> 
> >> Do you by any chance know if there is a difference in how lspci and the linux kernel scan / list pci devices ?
> >> (for example one by reading acpi tables .. the other one by real probing .. ?)
> 
> > The kernel and hvmloader all use the 'inb' and 'outb' to figure out
> > what the PCI space looks like.
> 
> > 'lspci' uses /sysfs. Thought if you use '-xxxx' I think it also does
> > 'inb' and 'outb'.
> 
> Hmm ok .. so i would expect that dumping with the:
> echo 1 > rom; cat rom > rom.bin; echo 0 > rom;
> sequence in /sys/bus/pci/devices/<BDF> would do everything according to the addresses that lspci gives for the rombar ..
> and when i do that the resulting rom.bin differs according to the emulated devices i put it.
> 
> The devices that have a rom in the guest are always in the order (with there BDF):
> NIC
> Emulated VGA
> Passthroughed VGA
> 
> When all enabled, i end up with the rom of the emulated VGA
> When i disable that with vga="none", i end up with the rom of the NIC
> When i also disable that .. by not specifying any vif and using xen_platform_pci=0 ..
>      the kernel complains when trying to dump that it's not a valid rom (and when forced too it's all zero's so it's
>      right because it can't find the start signature of a rom)
> 

Ok,so the passthrough ROM is definitly not showing up. Which is
odd, b/c it does show up for the physical NIC.

If you do the 'echo 1' .. rune in dom0, can you extract your
VGA (Radeon) BIOS? Perhaps the ROM extraction part for video
cards is different?

> Is there an easy tool to dump the content of arbitrary mem addresses ?
> (except enabling the /dev/kmem and trying to do some voodoo calculations and use "dd" ?)

That is the easiest.
> 
> Hmm when i make a core dump using "xl dump-core" would/should that also contain the content of the passed throughed rom ?

I think not. As the ROM is not RAM it should not include that.

> Then at least i could search that dump for the strings of the real vga rom of the passedtroughed card  and see if it is there .. perhaps on some other
> address as expected .. or at least know it isn't.
> 
> Because at the moment i'm not very well in progressing with ruling things out (which seems to be the only method),
> i only have some symptoms that differ. Wish i had another NIC device with a rom to see if that does work.
> 
> 
> > So it all seems to point to QEMU.
> >> 
> >> --
> >> Sander
> >> 
> >> 
> >> >> I have not yet done the GPU - this issue was preventing me from using
> >> >> qemu-xen as it would always blow up before SeaBIOS was in the picture.
> >> >> 
> >> >> If you would like to put 'Reported-and-Tested-by: Konrad Rzeszutek Wilk
> >> >> <konrad.wilk@oracle.com>' please do.
> >> 
> >> > Will do.
> >> 
> >> >> Thank you!
> >> >> > 
> >> >> > 
> >> >> > diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c
> >> >> > index 6dd7a68..2bbdb6d 100644
> >> >> > --- a/hw/xen/xen_pt.c
> >> >> > +++ b/hw/xen/xen_pt.c
> >> >> > @@ -440,8 +440,8 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s)
> >> >> >  
> >> >> >          s->bases[PCI_ROM_SLOT].access.maddr = d->rom.base_addr;
> >> >> >  
> >> >> > -        memory_region_init_rom_device(&s->rom, OBJECT(s), NULL, NULL,
> >> >> > -                                      "xen-pci-pt-rom", d->rom.size);
> >> >> > +        memory_region_init_io(&s->rom, OBJECT(s), &ops, &s->dev,
> >> >> > +                              "xen-pci-pt-rom", d->rom.size);
> >> >> >          pci_register_bar(&s->dev, PCI_ROM_SLOT, PCI_BASE_ADDRESS_MEM_PREFETCH,
> >> >> >                           &s->rom);
> >> >> >  
> >> >> > 
> >> >> > -- 
> >> >> > Anthony PERARD
> >> 
> >> 
> 
>