Re: qemu-xen-dir + PCI passthrough = BOOM

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Wed, Dec 18, 2013 at 02:57:26PM +0000, Ian Campbell wrote:
> On Wed, 2013-12-18 at 14:48 +0000, Anthony PERARD wrote:
> > On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk wrote:
> > > On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
> > > > On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
> > > > [...]
> > > > > > Those Xen report something like:
> > > > > > (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46: 131329 >
> > > > > > 131328
> > > > > > (XEN) memory.c:132:d0 Could not allocate order=0 extent: id=46
> > > > > > memflags=0 (62 of 64)
> > > > > > 
> > > > > > ?
> > > > > > 
> > > > > > (I tryied to reproduce the issue by simply add many emulated e1000 in
> > > > > > QEMU :) )
> > > > > > 
> > 
> > > -bash-4.1# lspci -s 01:00.0 -v 
> > > 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
> > >         Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
> > >         Flags: fast devsel, IRQ 16
> > >         Memory at fbc20000 (32-bit, non-prefetchable) [disabled] [size=128K]
> > >         Memory at fb800000 (32-bit, non-prefetchable) [disabled] [size=4M]
> > >         I/O ports at e020 [disabled] [size=32]
> > >         Memory at fbc44000 (32-bit, non-prefetchable) [disabled] [size=16K]
> > >         Expansion ROM at fb400000 [disabled] [size=4M]
> > 
> > BTW, I think this is the issue, the Expansion ROM. qemu-xen will
> > allocate memory for it. Will have maybe have to find another way.
> > qemu-trad those not seems to allocate memory, but I haven't been very
> > far in trying to check that.
> 
> This might SeaBIOS, since it is the BIOS which runs the expansion roms.
> 
> ROMBIOS didn't run expansion ROMS, instead IIRC you have to extract the
> ones you care about from the device(s) and build them into hvmloader
> (somehow) to be predeployed.

In 'rombios_load_roms' I see:

106     option_rom_phys_addr = etherboot_phys_addr + etherboot_sz;                  
107     option_rom_sz = pci_load_option_roms(OPTIONROM_PHYSICAL_END,                
108                                          option_rom_phys_addr);                 
109                                                                                 
110     printf("Option ROMs:\n");                                                   
111     if ( vgabios_sz )                                                           
112         printf(" %05x-%05x: VGA BIOS\n",                                        
113                VGABIOS_PHYSICAL_ADDRESS,                                        
114                VGABIOS_PHYSICAL_ADDRESS + vgabios_sz - 1);                      
115     if ( etherboot_sz )                                                         
116         printf(" %05x-%05x: Etherboot ROM\n",                                   
117                etherboot_phys_addr,                                             
118                etherboot_phys_addr + etherboot_sz - 1);                         
119     if ( option_rom_sz )                                                        
120         printf(" %05x-%05x: PCI Option ROMs\n",                                 
121                option_rom_phys_addr,                                            
122                option_rom_phys_addr + option_rom_sz - 1);                   


seems to imply that it sets it up. Doing a git log on rombios
shows this:


commit c2901a5cc44c85a9ceeebd47ff21946bb35ba930
Author: Keir Fraser <keir.fraser@citrix.com>
Date:   Tue Mar 31 14:04:50 2009 +0100

    hvmloader: ROMBIOS must POST in big real mode, since it supports PMM,
    and hence Option ROMs can expect noi segment limit checks.

which would imply that the Option ROMs would be loaded and run.

> 
> Could try building a seabios without CONFIG_OPTIONROMS (edit
> tools/firmware/seabios-config) and see if that helps -- although not if
> there is some other essential option rom of course.
> 
> That's not the solution but it would help diagnose the issue.

Looking at hvmloader in 'pci_load_option_roms' I see:

                                                                               
165         /*                                                                      
166          * Currently only scan options from mass storage devices and serial     
167          * bus controller (Fibre Channel included).                             
168          */                                                                     
169         if ( (class != 0x1) && (class != 0xc) )                                 
170             continue;        

Which would imply that the option rom for the NIC is not loaded.


But more interestingly, the hvmloader has this:

136 struct bios_config seabios_config = {                                           
137     .name = "SeaBIOS",                                                          
138                                                                                 
139     .image = seabios,                                                           
140     .image_size = sizeof(seabios),                                              
141                                                                                 
142     .bios_address = 0x100000 - sizeof(seabios),                                 
143                                                                                 
144     .load_roms = NULL,                                                          
145                                                                                 
146     .bios_load = NULL,                                                          
147                                                                                 

load_roms and bios_load are not set - so it wouldn't even do it.
It only does it for Bochs BIOS.
> 
> Ian.
>