Re: Graphics pass-through

[Jan Kiszka <jan.kiszka@siemens.com>]

On 2011-05-09 17:48, Alex Williamson wrote:
> On Mon, 2011-05-09 at 17:27 +0200, Jan Kiszka wrote:
>> On 2011-05-09 16:55, Prasad Joshi wrote:
>>> On Mon, May 9, 2011 at 12:14 PM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>>> On 2011-05-05 17:17, Alex Williamson wrote:
>>>>>> And what about the host? When does Linux release the legacy range?
>>>>>> Always or only when a specific (!=vga/vesa) framebuffer driver is loaded?
>>>>>
>>>>> Well, that's where it'd be nice if the vga arbiter was actually in more
>>>>> widespread use.  It currently seems to be nothing more than a shared
>>>>> mutex, but it would actually be useful if it included backends to do the
>>>>> chipset vga routing changes.  I think when I was testing this, I was
>>>>> externally poking PCI bridge chipset to toggle the VGA_EN bit.
>>>>
>>>> Right, we had to drop the approach to pass through the secondary card
>>>> for now, the arbiter was not switching properly. Haven't checked yet if
>>>> VGA_EN was properly set, though the kernel code looks like it should
>>>> take care of this.
>>>>
>>>> Even with handing out the primary adapter, we had only mixed success so
>>>> far. The onboard adapter worked well (in VESA mode), but the NVIDIA is
>>>> not displaying early boot messages at all. Maybe a vgabios issue.
>>>> Windows was booting nevertheless - until we installed the NVIDIA
>>>> drivers. Than it ran into a blue screen.
>>>>
>>>> BTW, what ATI adapter did you use precisely, and what did work, what not?
>>>
>>> Not hijacking the mail thread. Just wanted to provide some inputs.
>>
>> Much appreciated in fact!
>>
>>>
>>> Few days back I had tried passing through the secondary graphics card.
>>> I could pass-through two graphics cards to virtual machine.
>>>
>>> 02:00.0 VGA compatible controller: ATI Technologies Inc Redwood
>>> [Radeon HD 5670] (prog-if 00 [VGA controller])
>>> 	Subsystem: PC Partner Limited Device e151
>>> 	Flags: bus master, fast devsel, latency 0, IRQ 87
>>> 	Memory at d0000000 (64-bit, prefetchable) [size=256M]
>>> 	Memory at fe6e0000 (64-bit, non-prefetchable) [size=128K]
>>> 	I/O ports at b000 [size=256]
>>> 	Expansion ROM at fe6c0000 [disabled] [size=128K]
>>> 	Capabilities: <access denied>
>>> 	Kernel driver in use: radeon
>>> 	Kernel modules: radeon
>>>
>>> 07:00.0 VGA compatible controller: nVidia Corporation G86 [Quadro NVS
>>> 290] (rev a1) (prog-if 00 [VGA controller])
>>>        Subsystem: nVidia Corporation Device 0492
>>>        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>> ParErr-Stepping- SERR+ FastB2B- DisINTx-
>>>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast
>>>> TAbort-<TAbort- <MAbort- >SERR- <PERR- INTx-
>>>        Latency: 0, Cache Line Size: 64 bytes
>>>        Interrupt: pin A routed to IRQ 24
>>>        Region 0: Memory at fd000000 (32-bit, non-prefetchable) [size=16M]
>>>        Region 1: Memory at d0000000 (64-bit, prefetchable) [size=256M]
>>>        Region 3: Memory at fa000000 (64-bit, non-prefetchable) [size=32M]
>>>        Region 5: I/O ports at ec00 [size=128]
>>>        Expansion ROM at fe9e0000 [disabled] [size=128K]
>>>        Capabilities: <access denied>
>>>        Kernel driver in use: nouveau
>>>        Kernel modules: nouveau, nvidiafb
>>>
>>> Both of them are PCIe cards. I have one more ATI card and another
>>> NVIDIA card which does not work.
>>
>> Interesting. That may rule out missing PCIe capabilities as source for
>> the NVIDIA driver indisposition.
>>
>> Did you passed those cards each as primary to the guest, or was the
>> guest seeing multiple adapters? I presume you only got output after
>> early boot was completed, right?
>>
>> To avoid having to deal with legacy I/O forwarding, we started with a
>> dual adapter setup in the hope to leave the primary guest adapter at
>> know-to-work cirrus-vga. But already in a native setup with on-board
>> primary + NVIDIA secondary, the NVIDIA Windows drivers refused to talk
>> to its hardware in this constellation.
>>
>>>
>>> One of the reason the pass-through did not work is because of the
>>> limit on amount of pci configuration memory by SeaBIOS. SeaBIOS places
>>> a hard limit of 256MB or so on the amount of PCI memory space. Thus,
>>> for some of the VGA device that need more memory never worked for me.
>>>
>>> SeaBIOS allows this memory region to be extended to some value near
>>> 512MB, but even then the range is not enough.
>>>
>>> Another problem with SeaBIOS which limits the amount of memory space
>>> is: SeaBIOS allocates the BAR regions as they are encountered. As far
>>> as I know, the BAR regions should be naturally aligned. Thus the
>>> simple strategy of the SeaBIOS results in large fragmentation.
>>> Therefore, even after increasing the PCI memory space to 512MB the BAR
>>> regions were unallocated.
>>
>> That's an interesting trace! We'll check this here, but I bet it
>> contributes to the problems. Our FX 3800 has 1G memory...
> 
> Yes, qemu leaves far too little MMIO space to think about assigning
> graphics cards.  Both of my cards have 512MB and I hacked qemu to leave
> a bigger gap via something like:
> 
> diff --git a/hw/pc.c b/hw/pc.c
> index 0ea6d10..a6376f8 100644
> --- a/hw/pc.c
> +++ b/hw/pc.c
> @@ -879,6 +879,8 @@ void pc_cpus_init(const char *cpu_model)
>      }
>  }
>  
> +#define PC_MAX_LOW_RAM 0xc0000000
> +
>  void pc_memory_init(ram_addr_t ram_size,
>                      const char *kernel_filename,
>                      const char *kernel_cmdline,
> @@ -893,9 +895,9 @@ void pc_memory_init(ram_addr_t ram_size,
>      int bios_size, isa_bios_size;
>      void *fw_cfg;
>  
> -    if (ram_size >= 0xe0000000 ) {
> -        above_4g_mem_size = ram_size - 0xe0000000;
> -        below_4g_mem_size = 0xe0000000;
> +    if (ram_size >= PC_MAX_LOW_RAM ) {
> +        above_4g_mem_size = ram_size - PC_MAX_LOW_RAM;
> +        below_4g_mem_size = PC_MAX_LOW_RAM;
>      } else {
>          below_4g_mem_size = ram_size;
>      }
> 
> There's also a #define that needs to be changed in seabios config.h and
> and acpi dsdt update, but I can't seem to find patches for those.

Hmm, as this does not scale with the constantly growing memory sizes of
GPUs, I guess this should would for us as well, even with 1G. The
adapters likely only map a window to their on-board RAM.

>  Also
> pay attention to the cpu_register_physical_memory calls in
> i440fx_update_memory_mappings(), those can steal the legacy VGA MMIO
> range from you.  I just commented them out:
> 
> diff --git a/hw/piix_pci.c b/hw/piix_pci.c
> index b5589b9..1327563 100644
> --- a/hw/piix_pci.c
> +++ b/hw/piix_pci.c
> @@ -106,11 +106,11 @@ static void i440fx_update_memory_mappings(PCII440FXState *d)
>      }
>      smram = d->dev.config[I440FX_SMRAM];
>      if ((d->smm_enabled && (smram & 0x08)) || (smram & 0x40)) {
> -        cpu_register_physical_memory(0xa0000, 0x20000, 0xa0000);
> +        //cpu_register_physical_memory(0xa0000, 0x20000, 0xa0000);
>      } else {
>          for(addr = 0xa0000; addr < 0xc0000; addr += 4096) {
> -            cpu_register_physical_memory(addr, 4096,
> -                                         d->isa_page_descs[(addr - 0xa0000) >> 12]);
> +            //cpu_register_physical_memory(addr, 4096,
> +            //                             d->isa_page_descs[(addr - 0xa0000) >> 12]);
>          }
>      }
>  }
> 
> That's all the tricks I remember.  Thanks,

Yeah, we are already carrying half of the above in our tree (only the
second disabling is actually needed, KVM does not support SMM). I
started looking into fixing PAM/SMRAM mess, but it's not yet beautiful -
partly because we urgently need slot management at core level.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux