Re: [Qemu-devel] [RFC] map 64-bit PCI devices after all possible RAM

[Igor Mammedov <imammedo@redhat.com>]

On Wed, 09 Oct 2013 15:12:08 +0200
Gerd Hoffmann <kraxel@redhat.com> wrote:

> On Mi, 2013-10-09 at 14:23 +0200, Igor Mammedov wrote:
> > I'm posting it to get an oppinion on one of possible approaches
> > on where to map a hotplug memory.
> > 
> > This patch assumes that a space for hotplug memory is located right
> > after RamSizeOver4G region and QEMU will provide romfile to specify
> > where it ends so that BIOS could know from what base to start
> > 64-bit PCI devices mapping.
> 
> We should think about both pci hotplug and memory hotplug while being at
> it.
> 
> Today the 64bit pci window is mapped right above high memory and is
> sized (in acpi tables) according to what is needed to map the devices
> present at boot.
> 
> Effect is that there is no extra address space for 64bit bars of
> hotplugged pci devices.  And the window is also in the way when it comes
> to memory hotplug.
> 
> Given that some windows versions don't like the large 64bit windows we
> should make the window size configurable.
So far from QEMU side it's partially (only memory region mapping and not ACPI
window) configurable via {i440FX-pcihost|q35-pcihost}.pci-hole64-size property

> 
> The window location can either be made configurable too, or we simply
> place it at the top of the address space, with "address space" being
> what the cpu can address according to cpuinfo.
An earlier attempt by Michael to push complete PCI window placement info
via "etc/pci-info" romfile to Seabios was rejected in favor of letting Seabios
to program windows at hardcoded(32-bit/behind high mem) locations with a
64-bit window size (in ACPI) that covers all present devices but doesn't
account for future PCI hotplug either.

That behavior maintained in his "ACPI in QEMU" series, see:
http://patchwork.ozlabs.org/patch/281032/
acpi_get_pci_info()->i440fx_pcihost_get_pci_hole64_end()->pci_bus_get_w64_range()
which is then embedded in ACPI table. So end result stays the same as
before (no usable 64-bit PCI window for hotlug).

But 64-bit PCI window size, which is capped by QEMU to insane legacy 62 bits
(memory region size), is a bit of orthogonal to freeing space for memory
hotplug before it.

> 
> Current qemu reports this by default:
> 
>   $ cat /proc/cpuinfo 
>   model name	: QEMU Virtual CPU version 1.5.3
>   address sizes	: 40 bits physical, 48 bits virtual
> 
> 40 address lines allow 1TB, so we would place the window just below 1TB.
> 
> Comments?
More to the point if OS supports/enforces 1Tb physical address space,the RAM
and 64-bit PCI hole are going to contend for it, QEMU could abort on startup
if they both do not fit in CPU supported address space but I don't see what
else it could do. 

Proposed patch favors RAM vs 64-bit PCI hole and moves the hole behind the
possible RAM, which in present state of QEMU potentially leaves the rest of
address space up to 62 bits for hole.
It has drawback that one can't get a working VM if QEMU is started in
memory hotlug mode with old BIOS + PCI devices that require 64-bit bars,
otherwise it's backward compatible.

PS:
As for remedying BSODs because of huge CRS sizes of particular RAM device/PCI
window, it might be solved by splitting one big chunk in several smaller, at
least it works for RAM device.

> 
>   Gerd
> 
> 

> 


-- 
Regards,
  Igor