Re: [PATCH] libxl, hvmloader: Don't relocate memory for MMIO hole

[George Dunlap <george.dunlap@eu.citrix.com>]

On 17/06/13 17:43, George Dunlap wrote:
> At the moment, qemu-xen can't handle memory being relocated by
> hvmloader.  This may happen if a device with a large enough memory
> region is passed through to the guest.  At the moment, if this
> happens, then at some point in the future qemu will crash and the
> domain will hang.  (qemu-traditional is fine.)
>
> It's too late in the release to do a proper fix, so we try to do
> damage control.
>
> hvmloader already has mechanisms to relocate memory to 64-bit space
> if it can't make a big enough MMIO hole.  By default this is 2GiB; if
> we just refuse to make the hole bigger if it will overlap with guest
> memory, then this codepath will be taken by default.
>
> Signed-off-by: George Dunlap <george.dunlap@eu.citrix.com>
> CC: Ian Campbell <ian.campbell@citrix.com>
> CC: Ian Jackson <ian.jackson@citrix.com>
> CC: Stefano Stabellini <stefano.stabellini@citrix.com>
> CC: Hanweidong <hanweidong@huawei.com>

This isn't exactly what we had discussed, but looking at the code, I 
think this is the easiest thing to do.

A brief description of what hvmloader does:
1. enumerate all BARs from all devices, ordering them from large to small
2. Double* the size of the <4G MMIO hole until it is big enough to fit 
all the MMIO, or until it reaches 2GiB, whichever comes first
3. Move guest memory which overlaps the <4G MMIO hole above 4G.
4. Place devices in memory, placing devices above 4G if the hole isn't 
big enough for everything.

Unfortunately, the option of "just don't map the passed-through device 
if there's not enough MMIO space" doesn't really work -- there's no way 
to really tell which device is the one you shouldn't map.

Of course we have the same issue with relocating devices to 64-bit: we 
can't tell which devices will be located above 4G or not.  So for 
devices not capable of accessing physical address space above 4G, random 
devices will simply not appear; this may cause strange problems if the 
emulated network card or disk, for example, is not visible.  There may 
also be examples of assumptions baked into drivers or emulation code 
about the physical address space.  On the whole, given that this is 
KVM's behavior, I'm inclined to be optimistic about the state of guest 
OSes and drivers.

Ian Jackson had proposed just crashing in hvmloader if we were using 
qemu-xen and needed to relocate memory.  This changes a "crash randomly 
at some point after boot" into a "crash immediately before the BIOS even 
runs".  While that is certainly an improvement, I think on the whole 
that it's still bad enough that "mostly works but occasionally has 
strange failures" is still better; hence this patch.

  -George