memory map issues with PV PCI passthrough

[Daniel De Graaf <dgdegra@tycho.nsa.gov>]

I have a system with several reserved ranges low in the e820 map which
cause problems when starting PV domains with PCI devices. The machine
memory map looks like:

(XEN)  0000000000000000 - 0000000000060000 (usable)
(XEN)  0000000000060000 - 0000000000068000 (reserved)
(XEN)  0000000000068000 - 000000000009ac00 (usable)
(XEN)  000000000009ac00 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000000800000 (usable)
(XEN)  0000000000800000 - 000000000087d000 (unusable)
(XEN)  000000000087d000 - 0000000000f00000 (usable)
(XEN)  0000000000f00000 - 0000000001000000 (reserved)
(XEN)  0000000001000000 - 0000000020000000 (usable)
(XEN)  0000000020000000 - 0000000020200000 (reserved)
(XEN)  0000000020200000 - 0000000040000000 (usable)
(XEN)  0000000040000000 - 0000000040200000 (reserved)
(XEN)  0000000040200000 - 00000000c95d6000 (usable)
(XEN)  00000000c95d6000 - 00000000c961a000 (reserved)
(XEN)  00000000c961a000 - 00000000c99b7000 (usable)
(XEN)  00000000c99b7000 - 00000000c99e7000 (reserved)
(XEN)  00000000c99e7000 - 00000000c9be7000 (ACPI NVS)
(XEN)  00000000c9be7000 - 00000000c9bff000 (ACPI data)
(XEN)  00000000c9bff000 - 00000000c9c00000 (usable)
(XEN)  00000000c9f00000 - 00000000ca000000 (reserved)
(XEN)  00000000cb000000 - 00000000cf200000 (reserved)
(XEN)  00000000fed1c000 - 00000000fed30000 (reserved)
(XEN)  00000000ffc00000 - 00000000ffc20000 (reserved)
(XEN)  0000000100000000 - 000000042e000000 (usable)

When e820_sanitize is called on this memory map to create a PV domain, the
resulting map has only one usable region (0-0xf00000) below 4GB, and Linux
will not boot with this memory map.

I have a patch that reworks e820_sanitize to include later RAM regions as
valid RAM, which works as long as the domain being booted has permission
to map the PFNs from 0x20000-0x20200 and 0x40000-0x40200. If the domain is
not given this permission (the default, since these regions are not part of
the PCI device being passed to the guest) then the hypervisor crashes the
domain when it attempts to map these regions (during init_memory_mapping).

The domain will boot when these regions are not marked as reserved in the
e820 map or when the PFNs 0x20200-0x40000 and 0x40200-0xc95d6 are marked
as unusable. However, it is difficult to make this happen in any general
case without knowing what reserved regions actually need to be marked as
reserved in the guest.

If PCI hot-add is not needed, the problem becomes simpler: the PCI regions
for assigned devices can be included in the e820 map and other regions can
be ignored (marking as RAM so that the guest does not attempt direct map).

Any suggestions on the best way to resolve this?

-- 
Daniel De Graaf
National Security Agency