Re: HVM support for e820_host (Was: Bug: Limitation of <=2GB RAM in domU persists with 4.3.0)

[Gordan Bobic <gordan@bobich.net>]

On 09/05/2013 10:13 PM, Gordan Bobic wrote:

> I seem to be getting two different E820 table dumps with e820_host=1:
>
> (XEN) HVM1: BIOS map:
> (XEN) HVM1:  f0000-fffff: Main BIOS
> (XEN) HVM1: build_e820_table:91 got 8 op.nr_entries
> (XEN) HVM1: E820 table:
> (XEN) HVM1:  [00]: 00000000:00000000 - 00000000:3f790000: RAM
> (XEN) HVM1:  [01]: 00000000:3f790000 - 00000000:3f79e000: ACPI
> (XEN) HVM1:  [02]: 00000000:3f79e000 - 00000000:3f7d0000: NVS
> (XEN) HVM1:  [03]: 00000000:3f7d0000 - 00000000:3f7e0000: RESERVED
> (XEN) HVM1:  HOLE: 00000000:3f7e0000 - 00000000:3f7e7000
> (XEN) HVM1:  [04]: 00000000:3f7e7000 - 00000000:40000000: RESERVED
> (XEN) HVM1:  HOLE: 00000000:40000000 - 00000000:fee00000
> (XEN) HVM1:  [05]: 00000000:fee00000 - 00000000:fee01000: RESERVED
> (XEN) HVM1:  HOLE: 00000000:fee01000 - 00000000:ffc00000
> (XEN) HVM1:  [06]: 00000000:ffc00000 - 00000001:00000000: RESERVED
> (XEN) HVM1:  [07]: 00000001:00000000 - 00000001:68870000: RAM

I get it - this is the host e820 map. In dom0, dmesg shows:

e820: BIOS-provided physical RAM map:
Xen: [mem 0x0000000000000000-0x000000000009cfff] usable
Xen: [mem 0x000000000009d000-0x00000000000fffff] reserved
Xen: [mem 0x0000000000100000-0x000000003f78ffff] usable
Xen: [mem 0x000000003f790000-0x000000003f79dfff] ACPI data
Xen: [mem 0x000000003f79e000-0x000000003f7cffff] ACPI NVS
Xen: [mem 0x000000003f7d0000-0x000000003f7dffff] reserved
Xen: [mem 0x000000003f7e7000-0x000000003fffffff] reserved
Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
Xen: [mem 0x00000000ffc00000-0x00000000ffffffff] reserved
Xen: [mem 0x0000000100000000-0x0000000cbfffffff] usable

That tallies up with the above map exactly. So far so good. Not sure if 
the following is relevant, but here it is anyway just in case:

e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
e820: remove [mem 0x000a0000-0x000fffff] usable
[...]
e820: last_pfn = 0xcc0000 max_arch_pfn = 0x400000000
e820: last_pfn = 0x3f790 max_arch_pfn = 0x400000000
[...]
Zone ranges:
   DMA      [mem 0x00001000-0x00ffffff]
   DMA32    [mem 0x01000000-0xffffffff]
   Normal   [mem 0x100000000-0xcbfffffff]
[...]
e820: [mem 0x40000000-0xfedfffff] available for PCI devices


> (XEN) HVM1: E820 table:
> (XEN) HVM1:  [00]: 00000000:00000000 - 00000000:0009e000: RAM
> (XEN) HVM1:  [01]: 00000000:0009e000 - 00000000:000a0000: RESERVED
> (XEN) HVM1:  HOLE: 00000000:000a0000 - 00000000:000e0000
> (XEN) HVM1:  [02]: 00000000:000e0000 - 00000000:00100000: RESERVED
> (XEN) HVM1:  [03]: 00000000:00100000 - 00000000:a7800000: RAM
> (XEN) HVM1:  HOLE: 00000000:a7800000 - 00000000:fc000000
> (XEN) HVM1:  [04]: 00000000:fc000000 - 00000001:00000000: RESERVED
> (XEN) HVM1: Invoking ROMBIOS ...

Comparing this to the above, it seems that 9d000-9e000 is marked as 
reserved in dom0, but RAM in domU. Am I right in thinking that
dom0(usable) == domU(RAM) in terms of meaning?

What does "HOLE" actually mean in domU? Does it mean this space is OK to 
map domU IOMEM into? Or something else? Either way full possible chasl 
summary:

dom0: reserved	9d000-9e000
domU: RAM	9d000-9e000

dom0: reserved	a0000-dffff
domU: HOLE	a0000-dffff

dom0: ACPI data	3f790000-3f79dfff
dom0: ACPI NVS	3f79e000-3f7cffff
dom0: reserved	3f7d0000-3f7dffff
dom0: reserved	
domU: RAM	00100000-a7800000

Then there seems to be a hole in dom0:
40000000-fedfffff which talles up with the dom0 dmesg output above about 
it being for the PCI devices, i.e. that's the IOMEM region (from 1GB to 
a lilttle under 4GB).

But in domU, the 40000000-a77fffff is available as RAM.

On the face of it, that's actually fine - my PCI IOMEM mappings show the 
lowest mapping (according to lspci -vvv) starts at a8000000, which falls 
into the domU area marked as "HOLE" (a7800000-fc000000). And this does 
in fact appears to be where domU maps the GPU in both of my VMs:

E0000000-E7FFFFFF
E8000000-EBFFFFFF
EC000000-EDFFFFFF

and this doesn't overlap with any mapped PCI IOMEM according to lspci.

If we assume that anything below a8000000 doesn't actually matter in 
this case (since if I give up to a8000000 memory to a domU everything 
works absolutely fine indefinitely, I am at a loss to explain what is 
actually going wrong and why the crash is still occuring - unless some 
other piece of hardware is having it's domU IOMEM mapped somewhere in 
the range f3df4000-fec8b000 and that is causing a memory overwrite.

I am just not seeing any obvious memory stomp at the moment...

Gordan