Re: HVM support for e820_host (Was: Bug: Limitation of <=2GB RAM in domU persists with 4.3.0)

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Thu, Sep 05, 2013 at 11:33:18PM +0100, Gordan Bobic wrote:
> On 09/05/2013 10:13 PM, Gordan Bobic wrote:
> 
> >I seem to be getting two different E820 table dumps with e820_host=1:
> >
> >(XEN) HVM1: BIOS map:
> >(XEN) HVM1:  f0000-fffff: Main BIOS
> >(XEN) HVM1: build_e820_table:91 got 8 op.nr_entries
> >(XEN) HVM1: E820 table:
> >(XEN) HVM1:  [00]: 00000000:00000000 - 00000000:3f790000: RAM
> >(XEN) HVM1:  [01]: 00000000:3f790000 - 00000000:3f79e000: ACPI
> >(XEN) HVM1:  [02]: 00000000:3f79e000 - 00000000:3f7d0000: NVS
> >(XEN) HVM1:  [03]: 00000000:3f7d0000 - 00000000:3f7e0000: RESERVED
> >(XEN) HVM1:  HOLE: 00000000:3f7e0000 - 00000000:3f7e7000
> >(XEN) HVM1:  [04]: 00000000:3f7e7000 - 00000000:40000000: RESERVED
> >(XEN) HVM1:  HOLE: 00000000:40000000 - 00000000:fee00000
> >(XEN) HVM1:  [05]: 00000000:fee00000 - 00000000:fee01000: RESERVED
> >(XEN) HVM1:  HOLE: 00000000:fee01000 - 00000000:ffc00000
> >(XEN) HVM1:  [06]: 00000000:ffc00000 - 00000001:00000000: RESERVED
> >(XEN) HVM1:  [07]: 00000001:00000000 - 00000001:68870000: RAM
> 
> I get it - this is the host e820 map. In dom0, dmesg shows:
> 
> e820: BIOS-provided physical RAM map:
> Xen: [mem 0x0000000000000000-0x000000000009cfff] usable
> Xen: [mem 0x000000000009d000-0x00000000000fffff] reserved
> Xen: [mem 0x0000000000100000-0x000000003f78ffff] usable
> Xen: [mem 0x000000003f790000-0x000000003f79dfff] ACPI data
> Xen: [mem 0x000000003f79e000-0x000000003f7cffff] ACPI NVS
> Xen: [mem 0x000000003f7d0000-0x000000003f7dffff] reserved
> Xen: [mem 0x000000003f7e7000-0x000000003fffffff] reserved
> Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
> Xen: [mem 0x00000000ffc00000-0x00000000ffffffff] reserved
> Xen: [mem 0x0000000100000000-0x0000000cbfffffff] usable
> 
> That tallies up with the above map exactly. So far so good. Not sure
> if the following is relevant, but here it is anyway just in case:
> 
> e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
> e820: remove [mem 0x000a0000-0x000fffff] usable
> [...]
> e820: last_pfn = 0xcc0000 max_arch_pfn = 0x400000000
> e820: last_pfn = 0x3f790 max_arch_pfn = 0x400000000
> [...]
> Zone ranges:
>   DMA      [mem 0x00001000-0x00ffffff]
>   DMA32    [mem 0x01000000-0xffffffff]
>   Normal   [mem 0x100000000-0xcbfffffff]
> [...]
> e820: [mem 0x40000000-0xfedfffff] available for PCI devices
> 
> 
> >(XEN) HVM1: E820 table:
> >(XEN) HVM1:  [00]: 00000000:00000000 - 00000000:0009e000: RAM
> >(XEN) HVM1:  [01]: 00000000:0009e000 - 00000000:000a0000: RESERVED
> >(XEN) HVM1:  HOLE: 00000000:000a0000 - 00000000:000e0000
> >(XEN) HVM1:  [02]: 00000000:000e0000 - 00000000:00100000: RESERVED
> >(XEN) HVM1:  [03]: 00000000:00100000 - 00000000:a7800000: RAM
> >(XEN) HVM1:  HOLE: 00000000:a7800000 - 00000000:fc000000
> >(XEN) HVM1:  [04]: 00000000:fc000000 - 00000001:00000000: RESERVED
> >(XEN) HVM1: Invoking ROMBIOS ...
> 
> Comparing this to the above, it seems that 9d000-9e000 is marked as
> reserved in dom0, but RAM in domU. Am I right in thinking that
> dom0(usable) == domU(RAM) in terms of meaning?
> 
> What does "HOLE" actually mean in domU? Does it mean this space is
> OK to map domU IOMEM into? Or something else? Either way full
> possible chasl summary:
> 
> dom0: reserved	9d000-9e000
> domU: RAM	9d000-9e000
> 
> dom0: reserved	a0000-dffff
> domU: HOLE	a0000-dffff
> 
> dom0: ACPI data	3f790000-3f79dfff
> dom0: ACPI NVS	3f79e000-3f7cffff
> dom0: reserved	3f7d0000-3f7dffff
> dom0: reserved	


.. you are missing a range here.

> domU: RAM	00100000-a7800000
> 
> Then there seems to be a hole in dom0:
> 40000000-fedfffff which talles up with the dom0 dmesg output above
> about it being for the PCI devices, i.e. that's the IOMEM region
> (from 1GB to a lilttle under 4GB).
> 
> But in domU, the 40000000-a77fffff is available as RAM.

OK, so that is the goal - make hvmloader construct the E820 memory
layout and all of its pieces to fit that layout.

> 
> On the face of it, that's actually fine - my PCI IOMEM mappings show
> the lowest mapping (according to lspci -vvv) starts at a8000000,

<surprise>

> which falls into the domU area marked as "HOLE" (a7800000-fc000000).
> And this does in fact appears to be where domU maps the GPU in both
> of my VMs:
> 
> E0000000-E7FFFFFF
> E8000000-EBFFFFFF
> EC000000-EDFFFFFF
> 
> and this doesn't overlap with any mapped PCI IOMEM according to lspci.
> 
> If we assume that anything below a8000000 doesn't actually matter in
> this case (since if I give up to a8000000 memory to a domU
> everything works absolutely fine indefinitely, I am at a loss to


Just to make sure I am not leading you astray. You are getting _no_ crashes
when you have a guest with 1GB?

> explain what is actually going wrong and why the crash is still
> occuring - unless some other piece of hardware is having it's domU
> IOMEM mapped somewhere in the range f3df4000-fec8b000 and that is
> causing a memory overwrite.
> 
> I am just not seeing any obvious memory stomp at the moment...

Neither am I.
> 
> Gordan