From mboxrd@z Thu Jan 1 00:00:00 1970 From: Gordan Bobic Subject: Re: HVM support for e820_host (Was: Bug: Limitation of <=2GB RAM in domU persists with 4.3.0) Date: Thu, 05 Sep 2013 23:33:18 +0100 Message-ID: <522906AE.3070402@bobich.net> References: <1374798084.10269.2.camel@hastur.hellion.org.uk> <20130729180431.GQ5848@phenom.dumpdata.com> <20130903145934.GC1487@konrad-lan.dumpdata.com> <52263CBD.1090402@bobich.net> <52264826.3010402@bobich.net> <20130903210833.GB13777@phenom.dumpdata.com> <20130905020442.GA2459@phenom.dumpdata.com> <5228F3E2.8090905@bobich.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <5228F3E2.8090905@bobich.net> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Konrad Rzeszutek Wilk Cc: xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org On 09/05/2013 10:13 PM, Gordan Bobic wrote: > I seem to be getting two different E820 table dumps with e820_host=1: > > (XEN) HVM1: BIOS map: > (XEN) HVM1: f0000-fffff: Main BIOS > (XEN) HVM1: build_e820_table:91 got 8 op.nr_entries > (XEN) HVM1: E820 table: > (XEN) HVM1: [00]: 00000000:00000000 - 00000000:3f790000: RAM > (XEN) HVM1: [01]: 00000000:3f790000 - 00000000:3f79e000: ACPI > (XEN) HVM1: [02]: 00000000:3f79e000 - 00000000:3f7d0000: NVS > (XEN) HVM1: [03]: 00000000:3f7d0000 - 00000000:3f7e0000: RESERVED > (XEN) HVM1: HOLE: 00000000:3f7e0000 - 00000000:3f7e7000 > (XEN) HVM1: [04]: 00000000:3f7e7000 - 00000000:40000000: RESERVED > (XEN) HVM1: HOLE: 00000000:40000000 - 00000000:fee00000 > (XEN) HVM1: [05]: 00000000:fee00000 - 00000000:fee01000: RESERVED > (XEN) HVM1: HOLE: 00000000:fee01000 - 00000000:ffc00000 > (XEN) HVM1: [06]: 00000000:ffc00000 - 00000001:00000000: RESERVED > (XEN) HVM1: [07]: 00000001:00000000 - 00000001:68870000: RAM I get it - this is the host e820 map. In dom0, dmesg shows: e820: BIOS-provided physical RAM map: Xen: [mem 0x0000000000000000-0x000000000009cfff] usable Xen: [mem 0x000000000009d000-0x00000000000fffff] reserved Xen: [mem 0x0000000000100000-0x000000003f78ffff] usable Xen: [mem 0x000000003f790000-0x000000003f79dfff] ACPI data Xen: [mem 0x000000003f79e000-0x000000003f7cffff] ACPI NVS Xen: [mem 0x000000003f7d0000-0x000000003f7dffff] reserved Xen: [mem 0x000000003f7e7000-0x000000003fffffff] reserved Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved Xen: [mem 0x00000000ffc00000-0x00000000ffffffff] reserved Xen: [mem 0x0000000100000000-0x0000000cbfffffff] usable That tallies up with the above map exactly. So far so good. Not sure if the following is relevant, but here it is anyway just in case: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved e820: remove [mem 0x000a0000-0x000fffff] usable [...] e820: last_pfn = 0xcc0000 max_arch_pfn = 0x400000000 e820: last_pfn = 0x3f790 max_arch_pfn = 0x400000000 [...] Zone ranges: DMA [mem 0x00001000-0x00ffffff] DMA32 [mem 0x01000000-0xffffffff] Normal [mem 0x100000000-0xcbfffffff] [...] e820: [mem 0x40000000-0xfedfffff] available for PCI devices > (XEN) HVM1: E820 table: > (XEN) HVM1: [00]: 00000000:00000000 - 00000000:0009e000: RAM > (XEN) HVM1: [01]: 00000000:0009e000 - 00000000:000a0000: RESERVED > (XEN) HVM1: HOLE: 00000000:000a0000 - 00000000:000e0000 > (XEN) HVM1: [02]: 00000000:000e0000 - 00000000:00100000: RESERVED > (XEN) HVM1: [03]: 00000000:00100000 - 00000000:a7800000: RAM > (XEN) HVM1: HOLE: 00000000:a7800000 - 00000000:fc000000 > (XEN) HVM1: [04]: 00000000:fc000000 - 00000001:00000000: RESERVED > (XEN) HVM1: Invoking ROMBIOS ... Comparing this to the above, it seems that 9d000-9e000 is marked as reserved in dom0, but RAM in domU. Am I right in thinking that dom0(usable) == domU(RAM) in terms of meaning? What does "HOLE" actually mean in domU? Does it mean this space is OK to map domU IOMEM into? Or something else? Either way full possible chasl summary: dom0: reserved 9d000-9e000 domU: RAM 9d000-9e000 dom0: reserved a0000-dffff domU: HOLE a0000-dffff dom0: ACPI data 3f790000-3f79dfff dom0: ACPI NVS 3f79e000-3f7cffff dom0: reserved 3f7d0000-3f7dffff dom0: reserved domU: RAM 00100000-a7800000 Then there seems to be a hole in dom0: 40000000-fedfffff which talles up with the dom0 dmesg output above about it being for the PCI devices, i.e. that's the IOMEM region (from 1GB to a lilttle under 4GB). But in domU, the 40000000-a77fffff is available as RAM. On the face of it, that's actually fine - my PCI IOMEM mappings show the lowest mapping (according to lspci -vvv) starts at a8000000, which falls into the domU area marked as "HOLE" (a7800000-fc000000). And this does in fact appears to be where domU maps the GPU in both of my VMs: E0000000-E7FFFFFF E8000000-EBFFFFFF EC000000-EDFFFFFF and this doesn't overlap with any mapped PCI IOMEM according to lspci. If we assume that anything below a8000000 doesn't actually matter in this case (since if I give up to a8000000 memory to a domU everything works absolutely fine indefinitely, I am at a loss to explain what is actually going wrong and why the crash is still occuring - unless some other piece of hardware is having it's domU IOMEM mapped somewhere in the range f3df4000-fec8b000 and that is causing a memory overwrite. I am just not seeing any obvious memory stomp at the moment... Gordan