Re: [PATCH v2 1/1] tools: Handle xc_maxmem adjustments

[Don Slutz <dslutz@verizon.com>]

On 04/15/15 07:48, Stefano Stabellini wrote:
> On Wed, 15 Apr 2015, Andrew Cooper wrote:
>> On 15/04/15 10:57, Stefano Stabellini wrote:
>>> On Wed, 15 Apr 2015, Andrew Cooper wrote:
>>>> On 14/04/15 23:06, Don Slutz wrote:
>>>>> This fixes an issue where "xl save" followed by "xl restore" reports:
>>>>> "xc: error: Failed to allocate memory for batch.!: Internal error"
>>>>>
>>>>> One of the ways to get into this state is to have more then 4 e1000
>>>>> nics configured.
>>>>>
>>>>> Signed-off-by: Don Slutz <dslutz@verizon.com>
>>>> I still don't think this is the correct solution, although I will
>>>> concede that this is a far better patch than v1.
>>>>
>>>> Going back to the original problem, why does Qemu need to change maxmem
>>>> in the first place?  You talk about e1000 option roms, but the option
>>>> roms themselves must be allocated in an appropriate PCI bridge window.
>>> QEMU allocates actual memory to store the option roms using
>>> xc_domain_populate_physmap_exact, increasing the total amount of ram
>>> allocated to the guest. QEMU needs to increase maxmem too, otherwise
>>> memory allocations will fail after a few option roms.
>>
>> It doesn't need to.  The domain already has that much ram allocated to
>> itself and otherwise wasted.  Qemu should reuse that ram rather than
>> wasting more.

This "otherwise wasted" I do not understand at all.

for:

maxmem = "2056"
memory = "2056"
mmio_hole = "3072"
videoram = "8"
vif = [
 "model=e1000,bridge=xenbr0,mac=00:0c:29:86:44:a0",
 "model=e1000,bridge=xenbr0,mac=00:0c:29:86:44:aa",
 "model=e1000,bridge=xenbr0,mac=00:0c:29:86:44:b4",
 "model=e1000,bridge=xenbr0,mac=00:0c:29:86:44:be",
]


xc: detail: VIRTUAL MEMORY ARRANGEMENT:
xc: detail:   Loader:   0000000000100000->00000000001c4a1c
xc: detail:   Modules:  0000000000000000->0000000000000000
xc: detail:   TOTAL:    0000000000000000->0000000080000000
xc: detail:   ENTRY:    0000000000100038
xc: detail: PHYSICAL MEMORY ALLOCATION:
xc: detail:   4KB PAGES: 0x0000000000000200
xc: detail:   2MB PAGES: 0x00000000000001ff
xc: detail:   1GB PAGES: 0x0000000000000001


xl list:

e1000x4-2g                                  36  2057     1     --p---
    0.0

2057 is 2056 + LIBXL_MAXMEM_CONSTANT

As reported by xc_domain_getinfo()

info.max_memkb == 2106432 ==> pages 526608
info.nr_pages  == 526592  ==> 2106368 KiB

So there are 16 "wasted" pages, part of the pages from physical address
0xa0000 to 0xbffff.

hvmloader:

(d36) [2015-04-15 21:03:45] BIOS map:
(d36) [2015-04-15 21:03:45]  10000-100d3: Scratch space
(d36) [2015-04-15 21:03:45]  c0000-fffff: Main BIOS
(d36) [2015-04-15 21:03:45] E820 table:
(d36) [2015-04-15 21:03:45]  [00]: 00000000:00000000 -
00000000:000a0000: RAM
(d36) [2015-04-15 21:03:45]  HOLE: 00000000:000a0000 - 00000000:000c0000
(d36) [2015-04-15 21:03:45]  [01]: 00000000:000c0000 -
00000000:00100000: RESERVED
(d36) [2015-04-15 21:03:45]  [02]: 00000000:00100000 -
00000000:40000000: RAM
(d36) [2015-04-15 21:03:45]  HOLE: 00000000:40000000 - 00000000:fc000000
(d36) [2015-04-15 21:03:45]  [03]: 00000000:fc000000 -
00000001:00000000: RESERVED
(d36) [2015-04-15 21:03:45]  [04]: 00000001:00000000 -
00000001:40000000: RAM


Linux reports:

BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000003ffff000 (usable)
 BIOS-e820: 000000003ffff000 - 0000000040000000 (reserved)
 BIOS-e820: 00000000fc000000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 0000000140000000 (usable)


Note: the e820 map is missing the VGA hole of 0xa0000 - 0xc0000.

So I see 16 pages < 1GiB below 4GiB and 1GiB above 4Gib
8 MiB of video RAM, which is real close to the 2056 KiB
of ram the guest was configured with.

If I drop all the e1000 nics (the QEMU option ROMs
that contain code that is executed by the guest to PXE boot
from SeaBIOS):

xc: detail: VIRTUAL MEMORY ARRANGEMENT:
xc: detail:   Loader:   0000000000100000->00000000001c4a1c
xc: detail:   Modules:  0000000000000000->0000000000000000
xc: detail:   TOTAL:    0000000000000000->0000000080000000
xc: detail:   ENTRY:    0000000000100038
xc: detail: PHYSICAL MEMORY ALLOCATION:
xc: detail:   4KB PAGES: 0x0000000000000200
xc: detail:   2MB PAGES: 0x00000000000001ff
xc: detail:   1GB PAGES: 0x0000000000000001


(d2) [2015-04-15 21:35:18]  c0000-fffff: Main BIOS
(d2) [2015-04-15 21:35:18] E820 table:
(d2) [2015-04-15 21:35:18]  [00]: 00000000:00000000 - 00000000:000a0000: RAM
(d2) [2015-04-15 21:35:18]  HOLE: 00000000:000a0000 - 00000000:000c0000
(d2) [2015-04-15 21:35:18]  [01]: 00000000:000c0000 - 00000000:00100000:
RESERVED
(d2) [2015-04-15 21:35:18]  [02]: 00000000:00100000 - 00000000:40000000: RAM
(d2) [2015-04-15 21:35:18]  HOLE: 00000000:40000000 - 00000000:fc000000
(d2) [2015-04-15 21:35:18]  [03]: 00000000:fc000000 - 00000001:00000000:
RESERVED
(d2) [2015-04-15 21:35:18]  [04]: 00000001:00000000 - 00000001:40000000: RAM
(d2) [2015-04-15 21:35:18] Invoking SeaBIOS ...
...
(d2) [2015-04-15 21:35:21] Returned 258048 bytes of ZoneHigh
(d2) [2015-04-15 21:35:21] e820 map has 7 items:
(d2) [2015-04-15 21:35:21]   0: 0000000000000000 - 000000000009fc00 = 1 RAM
(d2) [2015-04-15 21:35:21]   1: 000000000009fc00 - 00000000000a0000 = 2
RESERVED
(d2) [2015-04-15 21:35:21]   2: 00000000000f0000 - 0000000000100000 = 2
RESERVED
(d2) [2015-04-15 21:35:21]   3: 0000000000100000 - 000000003ffff000 = 1 RAM
(d2) [2015-04-15 21:35:21]   4: 000000003ffff000 - 0000000040000000 = 2
RESERVED
(d2) [2015-04-15 21:35:21]   5: 00000000fc000000 - 0000000100000000 = 2
RESERVED
(d2) [2015-04-15 21:35:21]   6: 0000000100000000 - 0000000140000000 = 1 RAM

And linux:

BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000003ffff000 (usable)
 BIOS-e820: 000000003ffff000 - 0000000040000000 (reserved)
 BIOS-e820: 00000000fc000000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 0000000140000000 (usable)



max_pages=526592 (2106368(0x202400) KiB) nr_pages=526336
(2105344(0x202000) KiB) free_pages=256

And with LIBXL_MAXMEM_CONSTANT=1024, the 256 "free (wasted?)"
pages are this.

> 
> QEMU has no knowledge about what the e820 contains and considers the
> memory relocated from the PCI hole to the top of ram as valid guest
> memory. Libxl tells QEMU how big the PCI hole is by passing
> max-ram-below-4g to QEMU.
> 
> I think that we should fix the guest e820 as it is actually valid guest
> memory.
> 

If you are talking about the 1GiB over 4GiB in my example, it is already
in the e820 map.

So I do not understand the "wasted" ram issue.

    -Don Slutz