Re: Debian linux-image-2.6.32-4-xen-amd64 2.6.32-11 doesn't boot with > 4 GiB; resets immediatelly, no log messages

[Jeremy Fitzhardinge <jeremy@goop.org>]

On 04/13/2010 03:27 PM, Thomas Goirand wrote:
> Jeremy Fitzhardinge wrote:
>   
>> On 04/12/2010 06:20 PM, Thomas Goirand wrote:
>>     
>>> Then do you have an idea why "xm info" shows a bit more, 900 MB free /
>>> 3990 MB total, while "xm list" shows only a 2048 dom0 running??? I'm not
>>> talking about just few bytes here, but one third of my RAM gone ...
>>> What's going on?
>>>   
>>>       
>> Is this immediately after boot, or have you started/shutdown some domains?
>>
>> Also, can you include the xen console output from "xm debug-keys m" as well?
>>
>>     J
>>     
> Hi Jeremy,
>
> Let me explain again.
>
> After boot-up situation (running KDE 3.5 and typing this message), xm
> info shows 3990 RAM (I believe the 4GB is 4096MB, unless Xen counts RAM
> like HDD vendors...)

Yes, you system has 3990MB of RAM; the hardware/BIOS is hiding, using or
otherwise disabling 106MB of memory.

>  and 645 MB free, and "xm list" shows that dom0 is
> using 3286 MB. Already, things are weird because 645 free + 3286 for the
> dom0 makes 3931 and not 3990, and I never asked that my dom0 takes
> anything (at boot-up it should take all the available memory).
>   

I would assume that extra 59MB is memory Xen has allocated for its own
purposes on behalf of the domain.  And Xen won't give all memory to dom0
at boot; it will always leave some memory free.

> Then I do:
>
> xm mem-set Domain-0 2048
>
> and then I get:
> xm list Domain-0: 2048
> xm info Free RAM: 1627
>
> That makes a total of 3675 MB, which is less than the total of what I
> had at boot time without using xm mem-set.

So the apparent overhead has grown from 59MB to 315MB?

>  Ever more weirdness: the
> amount of free RAM after a xm mem-set call does vary randomly each time
> I boot my laptop (a one year old Lenovo T500).
>   

How much variation?  By a few pages?  10s of MB?

What happens if you just boot with dom0_mem=2048MB at boot time rather
than shrinking it after boot?

> Note that I tried upgrading to the very latest version of Xen and pv_ops
> kernel from the pkg-xen team, so I tried both kernel 2.6.32-10 /
> hypervisor 3.4.2-3 and kernel 2.6.32-11 / hypervisor 3.4.2-4. They both
> had the issue. I'm using them in Lenny (I rebuilt the Debian package
> from source to create my own backports, which I don't believe is an
> issue as the same thing has been uploaded to backports.org).
>   

This should all be an internal-to-Xen issue, so I wouldn't expect the
kernel to have any effect.

> "xm debug-keys m" shows nothing, when exactly should I type it (sorry,
> I'm not familiar with Xen debugging...).
>   

The result will appear on the console, so you may need to follow it with
"xm dmesg" to see the output.

I don't know all that much about how Xen does its internal memory
management, but aside from the ~300MB apparently lost by shrinking dom0
the amounts of memory you're talking about are small, and well within
the sort of range I'd expect for allocator overhead.  But even then,
looking at "free memory" numbers is notoriously unreliable, because that
doesn't necessarily represent the amount of memory that's actually
available for allocation - it could be higher, because something that's
using memory can be convinced to release it if needed.

I did some similar experiments on my system and none of the numbers add
up properly for me either.  Which suggests to me that the "free" number
in xm info is not the whole truth, and that the memory accounting is
keeping multiple books...

    J