Re: Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0

[Joshua West <jwest@brandeis.edu>]

Hi Dan,

Dan Magenheimer wrote:
> Joshua --
>
> Why do you need so much memory on dom0?  Are you thinking
> that the total memory available to guests is limited to
> what dom0 has?  Or are you running some memory-hungry
> applications on dom0?
>   

Hmm, I thought that a 32-bit dom0 system can only have up to 16GB of 
memory available to it and its domU's -- and a 64-bit dom0 can have a 
max of 32GB.  I only allocate ~512MB of memory to the dom0's 
themselves.  "xm info" shows us the rest of whats available for use by 
domU's.

Thus, even if I have 32GB of RAM in a server, the total memory that will 
be used in dom0 + all domU's can only add up to 16GB.  Am I just plain 
old wrong?

> Note that 64-bit Xen + 32-bit Dom0 will support both 32-bit
> and 64-bit domU's. (Oracle VM uses this configuration by
> default on 64-bit capable machines.)  And save/restore/migration
> has been tested to work for this (with the patch Kurt posted).
>   

Interesting.  For some reason I thought that the Hypervisor had to match 
the dom0's kernel.  So you're saying that if I compile the Xen 
Hypervisor as 64-bit and my dom0 + domU's both as 32-bit, I can reach 
the full capacity of my servers with respect to RAM (32GB) instead of 
being limited to 16GB?  (And this only works with Kurt's patches?).

> Dan
>
>   
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com
>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of Joshua West
>> Sent: Friday, January 18, 2008 4:06 PM
>> To: Keir Fraser
>> Cc: Kurt Hackel; xen-devel@lists.xensource.com
>> Subject: Re: [Xen-devel] Migrate/Save of 32-bit domU Broken 
>> on Xen 3.1.2
>> 64-bit dom0
>>
>>
>> Dang, doesn't work in Xen 3.1.3-rc1 (like you said/expected), with or
>> without Kurt's patches.
>>
>> Hmm... I thought 32-bit paravirtualized domU's on 64-bit dom0's were
>> supported in Xen 3.1.x?  Or am I mistaken?
>>
>> Perhaps standard features such as start/shutdown/pause of 32-bit PV
>> domU's on 64-bit dom0's are supported, but not the more advanced
>> management features like save and migrate - and I was just 
>> unclear from
>> the beginning?
>>
>> And of course... the final question that all developers hate... when
>> s/could one _hope_ (not expect!) to see save/migrate functionality of
>> 32-bit PV domU's on 64-bit dom0's? :-D
>>
>> Our whole reason for moving to 64-bit dom0's was to utilize 32GB of
>> memory, instead of the 16GB limitation on 32-bit PAE dom0's.  We can't
>> convert all of our virtual machines (100+) to 64-bit linux as its too
>> much work and some actually have to be 32-bit due to vendor supplied
>> software (Oracle, Zimbra, etc).  I suppose we'll wait to move 
>> to 64-bit
>> until [live] migration is available, as its used quite 
>> frequently in our
>> machine rooms.
>>
>> Thanks for all of your help Keir and Kurt.
>>
>> Keir Fraser wrote:
>>     
>>> Looking at the issues that need to be handled to support 
>>>       
>> this, it cannot be
>>     
>>> expected to work.
>>>
>>>  -- Keir
>>>
>>> On 18/1/08 22:13, "Joshua West" <jwest@brandeis.edu> wrote:
>>>
>>>
>>>       
>>>> Ahh.  Well, I'm building xen-3.1.3-rc1 right now and I'll 
>>>>         
>> see how that goes.
>>     
>>>> Keir, have you seen this issue before?  Where one cannot 
>>>>         
>> xm save/migrate
>>     
>>>> a 32-bit domU on a 64-bit dom0?  Or is this expected to 
>>>>         
>> work and just
>>     
>>>> doesn't go well for me?
>>>>
>>>> Keir Fraser wrote:
>>>>
>>>>         
>>>>> On 18/1/08 22:05, "Joshua West" <jwest@brandeis.edu> wrote:
>>>>>
>>>>>
>>>>>
>>>>>           
>>>>>> Just to make sure the patches aren't the cause of this new error
>>>>>> message, I'm going to see what happens with Xen 
>>>>>>             
>> 3.1.3-rc1 w/o patches.
>>     
>>>>>> Kurt, Keir, everybody else... any ideas as to why "xm 
>>>>>>             
>> save" (and thus
>>     
>>>>>> "xm migrate") of 32-bit domU's does not work on 64-bit 
>>>>>>             
>> dom0's?  Has
>>     
>>>>>> anybody found this to be successful in the past?
>>>>>>
>>>>>>
>>>>>>             
>>>>> The extent of the patches that Kurt posted indicate its 
>>>>>           
>> pretty unlikely to
>>     
>>>>> work without considerable code modification. I guess 
>>>>>           
>> those patches aren't
>>     
>>>>> quite fully baked yet, unfortunately, or haven't been 
>>>>>           
>> written with your
>>     
>>>>> precise case in mind.
>>>>>
>>>>>  -- Keir
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>           
>>>
>>>       
>> --
>> Joshua West
>> Systems Engineer
>> Brandeis University
>> http://www.brandeis.edu
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
>>     
>
>   


-- 
Joshua West
Systems Engineer
Brandeis University
http://www.brandeis.edu