Dynamic Memory Management and Migration (3.0.4-0)

Dynamic Memory Management and Migration (3.0.4-0)

[Timo Benk]

Hello,

i noticed the following Problem:

if i want to allow dynamic memory management of Linux-domains, i need to
set the following kernel parameter. Otherwise it is not possible, to
increase the memory of the domain above the initial assigned value:

extra = "mem=10240M"

But now it is not possible to migrate the domain. If try an offline
migration, the xm-command hangs forever.

One time i tried a live migration, the domain just disappears. The
domain was neither running on the src-domain-0 nor on the dst-domain-0,
the domain just vanished.

Greetings,
-timo
-- 
Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731
PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc

Re: Dynamic Memory Management and Migration (3.0.4-0)

[Tim Wood]

On 1/31/07, Timo Benk <timo.benk@gmx.de> wrote:
> Hello,
>
> i noticed the following Problem:
>
> if i want to allow dynamic memory management of Linux-domains, i need to
> set the following kernel parameter. Otherwise it is not possible, to
> increase the memory of the domain above the initial assigned value:
>
> extra = "mem=10240M"
>
> But now it is not possible to migrate the domain. If try an offline
> migration, the xm-command hangs forever.

You may want to try initially giving the domain the full allocation
and then decreasing its share once it has booted.  You should be able
to expand back to the full amount.  I have used this method and
successfully performed migrations in the past.

>
> One time i tried a live migration, the domain just disappears. The
> domain was neither running on the src-domain-0 nor on the dst-domain-0,
> the domain just vanished.
>

I have noticed that live migration of domains which are not currently
allocated their full amount of memory can take much longer.  There was
speculation that this was because of millions of debug logs which
inadvertently get printed out and slow progress to a crawl.  This may
well have been fixed in recent versions of xen, I'm still stuck back a
generation or two on my systems.

Re: Dynamic Memory Management and Migration (3.0.4-0)

[Timo Benk]

Tim Wood wrote:
> On 1/31/07, Timo Benk <timo.benk@gmx.de> wrote:
>> Hello,
>>
>> i noticed the following Problem:
>>
>> if i want to allow dynamic memory management of Linux-domains, i need to
>> set the following kernel parameter. Otherwise it is not possible, to
>> increase the memory of the domain above the initial assigned value:
>>
>> extra = "mem=10240M"
>>
>> But now it is not possible to migrate the domain. If try an offline
>> migration, the xm-command hangs forever.
> 
> You may want to try initially giving the domain the full allocation
> and then decreasing its share once it has booted.  You should be able
> to expand back to the full amount.  I have used this method and
> successfully performed migrations in the past.
Yepp, that is the workaround. But if you have already 10 domains using
most of the memory, the domain  in question will not start 'cause there
is not enough memory.

Greetings,
-timo

-- 
Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731
PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc

Re: Dynamic Memory Management and Migration (3.0.4-0)

[Steven Hand]

> You may want to try initially giving the domain the full allocation
> and then decreasing its share once it has booted.  You should be able
> to expand back to the full amount.  I have used this method and
> successfully performed migrations in the past.

Yes, this should work in 3.0.4.

>> One time i tried a live migration, the domain just disappears. The
>> domain was neither running on the src-domain-0 nor on the dst-domain-0,
>> the domain just vanished.

This issue (better handling of failure during live migration) is planned to 
be
fixed for 3.0.5.

> I have noticed that live migration of domains which are not currently
> allocated their full amount of memory can take much longer.  There was
> speculation that this was because of millions of debug logs which
> inadvertently get printed out and slow progress to a crawl.  This may
> well have been fixed in recent versions of xen, I'm still stuck back a
> generation or two on my systems.

There's two issues here:
   1. excessive printfs() for ballooned down domains - this is fixed in both
       3.0.4-1 and -unstable
   2. lazy allocation of memory for ballooned down domains (so you don't
       need 'maxem' on the destination, just the current allocation); this 
is
       fixed in current -unstable.

In general it's worth upgrading to either the latest minor release or the 
tip
of the latest -testing tree.

cheers,

S.

Re: Dynamic Memory Management and Migration (3.0.4-0)

[Daniel P. Berrange]

On Wed, Jan 31, 2007 at 04:29:25PM +0100, Timo Benk wrote:
> Hello,
> 
> i noticed the following Problem:
> 
> if i want to allow dynamic memory management of Linux-domains, i need to
> set the following kernel parameter. Otherwise it is not possible, to
> increase the memory of the domain above the initial assigned value:
> 
> extra = "mem=10240M"

That should not be neccessary. We fixed the Hypervisor to implement the
neccessary magic so that XenD can utilize the 'maxmem' parameter in
guest config files correctly So for example

  mem=500
  maxmem=800

Will make the guest VM boot up with 500 MB of initial RAM. It will also
tell it to allocate page tables entries big enough to address 800 MB of
RAM. So after boot up, you will be able to balloon the guest upto this
maxmimum of 800 MB simply using 'xm mem-set 800'.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: Dynamic Memory Management and Migration (3.0.4-0)

[Timo Benk]

Daniel P. Berrange wrote:
> On Wed, Jan 31, 2007 at 04:29:25PM +0100, Timo Benk wrote:
>> Hello,
>>
>> i noticed the following Problem:
>>
>> if i want to allow dynamic memory management of Linux-domains, i need to
>> set the following kernel parameter. Otherwise it is not possible, to
>> increase the memory of the domain above the initial assigned value:
>>
>> extra = "mem=10240M"
> 
> That should not be neccessary. We fixed the Hypervisor to implement the
> neccessary magic so that XenD can utilize the 'maxmem' parameter in
> guest config files correctly So for example
> 
>   mem=500
>   maxmem=800
> 
> Will make the guest VM boot up with 500 MB of initial RAM. It will also
> tell it to allocate page tables entries big enough to address 800 MB of
> RAM. So after boot up, you will be able to balloon the guest upto this
> maxmimum of 800 MB simply using 'xm mem-set 800'.
A quick test with 3.0.4-0 does not work on my side. I will try it again
tomorrow with 3.0.4-1. I dunno when the patch was applied to the xen source.

Her is what i have tried:

Domain-0:~ # xm create /etc/xen/vm/xendom1 -n
Using config file "/etc/xen/vm/xendom1".
(vm
    (name xendom1)
    (memory 256)
    (maxmem 512)
    (vcpus 1)
    (on_xend_start ignore)
    (on_xend_stop ignore)
    (cpu_cap 100)
    (cpu_weight 100)
    (image
        (linux
            (kernel /boot/vmlinuz-2.6.16.33-xenU)
            (ramdisk /boot/initrd-2.6.16.33-xenU)
            (root '/dev/sda1 ro')
        )
    )
    (device (vbd (uname phy:/dev/xenvol/xendom1_root) (dev sda1) (mode w)))
    (device (vbd (uname phy:/dev/xenvol/xendom1_swap) (dev sda2) (mode w)))
    (device (vif (mac 00:11:25:67:30:65)))
)

Domain-0:~ # xm create /etc/xen/vm/xendom1
Using config file "/etc/xen/vm/xendom1".
Started domain xendom1

Domain-0:~ # xm list
Name                                      ID   Mem VCPUs      State   Time(s)
Domain-0                                   0  2265     1     r-----   4834.5
xendom1                                    6   256     1     -b----      4.3

Domain-0:~ # xm mem-set xendom1 300
Domain-0:~ # xm list
Name                                      ID   Mem VCPUs      State   Time(s)

Domain-0                                   0  2265     1     r-----   4835.2
xendom1                                    6   257     1     ------     13.1

Domain-0:~ # ssh root@172.17.0.101
Last login: Wed Jan 31 23:56:00 2007 from 172.17.0.1
xendom1:~ # free
             total       used       free     shared    buffers     cached
Mem:        262384     108404     153980          0       5840      63840
-/+ buffers/cache:      38724     223660
Swap:       524280          0     524280

xendom1:~ # logout
Connection to 172.17.0.101 closed.

Domain-0:~ # xm mem-set xendom1 400
Domain-0:~ # ssh root@172.17.0.101
Last login: Wed Jan 31 23:56:08 2007 from 172.17.0.128
xendom1:~ # free
             total       used       free     shared    buffers     cached
Mem:        262384     108908     153476          0       6388      63812
-/+ buffers/cache:      38708     223676
Swap:       524280          0     524280
xendom1:~ #


Greetings,
-timo
-- 
Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731
PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc

Re: Dynamic Memory Management and Migration (3.0.4-0)

[Timo Benk]

Timo Benk wrote:
> Daniel P. Berrange wrote:
>> On Wed, Jan 31, 2007 at 04:29:25PM +0100, Timo Benk wrote:
>>> Hello,
>>>
>>> i noticed the following Problem:
>>>
>>> if i want to allow dynamic memory management of Linux-domains, i need to
>>> set the following kernel parameter. Otherwise it is not possible, to
>>> increase the memory of the domain above the initial assigned value:
>>>
>>> extra = "mem=10240M"
>> That should not be neccessary. We fixed the Hypervisor to implement the
>> neccessary magic so that XenD can utilize the 'maxmem' parameter in
>> guest config files correctly So for example
>>
>>   mem=500
>>   maxmem=800
>>
>> Will make the guest VM boot up with 500 MB of initial RAM. It will also
>> tell it to allocate page tables entries big enough to address 800 MB of
>> RAM. So after boot up, you will be able to balloon the guest upto this
>> maxmimum of 800 MB simply using 'xm mem-set 800'.
> A quick test with 3.0.4-0 does not work on my side. I will try it again
> tomorrow with 3.0.4-1. I dunno when the patch was applied to the xen source.

Setting only the maxmem Parameter does not work in 3.0.4-1 too.

Conclusion: Currently Dynamic Memory Management and Migration does not
work together in the current stable tree.

Anybody who knows when this feature will be available to the masses?
3.0.5 i assume?

greetings,
-timo
-- 
Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731
PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc

Re: Dynamic Memory Management and Migration (3.0.4-0)

[Henning Sprang]

On 1/31/07, Timo Benk <timo.benk@gmx.de> wrote:
> Hello,
>
> i noticed the following Problem:
>
> if i want to allow dynamic memory management of Linux-domains, i need to
> set the following kernel parameter. Otherwise it is not possible, to
> increase the memory of the domain above the initial assigned value:
>
> extra = "mem=10240M"
>
> But now it is not possible to migrate the domain. If try an offline
> migration, the xm-command hangs forever.
>
> One time i tried a live migration, the domain just disappears. The
> domain was neither running on the src-domain-0 nor on the dst-domain-0,
> the domain just vanished.

As promised I am starting slowly to get bugzilla a bit sorted. As I
did not work much with memory management scenarios, I'd like to know
if this might be connected to this bug:

http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=24

Otherwise I'd close this bug as ian suggested to close everything
older than some months.

Henning

RE: Dynamic Memory Management and Migration (3.0.4-0)

[Ian Pratt]

> > if i want to allow dynamic memory management of Linux-domains, i
need to
> > set the following kernel parameter. Otherwise it is not possible, to
> > increase the memory of the domain above the initial assigned value:
> >
> > extra = "mem=10240M"
> >
> > But now it is not possible to migrate the domain. If try an offline
> > migration, the xm-command hangs forever.
> >
> > One time i tried a live migration, the domain just disappears. The
> > domain was neither running on the src-domain-0 nor on the
dst-domain-0,
> > the domain just vanished.
> 
> As promised I am starting slowly to get bugzilla a bit sorted. 

Great, thanks.

> As I did not work much with memory management scenarios, I'd like to
know
> if this might be connected to this bug:
> 
> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=24
> 
> Otherwise I'd close this bug as ian suggested to close everything
> older than some months.

I doubt this is related to Timo's problems, but I think the actual
bugzilla issue #24 is now fixed: Steve -- you checked in some lazy
memory allocation in xc_restore, right?


Ian

Can Xen run 32, 32PAE and 64-bit fully virtualized domains run side-by-side?

[Liang Yang]

Hi,

Can Xen run 32, 32PAE and 64-bit fully virtualized domains run side-by-side? 
Or all DomainUF has to be the same mode?

Thanks,

Liang

Re: Dynamic Memory Management and Migration (3.0.4-0)

[Steven Hand]

>I doubt this is related to Timo's problems, but I think the actual
>bugzilla issue #24 is now fixed: Steve -- you checked in some lazy
>memory allocation in xc_restore, right?

Yes, in -unstable cset 13386:895d873a00b47cb7b0edf3d0b6a42f47a3f4854c 
(Jan 16th). 


cheers,

S.