RE: out-of-memory problem when restoringballooneddomains

RE: out-of-memory problem when restoringballooneddomains

[Ian Pratt]

> No, unfortunately that didn't help.  (I was already running
> 2.0-testing from this morning PST, so I guess I already had most
> (all?) of the patches, but I guess it was worth a try...)
> 
> Perhaps it is not related to memory... When I try
> to restore only t1 and t3, they also fail with 'Error: 1'

The main thing to look at is /var/log/xfrd.log

Ian


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click

RE: out-of-memory problem when restoringballooneddomains

[Lars Rasmusson]

[-- Attachment #1: Type: text/plain, Size: 1266 bytes --]


I think the relevant part from the log is this.

xc_linux_restore start
Could not create domain. pfns=460800, 1843200KB
Could not create domain. pfns=460800, 1843200KB
4526 [INF] XFRD> Xfr service err=1


I've attached the log which is from a fresh reboot
where I did

# xm restore t1.saved
# xm restore t2.saved
# xm restore t3.saved
Error: 1



/Lars


On Thu, 2005-03-10 at 23:07 +0000, Ian Pratt wrote:
> > No, unfortunately that didn't help.  (I was already running
> > 2.0-testing from this morning PST, so I guess I already had most
> > (all?) of the patches, but I guess it was worth a try...)
> > 
> > Perhaps it is not related to memory... When I try
> > to restore only t1 and t3, they also fail with 'Error: 1'
> 
> The main thing to look at is /var/log/xfrd.log
> 
> Ian
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel

[-- Attachment #2: Type: text/x-log, Size: 3468 bytes --]

3958 [INF] XFRD> Accepted connection from 127.0.0.1:1062 on 2
4441 [INF] XFRD> Xfr service for 127.0.0.1:1062
[DEBUG] Conn_init> flags=1
[DEBUG] Conn_init> write stream...
[DEBUG] stream_init>mode=w flags=1 compress=0
[DEBUG] stream_init> unbuffer...
[DEBUG] stream_init< err=0
[DEBUG] Conn_init> read stream...
[DEBUG] stream_init>mode=r flags=1 compress=0
[DEBUG] stream_init> unbuffer...
[DEBUG] stream_init< err=0
[DEBUG] Conn_sxpr>
(xfr.hello 1 0)[DEBUG] Conn_sxpr< err=0
[DEBUG] Conn_sxpr>
(xfr.restore /home/testlars/t1.saved)[DEBUG] Conn_sxpr< err=0
[1110497241.580504] xc_linux_restore start

xc_linux_restore start
[1110497241.732303] Created domain 1

Created domain 1
(Domain-1 Domain-0)'domain id=1 name=t1 memory=1800 console=9601 image=/boot/vmlinuz-2.6.10-xenU'[1110497242.007297] Reloading memory pages:   0%
Reloading memory pages:   0%\b\b\b\b  5%
\b\b\b\b  5%[1110497244.989982] Received all pages

Received all pages
\b\b\b\b100%

\b\b\b\b100%
[1110497245.012152] Memory reloaded.

Memory reloaded.
Decreased reservation by 428048 pages
[1110497245.307520] Domain ready to be built.

Domain ready to be built.
[1110497245.307720] Domain ready to be unpaused

Domain ready to be unpaused
[1110497245.307750] DOM=1

DOM=1
04441 [INF] XFRD> Xfr service err=0
3958 [INF] XFRD> Forked child pid=4441
3958 [INF] XFRD> Accepted connection from 127.0.0.1:1074 on 2
4484 [INF] XFRD> Xfr service for 127.0.0.1:1074
[DEBUG] Conn_init> flags=1
[DEBUG] Conn_init> write stream...
[DEBUG] stream_init>mode=w flags=1 compress=0
[DEBUG] stream_init> unbuffer...
[DEBUG] stream_init< err=0
[DEBUG] Conn_init> read stream...
[DEBUG] stream_init>mode=r flags=1 compress=0
[DEBUG] stream_init> unbuffer...
[DEBUG] stream_init< err=0
[DEBUG] Conn_sxpr>
(xfr.hello 1 0)[DEBUG] Conn_sxpr< err=0
[DEBUG] Conn_sxpr>
(xfr.restore /home/testlars/t2.saved)[DEBUG] Conn_sxpr< err=0
[1110497254.729583] xc_linux_restore start

xc_linux_restore start
[1110497254.876723] Created domain 2

Created domain 2
(Domain-2 Domain-0 t1)'domain id=2 name=t2 memory=1700 console=9602 image=/boot/vmlinuz-2.6.10-xenU'[1110497255.152161] Reloading memory pages:   0%
Reloading memory pages:   0%\b\b\b\b  5%
\b\b\b\b  5%[1110497258.315049] Received all pages

Received all pages
\b\b\b\b100%

\b\b\b\b100%
[1110497258.337450] Memory reloaded.

Memory reloaded.
Decreased reservation by 402445 pages
[1110497258.678350] Domain ready to be built.

Domain ready to be built.
[1110497258.678534] Domain ready to be unpaused

Domain ready to be unpaused
[1110497258.684520] DOM=2

DOM=2
04484 [INF] XFRD> Xfr service err=0
3958 [INF] XFRD> Forked child pid=4484
3958 [INF] XFRD> Accepted connection from 127.0.0.1:1088 on 2
4526 [INF] XFRD> Xfr service for 127.0.0.1:1088
[DEBUG] Conn_init> flags=1
[DEBUG] Conn_init> write stream...
[DEBUG] stream_init>mode=w flags=1 compress=0
[DEBUG] stream_init> unbuffer...
[DEBUG] stream_init< err=0
[DEBUG] Conn_init> read stream...
[DEBUG] stream_init>mode=r flags=1 compress=0
[DEBUG] stream_init> unbuffer...
[DEBUG] stream_init< err=0
[DEBUG] Conn_sxpr>
(xfr.hello 1 0)[DEBUG] Conn_sxpr< err=0
[DEBUG] Conn_sxpr>
(xfr.restore /home/testlars/t3.saved)[DEBUG] Conn_sxpr< err=0
[1110497266.862455] xc_linux_restore start

xc_linux_restore start
Could not create domain. pfns=460800, 1843200KB
Could not create domain. pfns=460800, 1843200KB
4526 [INF] XFRD> Xfr service err=1

Re: out-of-memory problem when restoringballooneddomains

[Mark A. Williamson]

> xc_linux_restore start
> Could not create domain. pfns=460800, 1843200KB
> Could not create domain. pfns=460800, 1843200KB
> 4526 [INF] XFRD> Xfr service err=1

Ahhhhhh, that explains it.

tools/libxc/xc_linux_restore.c::xc_linux_restore()

restores a domain by first reserving its peak reservation, then loading the 
memory image, then returning previously ballooned pages to Xen.  You don't 
have enough memory to satisfy the peak, so it won't restore :-(

The restore code will need working to handle this case (I'm afraid I don't 
have the spare cycles to do so myself).

Cheers,
Mark

>
>
> I've attached the log which is from a fresh reboot
> where I did
>
> # xm restore t1.saved
> # xm restore t2.saved
> # xm restore t3.saved
> Error: 1
>
>
>
> /Lars
>
> On Thu, 2005-03-10 at 23:07 +0000, Ian Pratt wrote:
> > > No, unfortunately that didn't help.  (I was already running
> > > 2.0-testing from this morning PST, so I guess I already had most
> > > (all?) of the patches, but I guess it was worth a try...)
> > >
> > > Perhaps it is not related to memory... When I try
> > > to restore only t1 and t3, they also fail with 'Error: 1'
> >
> > The main thing to look at is /var/log/xfrd.log
> >
> > Ian
> >
> >
> > -------------------------------------------------------
> > SF email is sponsored by - The IT Product Guide
> > Read honest & candid reviews on hundreds of IT Products from real users.
> > Discover which products truly live up to the hype. Start reading now.
> > http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: out-of-memory problem when restoringballooneddomains

[Keir Fraser]

On 10 Mar 2005, at 23:35, Lars Rasmusson wrote:

> I think the relevant part from the log is this.
>
> xc_linux_restore start
> Could not create domain. pfns=460800, 1843200KB
> Could not create domain. pfns=460800, 1843200KB
> 4526 [INF] XFRD> Xfr service err=1

Ah! The problem is that we initially create the domain with its 
'maximum possible' memory allocation, and then automatically 'balloon 
it down' at the end of the restore operation. Of course by the third 
domain you do not have 1.8GB of headroom and so the third restore 
fails.

This probably made live relocation easier to implement, but we need to 
change this to allocate only the memory that the domain is currently 
using during each round of pre-copy migration. Getting non-live moved 
over is the first step, and then fix live relocation, I guess.

  -- Keir



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: out-of-memory problem when restoringballooneddomains

[Lars Rasmusson]

Hi, about two months ago I inquired about if xen could restore (or 
migrate) a *ballooned* domain if its unballoned size won't fit.

Example: on a 2G machine

  xm create domain1 memory=1500
  xm balloon domain1 128
  xm save domain1 saved1

  xm create domain2 memory=1500

  xm restore saved1
  *** BEEP! Xen refuses ***

Is there any news on this issue?  

I think this would be a very useful feature to have.

Even better would be if one could boot a machine giving it the impression 
pf having the whole memory space but without actually ever giving it all 
of memory, something like

  xm create domain1 memory=1500 balloon=128

if you see what I mean.

Anyway, thanks for all the cool stuff you're doing!

/Lars



On Fri, 11 Mar 2005, Keir Fraser wrote:

> 
> On 10 Mar 2005, at 23:35, Lars Rasmusson wrote:
> 
> > I think the relevant part from the log is this.
> >
> > xc_linux_restore start
> > Could not create domain. pfns=460800, 1843200KB
> > Could not create domain. pfns=460800, 1843200KB
> > 4526 [INF] XFRD> Xfr service err=1
> 
> Ah! The problem is that we initially create the domain with its 
> 'maximum possible' memory allocation, and then automatically 'balloon 
> it down' at the end of the restore operation. Of course by the third 
> domain you do not have 1.8GB of headroom and so the third restore 
> fails.
> 
> This probably made live relocation easier to implement, but we need to 
> change this to allocate only the memory that the domain is currently 
> using during each round of pre-copy migration. Getting non-live moved 
> over is the first step, and then fix live relocation, I guess.
> 
>   -- Keir
> 
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
> 

Re: out-of-memory problem when restoringballooneddomains

[Kip Macy]

If it doesn't work already it is a neccessity long term to leverage
Xen for zero downtime upgrades.  Otherwise you end up having to leave
spare capacity available to ensure that a guest can be migrated.


           -Kip


On 5/4/05, Lars Rasmusson <Lars.Rasmusson@hp.com> wrote:
> 
> Hi, about two months ago I inquired about if xen could restore (or
> migrate) a *ballooned* domain if its unballoned size won't fit.
> 
> Example: on a 2G machine
> 
>   xm create domain1 memory=1500
>   xm balloon domain1 128
>   xm save domain1 saved1
> 
>   xm create domain2 memory=1500
> 
>   xm restore saved1
>   *** BEEP! Xen refuses ***
> 
> Is there any news on this issue?
> 
> I think this would be a very useful feature to have.
> 
> Even better would be if one could boot a machine giving it the impression
> pf having the whole memory space but without actually ever giving it all
> of memory, something like
> 
>   xm create domain1 memory=1500 balloon=128
> 
> if you see what I mean.
> 
> Anyway, thanks for all the cool stuff you're doing!
> 
> /Lars
> 
> 
> On Fri, 11 Mar 2005, Keir Fraser wrote:
> 
> >
> > On 10 Mar 2005, at 23:35, Lars Rasmusson wrote:
> >
> > > I think the relevant part from the log is this.
> > >
> > > xc_linux_restore start
> > > Could not create domain. pfns=460800, 1843200KB
> > > Could not create domain. pfns=460800, 1843200KB
> > > 4526 [INF] XFRD> Xfr service err=1
> >
> > Ah! The problem is that we initially create the domain with its
> > 'maximum possible' memory allocation, and then automatically 'balloon
> > it down' at the end of the restore operation. Of course by the third
> > domain you do not have 1.8GB of headroom and so the third restore
> > fails.
> >
> > This probably made live relocation easier to implement, but we need to
> > change this to allocate only the memory that the domain is currently
> > using during each round of pre-copy migration. Getting non-live moved
> > over is the first step, and then fix live relocation, I guess.
> >
> >   -- Keir
> >
> >
> >
> > -------------------------------------------------------
> > SF email is sponsored by - The IT Product Guide
> > Read honest & candid reviews on hundreds of IT Products from real users.
> > Discover which products truly live up to the hype. Start reading now.
> > http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
> >
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: out-of-memory problem when restoringballooneddomains

[Mark Williamson]

> Even better would be if one could boot a machine giving it the impression
> pf having the whole memory space but without actually ever giving it all
> of memory, something like
>
>   xm create domain1 memory=1500 balloon=128
>
> if you see what I mean.

I'm not sure I do - could you elaborate ;-)

Cheers,
Mark

> Anyway, thanks for all the cool stuff you're doing!
>
> /Lars
>
> On Fri, 11 Mar 2005, Keir Fraser wrote:
> > On 10 Mar 2005, at 23:35, Lars Rasmusson wrote:
> > > I think the relevant part from the log is this.
> > >
> > > xc_linux_restore start
> > > Could not create domain. pfns=460800, 1843200KB
> > > Could not create domain. pfns=460800, 1843200KB
> > > 4526 [INF] XFRD> Xfr service err=1
> >
> > Ah! The problem is that we initially create the domain with its
> > 'maximum possible' memory allocation, and then automatically 'balloon
> > it down' at the end of the restore operation. Of course by the third
> > domain you do not have 1.8GB of headroom and so the third restore
> > fails.
> >
> > This probably made live relocation easier to implement, but we need to
> > change this to allocate only the memory that the domain is currently
> > using during each round of pre-copy migration. Getting non-live moved
> > over is the first step, and then fix live relocation, I guess.
> >
> >   -- Keir
> >
> >
> >
> > -------------------------------------------------------
> > SF email is sponsored by - The IT Product Guide
> > Read honest & candid reviews on hundreds of IT Products from real users.
> > Discover which products truly live up to the hype. Start reading now.
> > http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel