saving domains

saving domains

[Tim Freeman]

There have been a number of recent questions and bug reports about save and
restore (Zhiyi, Arthur, Luciano, Todd).  Were there any workarounds or
resolutions?  

I am having the same problem, 2.4.28 (2.6.10 guest), dual Xeon, gcc 3.2.3 on
RH7 and also on 2.6.10 (2.6.10 guest), dual PIII, gcc 3.3.5 on Debian Sarge.
 (just one vif/vbd on the guests)

Thanks for any information,

Tim




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: saving domains

[Arthur Bergman]

I thought it was a debian problem, but you are seeing it with RH7

My workaround in xen 2.04 is to run xfrd under strace -fF or gdb with 
set-follow-fork child mode.This makes it work for 95% of the attempts 
Ian had some suggestions that I could try but I haven't had the chance. 
It seems like some bug inside xend/xfrd with regards to timings. It 
affects both save and migrate.

Maybe we should collect what version of python and twisted we have.

Python Version: 2.3.4-19
Twisted Version: 1.3.0-6

Cheers
Arthur

-----
CTO @ Fotango Ltd
+447834716919
http://www.fotango.com/

On 9 Feb 2005, at 23:24, Tim Freeman wrote:

> There have been a number of recent questions and bug reports about 
> save and
> restore (Zhiyi, Arthur, Luciano, Todd).  Were there any workarounds or
> resolutions?
>
> I am having the same problem, 2.4.28 (2.6.10 guest), dual Xeon, gcc 
> 3.2.3 on
> RH7 and also on 2.6.10 (2.6.10 guest), dual PIII, gcc 3.3.5 on Debian 
> Sarge.
>  (just one vif/vbd on the guests)
>
> Thanks for any information,
>
> Tim
>
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real 
> users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

RE: saving domains

[Ian Pratt]

> My workaround in xen 2.04 is to run xfrd under strace -fF or gdb with 
> set-follow-fork child mode.This makes it work for 95% of the attempts 
> Ian had some suggestions that I could try but I haven't had 
> the chance. 
> It seems like some bug inside xend/xfrd with regards to timings. It 
> affects both save and migrate.
> 
> Maybe we should collect what version of python and twisted we have.
> 
> Python Version: 2.3.4-19
> Twisted Version: 1.3.0-6

That would be a very useful thing to do. We've utterly failed to
reproduce these problems here. 

The sooner we dump Twisted the better...

Ian 


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click

saving domains

[Jared Rhine]

[Ian == m+Ian.Pratt@cl.cam.ac.uk on Wed, 9 Feb 2005 23:54:46 -0000]

  Ian> We've utterly failed to reproduce [xm save] problems here.

Seems reproducible on XenCD?

  Ian> The sooner we dump Twisted the better...

Well, maybe.  While "lighter weight" xend would be good, maybe C would
be a better implementation language for that lightweight tool?  Seems
like having the fat Python xend implementation use the fat Twisted
library makes sense...

Allen Short had mentioned that some features of interest (better
daemon management, remote objects, etc) were provided by Twisted, and
by replacing SrvDaemon with a Twisted call, might actual reduce code
size and bug count.  Might raise them, too :)

-- jared@wordzoo.com

http://www.geekcode.com/geek.html
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/B
d s:++>+ a-
C++(++++)$ ULBSH++++ P+++ L+++ E++(+++) W++++ N- !o !K w !O M V
PS+++(-) PE++(--) Y+ PGP>++
t@ 5 X+ R>+ tv++>-- b>++ DI+ D- G
e++ h- r++>+++ y+++
------END GEEK CODE BLOCK------


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: saving domains

[Nivedita Singhvi]

Jared Rhine wrote:

> Well, maybe.  While "lighter weight" xend would be good, maybe C would
> be a better implementation language for that lightweight tool?  Seems
> like having the fat Python xend implementation use the fat Twisted
> library makes sense...

Jared, think you meant "little sense" ? ;)
But otherwise, I agree. Is anyone tied to python as a solution?
Wouldn't it be better to carve out a more robust alternative
in C?

thanks,
Nivedita


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

RE: saving domains

[Ian Pratt]

> > Well, maybe.  While "lighter weight" xend would be good, 
> maybe C would
> > be a better implementation language for that lightweight 
> tool?  Seems
> > like having the fat Python xend implementation use the fat Twisted
> > library makes sense...
> 
> Jared, think you meant "little sense" ? ;)
> But otherwise, I agree. Is anyone tied to python as a solution?
> Wouldn't it be better to carve out a more robust alternative
> in C?

There's nothing wrong with using python, its just Twisted that turned
out to be a bad decision. I'd wager that a daemon written in python
using language level threads would be rather smaller (in code size) and
likely more robust than one written in C. I don't really buy the
resource usage argument as a python interpreter is just over 2.5MB RSS
and the daemon shouldn't be CPU intensive.

Ian


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click

Re: saving domains

[Adam Heath]

On Wed, 9 Feb 2005, Jared Rhine wrote:

> [Ian == m+Ian.Pratt@cl.cam.ac.uk on Wed, 9 Feb 2005 23:54:46 -0000]
>
>   Ian> We've utterly failed to reproduce [xm save] problems here.
>
> Seems reproducible on XenCD?
>
>   Ian> The sooner we dump Twisted the better...
>
> Well, maybe.  While "lighter weight" xend would be good, maybe C would
> be a better implementation language for that lightweight tool?  Seems
> like having the fat Python xend implementation use the fat Twisted
> library makes sense...

Having python for xend sucks on nfs-root dom0.  Esp. if dom0 is setup with a
low memory footprint, and not much is available for cache.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: saving domains

[Mark Williamson]

> There's nothing wrong with using python, its just Twisted that turned
> out to be a bad decision. I'd wager that a daemon written in python
> using language level threads would be rather smaller (in code size)

I thought Python's support for language level threads was poor (I rather 
assumed that was one of the drivers for using Twisted in the first place).

It seems a shame to rewrite in C but OTOH with the XCS framework, perhaps it's 
possible to implement things more simply than would be the case with a 
monolithic Xend.  Also, it's worth noting that components could potentially 
be written in multiple languages...

Cheers,
Mark

> and 
> likely more robust than one written in C. I don't really buy the
> resource usage argument as a python interpreter is just over 2.5MB RSS
> and the daemon shouldn't be CPU intensive.
>
> Ian
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=Click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

RE: saving domains

[Adam Heath]

On Thu, 10 Feb 2005, Ian Pratt wrote:

> > > Well, maybe.  While "lighter weight" xend would be good,
> > maybe C would
> > > be a better implementation language for that lightweight
> > tool?  Seems
> > > like having the fat Python xend implementation use the fat Twisted
> > > library makes sense...
> >
> > Jared, think you meant "little sense" ? ;)
> > But otherwise, I agree. Is anyone tied to python as a solution?
> > Wouldn't it be better to carve out a more robust alternative
> > in C?
>
> There's nothing wrong with using python, its just Twisted that turned
> out to be a bad decision. I'd wager that a daemon written in python
> using language level threads would be rather smaller (in code size) and
> likely more robust than one written in C. I don't really buy the
> resource usage argument as a python interpreter is

Threads in python suck.  Only one thread at a time can be in the python
interpeter.



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

RE: saving domains

[Ian Pratt]

 
> Having python for xend sucks on nfs-root dom0.  Esp. if dom0 
> is setup with a
> low memory footprint, and not much is available for cache.

A lot of the current daemon's size is down to Twisted, which won't be
featuring in the new tools. 

As regards your dom0 NFS root, have you investigated cachefs? I've heard
that in the last few weeks its become stable and works well.

Ian


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click

saving domains

[Jared Rhine]

[Nivedita == niv@us.ibm.com on Wed, 09 Feb 2005 17:35:50 -0800]

  Jared> While "lighter weight" xend would be good, maybe C would be a
  Jared> better implementation language for that lightweight tool?
  Jared> Seems like having the fat Python xend implementation use the
  Jared> fat Twisted library makes sense...

  Nivedita> Wouldn't it be better to carve out a more robust
  Nivedita> alternative in C?

Not better, just different.

By definition, the Python codebase will be a whole lot more "robust"
in terms of features up until the point that thousands of lines of
code exist is a C-based replacement.

Make no mistake, we absolutely need a C-based daemon which can be used
on leaner dom0s where Python is not installed.

But I definitely think the existing xend codebase is very valuable,
should retain the xend name, be kept in the tree, and available for
those who like its approach and extensibility.  People will add
features if they want them.

It seems likely that xend will always be a more featureful alternative
to the once-and-future C-based control daemon.  The same people who
want a lightweight dom0 are going to want small, secure, and
well-controlled feature sets in that codebase.

-- jared@wordzoo.com

"We suffer primarily not from our vices or our weaknesses, but from our
 illusions.  We are haunted, not by reality, but by those images we have put
 in place of reality." - Daniel J. Boorstin


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

RE: saving domains

[Ian Pratt]

 
> Threads in python suck.  Only one thread at a time can be in 
> the python interpeter.

...which is just fine for something that isn't compute intensive.

The new tools will be rather more modular, comprising multiple daemons,
some in C, some in python (at least initially). 

Ian


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click

saving domains

[Jared Rhine]

[Ian == m+Ian.Pratt@cl.cam.ac.uk on Thu, 10 Feb 2005 01:51:50 -0000]

  Ian> I don't really buy the resource usage argument as a python
  Ian> interpreter is just over 2.5MB RSS and the daemon shouldn't be
  Ian> CPU intensive.

I believe that those concerned about the Python dependency are more
concerned about the disk space and "fatness" of having python
installed at all.  I know I could cook a version of XenCD which fit
into 256Mb or less of RAM if the rootfs didn't have to have Python
installed.  Others are thinking way way smaller than that.

  Mark> It seems a shame to rewrite in C but OTOH with the XCS
  Mark> framework...

Yes, exactly.  Eventually, someone is going to get annoyed enough at
the Python dependency to port x2d2 to xcs or write a lightweight
replacement.  I'd like to see us standardize around xcs, and have a
healthy community of competing control tools.

Last time this came up, however, we got into intractable arguments
about TCP vs UDP  for xcs and whether layering daemons was advised.

Those contributing working code will rightly determine where this all
ends up.

-- jared@wordzoo.com

http://www.geekcode.com/geek.html
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/B
d s:++>+ a-
C++(++++)$ ULBSH++++ P+++ L+++ E++(+++) W++++ N- !o !K w !O M V
PS+++(-) PE++(--) Y+ PGP>++
t@ 5 X+ R>+ tv++>-- b>++ DI+ D- G
e++ h- r++>+++ y+++
------END GEEK CODE BLOCK------


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: saving domains

[Tim Freeman]

On Wed, 9 Feb 2005 23:30:06 +0000
Arthur Bergman <abergman@fotango.com> wrote:

> I thought it was a debian problem, but you are seeing it with RH7
> 
> My workaround in xen 2.04 is to run xfrd under strace -fF or gdb with 
> set-follow-fork child mode.This makes it work for 95% of the attempts 
> Ian had some suggestions that I could try but I haven't had the chance. 
> It seems like some bug inside xend/xfrd with regards to timings. It 
> affects both save and migrate.
> 
> Maybe we should collect what version of python and twisted we have.
> 
> Python Version: 2.3.4-19
> Twisted Version: 1.3.0-6

Hi, 

Both use Python 2.3.4 and Twisted 1.3.0rc1

Are you using an SMP host?  I would like to reboot into nosmp and try that,
but I can't right now (no access to the pxe grub entry).   I hear "timings"
and think this may be a good thing to check out (?).

Tim



> 
> Cheers
> Arthur
> 
> -----
> CTO @ Fotango Ltd
> +447834716919
> http://www.fotango.com/
> 
> On 9 Feb 2005, at 23:24, Tim Freeman wrote:
> 
> > There have been a number of recent questions and bug reports about 
> > save and
> > restore (Zhiyi, Arthur, Luciano, Todd).  Were there any workarounds or
> > resolutions?
> >
> > I am having the same problem, 2.4.28 (2.6.10 guest), dual Xeon, gcc 
> > 3.2.3 on
> > RH7 and also on 2.6.10 (2.6.10 guest), dual PIII, gcc 3.3.5 on Debian 
> > Sarge.
> >  (just one vif/vbd on the guests)
> >
> > Thanks for any information,
> >
> > Tim
> >
> >
> >
> >
> > -------------------------------------------------------
> > SF email is sponsored by - The IT Product Guide
> > Read honest & candid reviews on hundreds of IT Products from real 
> > users.
> > Discover which products truly live up to the hype. Start reading now.
> > http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
> 


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: saving domains

[Arthur Bergman]

killall xfrd
screen strace -fF xfrd

:-)

Yes once xm save works, xm migrate works as well.

Cheers
Arthur

-----
CTO @ Fotango Ltd
+447834716919
http://www.fotango.com/

On 9 Feb 2005, at 23:58, Amitabh Tamhane wrote:

> I am still having errors on xm save and/or xm migrate
> I think xm migrate would work, once xm save starts working.
>
> I didn't quite understand .... "what you mean by run xfrd under strace 
> -fF" ?
> I know strace utility, but isn't it that the xfrd is started by xend 
> itself for processing commands for save?
>
> I am using xen2.03 on SuSE 9.0 and i installed twisted using the make 
> install-twisted utility.
> python version is: 2.3.3
>
> I am clueless about this xm save error.
>
> Please do keep posted any solutions to this problem.
>
> Thanks,
>
> Amitabh



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: saving domains

[Arthur Bergman]

Hmmm...

Pretty much the same. I have hyperthreading enabled, but just one 
processor in the box. Let me try today to restart with nosmp and see if 
it does anything.

Cheers
Arthur

-----
CTO @ Fotango Ltd
+447834716919
http://www.fotango.com/

On 10 Feb 2005, at 02:27, Tim Freeman wrote:

>
> Hi,
>
> Both use Python 2.3.4 and Twisted 1.3.0rc1
>
> Are you using an SMP host?  I would like to reboot into nosmp and try 
> that,
> but I can't right now (no access to the pxe grub entry).   I hear 
> "timings"
> and think this may be a good thing to check out (?).
>
> Tim
>
>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: saving domains

[Zhiyi Huang]

I have tried the following but still no luck:(

I am using python 2.3.4 and twisted 1.3.0rc1 (same as Tim's), by the way.

> killall xfrd
> screen strace -fF xfrd
> 
> :-)
> 
> Yes once xm save works, xm migrate works as well.
> 
> Cheers
> Arthur
 

--------------------------
Zhiyi from EPFL Switzerland



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: saving domains

[Arthur Bergman]

Have you tried it under xen 2.0.4?

Cheers
Arthur

-----
CTO @ Fotango Ltd
+447834716919
http://www.fotango.com/

On 10 Feb 2005, at 09:07, Zhiyi Huang wrote:

> I have tried the following but still no luck:(
>
> I am using python 2.3.4 and twisted 1.3.0rc1 (same as Tim's), by the 
> way.
>
>> killall xfrd
>> screen strace -fF xfrd
>>
>> :-)
>>
>> Yes once xm save works, xm migrate works as well.
>>
>> Cheers
>> Arthur
>
>
> --------------------------
> Zhiyi from EPFL Switzerland
>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: saving domains

[tdc]

> Maybe we should collect what version of python and twisted we have.
> 
> Python Version: 2.3.4-19
> Twisted Version: 1.3.0-6
> 
> Cheers
> Arthur

Save/Migrate not working with Xen 2.0-2.0.3 on recent gentoo:

GCC 3.4.2-r2, ssp-3.4.1-1, pie-8.7.6.5
Twisted 1.3.0rc1
Python 2.3.4

I'll checkout 2.0.4 soon, but i don't expect it'll work

Dave




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

RE: saving domains

[Ronald G. Minnich]

On Wed, 9 Feb 2005, Ian Pratt wrote:

> The sooner we dump Twisted the better...

yeah!

ron


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

RE: saving domains

[Ronald G. Minnich]

On Thu, 10 Feb 2005, Ian Pratt wrote:

> I don't really buy the resource usage argument as a python interpreter
> is just over 2.5MB RSS and the daemon shouldn't be CPU intensive.

hmm. 

The issue is that in a cluster, you've got to make all those python
libraries available, and almost any way you do it is a headache, when
multiplied by 1024 nodes. Maybe, however, if the design in python is
straightforward enough, a C version won't be too bad to write.

I've done an awful lot of these "I/O mux" things in C, Python, Perl, ...  
while the interpretive languages always seem the best bet at first, issues
of diagnosis and fault tolerance have always led me back to C. Twisted is 
not the first such framework of its kind, it won't be the last, and they 
always lead to heartbreak. 

Python has other problems as well. Version skew issues are awful. We had a 
programming environment here for scientific users that would not behave 
correctly from mac->linux version, or even mac->mac versions! Hmm, isn't 
one of the arguments for Python a degree of portability? 

ron


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click