libxl - API call to return sxpr of a domain?

libxl - API call to return sxpr of a domain?

[Shriram Rajagopalan]

[-- Attachment #1.1: Type: text/plain, Size: 314 bytes --]

I am looking into adding Remus support for libxl. The easiest way is
to obtain the domain's sxpr, so that the rest of Remus python code stays as
is.

Is there an api call in libxl to return a domain's sxpr ? a grep on the
libxl code
base returned nothing. Or am I missing something pretty obvious?

thanks
shriram

[-- Attachment #1.2: Type: text/html, Size: 410 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: libxl - API call to return sxpr of a domain?

[Ian Campbell]

On Tue, 2011-06-07 at 04:30 +0100, Shriram Rajagopalan wrote:
> I am looking into adding Remus support for libxl. The easiest way is
> to obtain the domain's sxpr, so that the rest of Remus python code
> stays as is.
>  
> Is there an api call in libxl to return a domain's sxpr ? a grep on
> the libxl code
> base returned nothing. Or am I missing something pretty obvious?

xl has some code to do this but libxl doesn't. An sxpr representation of
a domain is rather a xend specific concept which is the only reason xl
has it.

There are some plans to allow libxl to generate json for any of the IDL
defined datastructures, mostly as a convenient pretty-printer but being
machine parsable is a handy side-effect. Currently this would just be
for individual datastructures though.

Where/how does remus use sxp? tools/python/xen/remus/vm.py:domtosxpr()
seems to consume a xend datastructure and make a Remus sxp out of it --
can an xl equivalent not be written using the python bindings? (NB
bindings may be incomplete, we can fix up as you discover stuff). Are
all usages of sxp in Remus of that particular sxp format or are there
others?

Are sxp's baked into the Remus wire-protocol? Personally I think moving
away from using SXPs internally towards using actual data structures
would be a good idea...

BTW, is tools/python/xen/remus part of xend or part of Remus? Or does it
straddle the boundary?

Ian.

Re: libxl - API call to return sxpr of a domain?

[Shriram Rajagopalan]

[-- Attachment #1.1: Type: text/plain, Size: 2949 bytes --]

On Tue, Jun 7, 2011 at 5:02 AM, Ian Campbell <Ian.Campbell@citrix.com>wrote:

> On Tue, 2011-06-07 at 04:30 +0100, Shriram Rajagopalan wrote:
> > I am looking into adding Remus support for libxl. The easiest way is
> > to obtain the domain's sxpr, so that the rest of Remus python code
> > stays as is.
> >
> > Is there an api call in libxl to return a domain's sxpr ? a grep on
> > the libxl code
> > base returned nothing. Or am I missing something pretty obvious?
>
> xl has some code to do this but libxl doesn't. An sxpr representation of
> a domain is rather a xend specific concept which is the only reason xl
> has it.
>
There are some plans to allow libxl to generate json for any of the IDL
> defined datastructures, mostly as a convenient pretty-printer but being
> machine parsable is a handy side-effect. Currently this would just be
> for individual datastructures though.
>
> Where/how does remus use sxp? tools/python/xen/remus/vm.py:domtosxpr()
> seems to consume a xend datastructure and make a Remus sxp out of it --
> can an xl equivalent not be written using the python bindings? (NB
> bindings may be incomplete, we can fix up as you discover stuff). Are
> all usages of sxp in Remus of that particular sxp format or are there
> others?
>
>
The only reason remus uses sxpr is because xend conveys info in that form.
Basically, it only needs the vif device name (vif1.0, etc), the disk device
name and
the access format (tap/drbd) for proper operation.

 save.py:MigrationSocket - establish connection with remote machine's xend
daemon
 vm.VM(domid): get sxpr (dominfo) from xend via xml rpc call.
 save.py:Saver - image.makeheader(dominfo) - serialize the sxpr and send to
remote machine.

everything else in vm.py is to parse the sxpr to extract the vif and disk
info.
 self.disks = getdisks(self.dom)
 self.vifs = getvifs(self.dom)

Are sxp's baked into the Remus wire-protocol?

Remus wire-protocol is whatever protocol xend requires.

The reason for bypassing the usual xend live migration code path is because
of the
callbacks, the checkpoint interval based suspend/resume, etc. Now that I
know that
xl/libxl doesnt use sxpr in its wire-protocol (dunce! :( ), the plan would
have to be different.

(a) Follow the same implementation style like that with xend (bypass xl's
live
migration mechanism) - involves some code duplication probably for
communicating with
remote machine, in xl's wire protocol. The advantage is most of remus'
python code (save.py,
 device.py, qdisc.py, code to install/parse IFB devices, tc rules, etc)
stays as is.

(b) integrate the remus control flow into xl/libxl stack - I dont know how
much work that would be
yet.

> Personally I think moving
> away from using SXPs internally towards using actual data structures
> would be a good idea...
>
>

> BTW, is tools/python/xen/remus part of xend or part of Remus? Or does it
> straddle the boundary?
>
> part of Remus stack.

> Ian.
>
> shriram

[-- Attachment #1.2: Type: text/html, Size: 4323 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: libxl - API call to return sxpr of a domain?

[Ian Campbell]

On Tue, 2011-06-07 at 16:30 +0100, Shriram Rajagopalan wrote:
> On Tue, Jun 7, 2011 at 5:02 AM, Ian Campbell <Ian.Campbell@citrix.com>
> wrote:
>         
>         On Tue, 2011-06-07 at 04:30 +0100, Shriram Rajagopalan wrote:
>         > I am looking into adding Remus support for libxl. The
>         easiest way is
>         > to obtain the domain's sxpr, so that the rest of Remus
>         python code
>         > stays as is.
>         >
>         > Is there an api call in libxl to return a domain's sxpr ? a
>         grep on
>         > the libxl code
>         > base returned nothing. Or am I missing something pretty
>         obvious?
>         
>         
>         xl has some code to do this but libxl doesn't. An sxpr
>         representation of
>         a domain is rather a xend specific concept which is the only
>         reason xl
>         has it. 
>         There are some plans to allow libxl to generate json for any
>         of the IDL
>         defined datastructures, mostly as a convenient pretty-printer
>         but being
>         machine parsable is a handy side-effect. Currently this would
>         just be
>         for individual datastructures though.
>         
>         Where/how does remus use sxp?
>         tools/python/xen/remus/vm.py:domtosxpr()
>         seems to consume a xend datastructure and make a Remus sxp out
>         of it --
>         can an xl equivalent not be written using the python bindings?
>         (NB
>         bindings may be incomplete, we can fix up as you discover
>         stuff). Are
>         all usages of sxp in Remus of that particular sxp format or
>         are there
>         others?
>         
> 
> The only reason remus uses sxpr is because xend conveys info in that
> form. Basically, it only needs the vif device name (vif1.0, etc), the
> disk device name and the access format (tap/drbd) for proper
> operation. 

ok, this stuff should be available to xl/libxl (as appropriate) pretty
easily.

[...]
> The reason for bypassing the usual xend live migration code path is
> because of the  callbacks, the checkpoint interval based
> suspend/resume, etc. Now that I know that xl/libxl doesnt use sxpr in
> its wire-protocol (dunce! :( ), the plan would have to be different.
> 
> (a) Follow the same implementation style like that with xend (bypass
> xl's live migration mechanism) - involves some code duplication
> probably for communicating with remote machine, in xl's wire protocol.
> The advantage is most of remus' python code (save.py,  device.py,
> qdisc.py, code to install/parse IFB devices, tc rules, etc) stays as
> is.
> 
> (b) integrate the remus control flow into xl/libxl stack - I dont know
> how much work that would be yet.

I don't know enough about the needs etc of Remus to make much in the way
of concrete proposals but in general plan b is the sort of thing we
would prefer since all toolstacks can then benefit (at least to some
extent).

Certainly I would prefer to see libxl functions which provide the
necessary interfaces (likely sharing common code within the library) etc
to duplication of the code.

Perhaps you could quickly explain the Remus architecture within the xend
world, which might help us to advise. e.g. How are things different on
the tx and rx sides with and without Remus? What additional callbacks
and control flow are there etc?

Do I gather correctly that the thing on the receiving end is not xend
but rather a Remus process?

Ian.

Re: libxl - API call to return sxpr of a domain?

[Shriram Rajagopalan]

[-- Attachment #1.1: Type: text/plain, Size: 7043 bytes --]

On Tue, Jun 7, 2011 at 12:16 PM, Ian Campbell <Ian.Campbell@eu.citrix.com>wrote:

> On Tue, 2011-06-07 at 16:30 +0100, Shriram Rajagopalan wrote:
> > On Tue, Jun 7, 2011 at 5:02 AM, Ian Campbell <Ian.Campbell@citrix.com>
> > wrote:
> >
> >         On Tue, 2011-06-07 at 04:30 +0100, Shriram Rajagopalan wrote:
> >         > I am looking into adding Remus support for libxl. The
> >         easiest way is
> >         > to obtain the domain's sxpr, so that the rest of Remus
> >         python code
> >         > stays as is.
> >         >
> >         > Is there an api call in libxl to return a domain's sxpr ? a
> >         grep on
> >         > the libxl code
> >         > base returned nothing. Or am I missing something pretty
> >         obvious?
> >
> >
> >         xl has some code to do this but libxl doesn't. An sxpr
> >         representation of
> >         a domain is rather a xend specific concept which is the only
> >         reason xl
> >         has it.
> >         There are some plans to allow libxl to generate json for any
> >         of the IDL
> >         defined datastructures, mostly as a convenient pretty-printer
> >         but being
> >         machine parsable is a handy side-effect. Currently this would
> >         just be
> >         for individual datastructures though.
> >
> >         Where/how does remus use sxp?
> >         tools/python/xen/remus/vm.py:domtosxpr()
> >         seems to consume a xend datastructure and make a Remus sxp out
> >         of it --
> >         can an xl equivalent not be written using the python bindings?
> >         (NB
> >         bindings may be incomplete, we can fix up as you discover
> >         stuff). Are
> >         all usages of sxp in Remus of that particular sxp format or
> >         are there
> >         others?
> >
> >
> > The only reason remus uses sxpr is because xend conveys info in that
> > form. Basically, it only needs the vif device name (vif1.0, etc), the
> > disk device name and the access format (tap/drbd) for proper
> > operation.
>
> ok, this stuff should be available to xl/libxl (as appropriate) pretty
> easily.
>
> [...]
> > The reason for bypassing the usual xend live migration code path is
> > because of the  callbacks, the checkpoint interval based
> > suspend/resume, etc. Now that I know that xl/libxl doesnt use sxpr in
> > its wire-protocol (dunce! :( ), the plan would have to be different.
> >
> > (a) Follow the same implementation style like that with xend (bypass
> > xl's live migration mechanism) - involves some code duplication
> > probably for communicating with remote machine, in xl's wire protocol.
> > The advantage is most of remus' python code (save.py,  device.py,
> > qdisc.py, code to install/parse IFB devices, tc rules, etc) stays as
> > is.
> >
> > (b) integrate the remus control flow into xl/libxl stack - I dont know
> > how much work that would be yet.
>
> I don't know enough about the needs etc of Remus to make much in the way
> of concrete proposals but in general plan b is the sort of thing we
> would prefer since all toolstacks can then benefit (at least to some
> extent).
>
> Certainly I would prefer to see libxl functions which provide the
> necessary interfaces (likely sharing common code within the library) etc
> to duplication of the code.
>
> Perhaps you could quickly explain the Remus architecture within the xend
> world, which might help us to advise. e.g. How are things different on
> the tx and rx sides with and without Remus? What additional callbacks
> and control flow are there etc?
>
> Do I gather correctly that the thing on the receiving end is not xend
> but rather a Remus process?
>
> On the receiving end, there is "no" Remus receiver process.
Well, there are some remus related patches, that have long been integrated
into xc_domain_restore, but apart from that, everything else is as-is.

The only remus specific part on rx side, is the blktap2 userspace driver
(block-remus),
which again gets activated by usual Xend control flow (as it tries to create
a tap device).
But I dont think this needs special treatment as long as xl can parse/accept
spec like
 tap:remus:backupHost:port|aio:/dev/foo (or tap2:remus:.. ).
and launch the appropriate blktap2 backend driver (this system is already in
place, afaik).

The bulk of Remus transmission data is in libxc and hence is agnostic to
both
xend/xl. It basically prolongs the last iteration for eternity. It supplies
a callback
handler for checkpoint, which adds the "wait" time before the next suspend
(e.g., suspend
every 50ms). In case of Xend, the checkpoint handler is not supplied and
hence the domain
is suspended as soon as the previous iteration finishes.

(a) On the sending side, without Remus, Xend control flow is as follows:
   xm migrate --live <domain> <host>
     (i) XendCheckpoint:save [which writes the signature record, sxp to the
socket]
          and issues "xc_save <params>"
     (ii) xc_save calls xc_domain_save with appropriate callback handlers
for suspend
            & switch_qemu_logdirty only. These handlers are in
libxc/xcutils/xc_save.c.
     (iv) xc_domain_save:
              send dirty pages for max_iters
              if (last_iter) suspend_callback()
              send final set of dirty pages
              send tailbuf data

The callback structure has two other handlers (postcopy aka postresume,
checkpoint) that
is used by Remus.
*************************
(b) On sending side, with Remus
      remus <domain> <host>
         (i) tools/remus/remus:
            - calls tools/python/xen/remus/vm.py:VM(domid)
            - vm.py:VM issues xmlrpc call to Xend to obtain domid's sxpr and
extract out the disk/vif info.
         (ii) create the "buffers" for disk & vif.
         (iii) Connect with remote host's Xend socket and send the sxp info.
[same as (i) for non Remus case]

          (iv) tools/python/xen/remus/save.py:Saver uses libcheckpoint to
initiate checkpointing.
                tools/python/xen/lowlevel/checkpoint: has suspend/resume
handlers similar to xc_save.c
                trampoline functions to bounce the callbacks for suspend,
postcopy and checkpoint to their
                python equivalents.


tools/python/xen/lowlevel/checkpoint/libcheckpoint.c:checkpoint_start calls
xc_domain_save with
                 all needed callback handlers.
                     ---> functionally equivalent to (ii) in non-Remus case.
           (v) xc_domain_save: (after the initial iterations)
              copypages:
               send dirty pages & tailbuf data
               postcopy_callback() [resumes domain]
               checkpoint_callback()
                   netbuffer_checkpoint() [python - communicates via netlink
to sch_plug]
                   diskbuffer_checkpoint() [python - communicates via fifo
to block-remus]
                   sleep(50ms) [or whatever checkpoint interval]
                   return
                suspend_callback()
                goto copypages

Hope that explains the control flow.

shriram

> Ian.
>
>

[-- Attachment #1.2: Type: text/html, Size: 8520 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: libxl - API call to return sxpr of a domain?

[Ian Campbell]

[-- Attachment #1: Type: text/plain, Size: 6619 bytes --]

On Wed, 2011-06-08 at 16:55 +0100, Shriram Rajagopalan wrote:

> On the receiving end, there is "no" Remus receiver process.
> Well, there are some remus related patches, that have long been
> integrated into xc_domain_restore, but apart from that, everything
> else is as-is.

OK.

> The only remus specific part on rx side, is the blktap2 userspace
> driver (block-remus), which again gets activated by usual Xend control
> flow (as it tries to create a tap device). But I dont think this needs
> special treatment as long as xl can parse/accept spec like
> tap:remus:backupHost:port|aio:/dev/foo (or tap2:remus:.. ).
> and launch the appropriate blktap2 backend driver (this system is
> already in place, afaik).

Hmm. Please see docs/misc/xl-disk-configuration.txt for the
configuration syntax understood by xl. Also note that IanJ has a series
outstanding which improves the syntax, including compat with xend
syntaxes and makes it more extensible for the future. The series
includes an updated version of the doc, you'd be better off reading the
new version than what is currently in the tree. A pre-patched version is
attached.

It doesn't currently support "remus:" and the "foo:" prefixes are in
general deprecated. It looks like "remus:" will fall into the category
of things which are supported via the script= directive. We've also
grandfathered some "foo:" prefixes as shorthand for the script syntax
(this is also how xend implemented them), so I think this will continue
to work (assuming calling a script is how this works in xend, if not
such a script might be needed).

The "foo|bar" syntax is completely new to me (and I suspect anyone else
not familiar with remus). How does it work? Is the full
"backupHost:port|aio:/dev/foo" considered the argument to Remus (in
which case I think it can work via the Remus script as above) or does
xend somehow parse this into "remus:backupHost:port" and "aio:/dev/foo"?
In the latter case I've no idea what to suggest!

Have you considered making Remus a more top-level domain configuration
option rather than disk specific? i.e. adding remus_backup = "..." to
the cfg. This would allow libxl to do the right thing internally and
setup the disks in the right way etc etc.

Doing The Right thing is something we are striving towards with libxl,
especially with disk config which is unnecessarily complex for users.

e.g. it should not be necessary for a user to specifically ask for tap
or phy etc, rather they should present the path to the thing and libxl
should figure out if blkback or blktap is needed. For example if Remus
were enabled then it should DTRT and always select blktap even if
blkback is otherwise suitable.

> The bulk of Remus transmission data is in libxc and hence is agnostic
> to both xend/xl. It basically prolongs the last iteration for
> eternity. It supplies a callback handler for checkpoint, which adds
> the "wait" time before the next suspend (e.g., suspend every 50ms). In
> case of Xend, the checkpoint handler is not supplied and hence the
> domain is suspended as soon as the previous iteration finishes.

I think exposing such a callback is within the scope of the libxl API.
For example libxl_domain_checkpoint(...., callback) and
libxl_domain_suspend(...) can probably backend onto the same internal
function.

Another option to the callbacks might be to integrate with the libxl
event handling mechanism. Note that IanJ wants to overhall this from its
current state. I'm less sure whether this would make sense.

> 
> (a) On the sending side, without Remus, Xend control flow is as
> follows:
[...]

looks mostly the same as xl, except xl does all the xc_domain_save stuff
in process rather than indirecting via an external binary. Also xl has
to take care of starting a receiver process on the other end and has a
bit more of a protocol interlock surrounding the actual migration to try
and ensure the other end really is ready and hasn't failed etc.

> The callback structure has two other handlers (postcopy aka
> postresume, checkpoint) that
> is used by Remus.
> *************************
> (b) On sending side, with Remus
>       remus <domain> <host>

I suppose here there is a choice between adding libxl/xl support to this
remus binary or implementing "xl remus <domain> <host>".

>          (i) tools/remus/remus:
>             - calls tools/python/xen/remus/vm.py:VM(domid)
>             - vm.py:VM issues xmlrpc call to Xend to obtain domid's
> sxpr and extract out the disk/vif info.

Could be done via the libxl python bindings in the xl case?

>          (ii) create the "buffers" for disk & vif.

Stays the same, I guess, if you stick with the remus tool.

>          (iii) Connect with remote host's Xend socket and send the sxp
> info. [same as (i) for non Remus case]

Hrm, this would involve duplicating a bunch of xl functionality to start
the receiver, and run the xl protocol etc.

That rather suggests that at least this bit should be in xl itself
rather than remus. This needn't necessarily involve putting everything
in xl, but just forking xl for this bit.

>           (iv) tools/python/xen/remus/save.py:Saver uses libcheckpoint
> to initiate checkpointing.
>                 tools/python/xen/lowlevel/checkpoint: has
> suspend/resume handlers similar to xc_save.c
>                 trampoline functions to bounce the callbacks for
> suspend, postcopy and checkpoint to their
>                 python equivalents.

I think either handling these inside libxl or bouncing them to the
caller (depending on the nature of the callback) would be reasonable.

>                 
> 
> tools/python/xen/lowlevel/checkpoint/libcheckpoint.c:checkpoint_start
> calls xc_domain_save with
>                  all needed callback handlers.
>                      ---> functionally equivalent to (ii) in non-Remus
> case.
>            (v) xc_domain_save: (after the initial iterations)
>               copypages:
>                send dirty pages & tailbuf data
>                postcopy_callback() [resumes domain]
>                checkpoint_callback()
>                    netbuffer_checkpoint() [python - communicates via
> netlink to sch_plug]
>                    diskbuffer_checkpoint() [python - communicates via
> fifo to block-remus]
>                    sleep(50ms) [or whatever checkpoint interval]
>                    return
>                 suspend_callback()
>                 goto copypages
> 
> Hope that explains the control flow.

I think so. Thanks.

Hopefully some of the suggestions even make sense and demonstrate my new
understanding ;-)

> 
> shriram
> 
>         Ian.
>         
> 


[-- Attachment #2: xl-disk-configuration.txt --]
[-- Type: text/plain, Size: 6589 bytes --]

                     ---------------------
                     XL DISK CONFIGURATION
                     ---------------------

This document specifies the xl config file format disk configuration
option.  It has the following form:

    disk = [ '<diskspec>', '<diskspec>', ... ]

where each diskspec is in this form:

   [<key>=<value>|<flag>,]*,
     [<target>, [<format>, [<vdev>, [<access>]]]],
     [<key>=<value>|<flag>,]*
     [target=<target>]


For example, these strings are equivalent:

  /dev/vg/guest-volume,,hda
  /dev/vg/guest-volume,raw,hda,rw
  format=raw, vdev=hda, access=rw, target=/dev/vg/guest-volume
  raw:/dev/vg/guest-volume,hda,w   (deprecated, see below)

As are these:

  /root/image.iso,,hdc,cdrom
  /root/image.iso,,hdc,,cdrom
  /root/image.iso,raw,hdc,devtype=cdrom
  format=raw, vdev=hdc, access=ro, devtype=cdrom, target=/root/image.iso
  raw:/root/image.iso,hdc:cdrom,ro   (deprecated, see below)

These might be specified in the domain config file like this:

  disk = [ '/dev/vg/guest-volume,,hda', '/root/image.iso,,hdc,cdrom' ]


More formally, the string is a series of comma-separated keyword/value
pairs, flags and positional parameters.  Parameters which are not bare
keywords and which do not contain "=" symbols are assigned to the
so-far-unspecified positional parameters, in the order below.  The
positional parameters may also be specified explicitly by name.

Each parameter may be specified at most once, either as a positional
parameter or a named parameter.  Default values apply if the parameter
is not specified, or if it is specified with an empty value (whether
positionally or explicitly).

Whitespace may appear before each parameter and will be ignored.


=====================
POSITIONAL PARAMETERS
=====================

target
------

Description:           Block device or image file path.  When this is
                       used as a path, /dev will be prepended
                       if the path doesn't start with a '/'.
Supported values:      N/A
Deprecated values:     N/A
Default value:         None.  While a path is provided in most cases
                       there is an exception: for a cdrom device, lack
                       of this attribute would imply an empty cdrom
                       drive.

Special syntax:

   When this parameter is specified by name, ie with the "target="
   syntax in the configuration file, it consumes the whole rest of the
   <diskspec>.  Therefore in that case it must come last.  This is
   permissible even if an empty value for the target was already
   specifed as a positional parameter.  This is the only way to
   specify a target string containing metacharacters such as commas
   and (in some cases) colons, which would otherwise be
   misinterpreted.

   Future parameter and flag names will start with an ascii letter and
   contain only ascii alphanumerics, hyphens and underscores, and will
   not be legal as vdevs.  Targets which might match that syntax
   should not be specified as positional parameters.


format
------

Description:           Specifies the format of image file.
Supported values:      raw, qcow, qcow2, vhd
Deprecated values:     None
Default value:         raw


vdev
----

Description:           Virtual device as seen by the guest (also
                       referred to as guest drive designation in some
                       specifications).  See docs/misc/vbd-interface.txt.
Supported values:      hd[x], xvd[x], sd[x] etc.  Please refer to the
                       above specification for further details.
Deprecated values:     None
Default Value:         None, this parameter is mandatory.


access
-------

Description:           Specified access control information.  Whether
                       or not the block device is provided to the
                       guest in read-only or read-write mode depends
                       on this attribute.
Supported values:      ro, r   (specifies read-only)
                       rw, w   (specifies read/write)
Deprecated values:     None
Default value:         rw
                       unless devtype=cdrom, in which case r



==========================
OTHER PARAMETERS AND FLAGS
==========================


devtype=<devtype>
-----------------

Description:           Qualifies virtual device type.
Supported values:      cdrom
Deprecated values:     None
Mandatory:             No


cdrom
-----

Convenience alias for "devtype=cdrom".


script=<script>
---------------

Specifies that <target> is not a normal host path, but rather
information to be interpreted by /etc/xen/scripts/block-<script>.



==================================
DEPRECATED PARAMETERS AND PREFIXES
==================================

Deprecated values are acceptable and are intended work compatibly with
xend and xl from xen 4.1.  In future they may print a warning.
Support for deprecated parameters and syntaxes are likely to be
dropped in future versions of xl.

There is also support for a deprecated old syntax for <diskspec>:

  [<format>:][<target>],<vdev>[:<devtype>],<access>   (deprecated)

This syntax also supports deprecated prefixes, described below.  These
are found prepended to the format parameter - eg "tap:aio:qcow:".


<format>:
---------

Description:           Specifies the format (deprecated)
Supported values:      raw:  qcow2:  vhd:

In xend and old versions of libxl it was necessary to specify the
format with a prefix.  For compatibility, these three prefixes are
recognised as specifying the corresponding format.  They are
equivalent to "format=<format>" or the specification of <format>
(without a colon) as a positional parameter.


<script>:
---------

Description:           Specifies the script (deprecated)
Supported values:      iscsi:  nbd:  enbd:  drbd:

In xend and old versions of libxl it was necessary to specify the
"script" (see above) with a prefix.  For compatibility, these four
prefixes are recognised as specifying the corresponding script.  They
are equivalent to "script=<script>".


<deprecated-prefix>:
--------------------

Description;          Deprecated prefix, ignored
Supported values:     tapdisk:  tap2:  aio:  ioemu:  file:  phy:

Various prefixes were required by xend and older versions of libxl to
make the block devices work.  In some cases these options would
override the backend type, but in other cases they would be ignored in
favour of "making it work"; in yet other cases it would be necessary
to specify several of these, for example:
  "tap:aio:/some/path..."

All of these prefixes are now stripped and ignored.

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: libxl - API call to return sxpr of a domain?

[Shriram Rajagopalan]

[-- Attachment #1.1: Type: text/plain, Size: 7894 bytes --]

On Thu, Jun 9, 2011 at 7:34 AM, Ian Campbell <Ian.Campbell@eu.citrix.com>wrote:

> On Wed, 2011-06-08 at 16:55 +0100, Shriram Rajagopalan wrote:
>
> > On the receiving end, there is "no" Remus receiver process.
> > Well, there are some remus related patches, that have long been
> > integrated into xc_domain_restore, but apart from that, everything
> > else is as-is.
>
> OK.
>
> > The only remus specific part on rx side, is the blktap2 userspace
> > driver (block-remus), which again gets activated by usual Xend control
> > flow (as it tries to create a tap device). But I dont think this needs
> > special treatment as long as xl can parse/accept spec like
> > tap:remus:backupHost:port|aio:/dev/foo (or tap2:remus:.. ).
> > and launch the appropriate blktap2 backend driver (this system is
> > already in place, afaik).
>
> Hmm. Please see docs/misc/xl-disk-configuration.txt for the
> configuration syntax understood by xl. Also note that IanJ has a series
> outstanding which improves the syntax, including compat with xend
> syntaxes and makes it more extensible for the future. The series
> includes an updated version of the doc, you'd be better off reading the
> new version than what is currently in the tree. A pre-patched version is
> attached.
>
> It doesn't currently support "remus:" and the "foo:" prefixes are in
> general deprecated. It looks like "remus:" will fall into the category
> of things which are supported via the script= directive. We've also
> grandfathered some "foo:" prefixes as shorthand for the script syntax
> (this is also how xend implemented them), so I think this will continue
> to work (assuming calling a script is how this works in xend, if not
> such a script might be needed).
>
> The "foo|bar" syntax is completely new to me (and I suspect anyone else
> not familiar with remus). How does it work? Is the full
> "backupHost:port|aio:/dev/foo" considered the argument to Remus (in
> which case I think it can work via the Remus script as above) or does
> xend somehow parse this into "remus:backupHost:port" and "aio:/dev/foo"?
> In the latter case I've no idea what to suggest!
>
> I dont think the script= directive is going to work (or even
necessary). The entire "foo|bar" part is handled by the blktap2 code base.
IOW, if the disk spec is tap:remus:host:port|aio:/dev/abc, then xl invokes
the blktap2 code and passes remus:host:port|aio:/dev/abc , which gets
parsed and both remus and aio drivers are created (remus driver on top of
aio).

Have you considered making Remus a more top-level domain configuration
> option rather than disk specific? i.e. adding remus_backup = "..." to
> the cfg. This would allow libxl to do the right thing internally and
> setup the disks in the right way etc etc.
>
> Yes I have, several times. Wading through xend code was not so much fun :(.
With xl, as long as it can construct the "remus:host:port|aio:/dev/abc" arg
and
pass it to the blktap2 code, things should be fine.
 With a DRBD based backend, nothing of this sort is required. Xend
automatically
invokes the block-drbd script, which does the rest. If xl does the same,
then things
should be fine.

> Doing The Right thing is something we are striving towards with libxl,
> especially with disk config which is unnecessarily complex for users.
>
> e.g. it should not be necessary for a user to specifically ask for tap
> or phy etc, rather they should present the path to the thing and libxl
> should figure out if blkback or blktap is needed. For example if Remus
> were enabled then it should DTRT and always select blktap even if
> blkback is otherwise suitable.

> The bulk of Remus transmission data is in libxc and hence is agnostic
> > to both xend/xl. It basically prolongs the last iteration for
> > eternity. It supplies a callback handler for checkpoint, which adds
> > the "wait" time before the next suspend (e.g., suspend every 50ms). In
> > case of Xend, the checkpoint handler is not supplied and hence the
> > domain is suspended as soon as the previous iteration finishes.
>
> I think exposing such a callback is within the scope of the libxl API.
> For example libxl_domain_checkpoint(...., callback) and
> libxl_domain_suspend(...) can probably backend onto the same internal
> function.
>
> Another option to the callbacks might be to integrate with the libxl
> event handling mechanism. Note that IanJ wants to overhall this from its
> current state. I'm less sure whether this would make sense.
>
> >
> > (a) On the sending side, without Remus, Xend control flow is as
> > follows:
> [...]
>
> looks mostly the same as xl, except xl does all the xc_domain_save stuff
> in process rather than indirecting via an external binary.

Do you mean that xl does all the xend stuff ? Because xl still calls
xc_domain_save
in libxl_dom.c:libxl__domain_suspend_common


> Also xl has
> to take care of starting a receiver process on the other end and has a
> bit more of a protocol interlock surrounding the actual migration to try
> and ensure the other end really is ready and hasn't failed etc.
>
>
> The callback structure has two other handlers (postcopy aka
> > postresume, checkpoint) that
> > is used by Remus.
> > *************************
> > (b) On sending side, with Remus
> >       remus <domain> <host>
>
> I suppose here there is a choice between adding libxl/xl support to this
> remus binary or implementing "xl remus <domain> <host>".
>
> The latter is what I wanted to do.

> >          (i) tools/remus/remus:
> >             - calls tools/python/xen/remus/vm.py:VM(domid)
> >             - vm.py:VM issues xmlrpc call to Xend to obtain domid's
> > sxpr and extract out the disk/vif info.
>
> Could be done via the libxl python bindings in the xl case?
>
> yep

> >          (ii) create the "buffers" for disk & vif.
>
> Stays the same, I guess, if you stick with the remus tool.
>
> >          (iii) Connect with remote host's Xend socket and send the sxp
> > info. [same as (i) for non Remus case]
>
> Hrm, this would involve duplicating a bunch of xl functionality to start
> the receiver, and run the xl protocol etc.
>
> That rather suggests that at least this bit should be in xl itself
> rather than remus. This needn't necessarily involve putting everything
> in xl, but just forking xl for this bit.
>
> >           (iv) tools/python/xen/remus/save.py:Saver uses libcheckpoint
> > to initiate checkpointing.
> >                 tools/python/xen/lowlevel/checkpoint: has
> > suspend/resume handlers similar to xc_save.c
> >                 trampoline functions to bounce the callbacks for
> > suspend, postcopy and checkpoint to their
> >                 python equivalents.
>
> I think either handling these inside libxl or bouncing them to the
> caller (depending on the nature of the callback) would be reasonable.
>
> >
> >
> > tools/python/xen/lowlevel/checkpoint/libcheckpoint.c:checkpoint_start
> > calls xc_domain_save with
> >                  all needed callback handlers.
> >                      ---> functionally equivalent to (ii) in non-Remus
> > case.
> >            (v) xc_domain_save: (after the initial iterations)
> >               copypages:
> >                send dirty pages & tailbuf data
> >                postcopy_callback() [resumes domain]
> >                checkpoint_callback()
> >                    netbuffer_checkpoint() [python - communicates via
> > netlink to sch_plug]
> >                    diskbuffer_checkpoint() [python - communicates via
> > fifo to block-remus]
> >                    sleep(50ms) [or whatever checkpoint interval]
> >                    return
> >                 suspend_callback()
> >                 goto copypages
> >
> > Hope that explains the control flow.
>
> I think so. Thanks.
>
> Hopefully some of the suggestions even make sense and demonstrate my new
> understanding ;-)
>
> >
> > shriram
> >
> >         Ian.
> >
> >
>
> shriram

[-- Attachment #1.2: Type: text/html, Size: 10601 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: libxl - API call to return sxpr of a domain?

[Ian Campbell]

(FYI, I'm about to disappear for a long weekend, back on Tuesday)

On Fri, 2011-06-10 at 00:59 +0100, Shriram Rajagopalan wrote:
> On Thu, Jun 9, 2011 at 7:34 AM, Ian Campbell
> <Ian.Campbell@eu.citrix.com> wrote:


>         The "foo|bar" syntax is completely new to me (and I suspect
>         anyone else not familiar with remus). How does it work? Is the
>         full "backupHost:port|aio:/dev/foo" considered the argument to
>         Remus (in which case I think it can work via the Remus script
>         as above) or does xend somehow parse this into
>         "remus:backupHost:port" and "aio:/dev/foo"?
>         In the latter case I've no idea what to suggest!
>         
> I dont think the script= directive is going to work (or even
> necessary). The entire "foo|bar" part is handled by the blktap2 code
> base. IOW, if the disk spec is tap:remus:host:port|aio:/dev/abc, then
> xl invokes the blktap2 code and passes remus:host:port|aio:/dev/abc ,
> which gets parsed and both remus and aio drivers are created (remus
> driver on top of aio).

OK, so in the terminology of xl-disk-configuration.txt "remus:host:port|
aio:/dev/abc" is the "target" and could potentially be handled
internally when libxl creates a blktap backend.

You mention below that block-drdb works so I very much expect that
block-remus would work too, even if ultimately it was just a thin
wrapper around tapctl etc. In that case I think the target would end up
just being "host:port|aio:/dev/abc" because the remus: would be a
shortcut for script=remus.

>         Have you considered making Remus a more top-level domain
>         configuration option rather than disk specific? i.e. adding
>         remus_backup = "..." to the cfg. This would allow libxl to do
>         the right thing internally and setup the disks in the right
>         way etc etc.

>         
> Yes I have, several times. Wading through xend code was not so much
> fun :(.

Yeah, I didn't mean for xend...

> With xl, as long as it can construct the "remus:host:port|
> aio:/dev/abc" arg and pass it to the blktap2 code, things should be
> fine.
>  With a DRBD based backend, nothing of this sort is required. Xend
> automatically invokes the block-drbd script, which does the rest. If
> xl does the same, then things should be fine.

[...]

>         looks mostly the same as xl, except xl does all the
>         xc_domain_save stuff
>         in process rather than indirecting via an external binary. 
> Do you mean that xl does all the xend stuff ? Because xl still calls
> xc_domain_save
> in libxl_dom.c:libxl__domain_suspend_common

What I meant was that in xend the xc_domain_save happens in a separate
helper process (xc_save) while in xl the xc_domain_save happens in the
original xl process itself (via libxl which calls libxc).

libxl uses libxc to do the low level stuff, like make hypercalls and do
migrations etc. Users of libxl (e.g. xl) don't get to see libxc though,
it is abstracted away.

Ian.