Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]

[Shriram Rajagopalan <rshriram@cs.ubc.ca>]

[-- Attachment #1.1: Type: text/plain, Size: 3561 bytes --]

On Mon, Nov 4, 2013 at 10:06 AM, Ian Jackson <Ian.Jackson@eu.citrix.com>wrote:

> Ian Campbell writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network
> buffering in remus callbacks [and 1 more messages]"):
> > Regardless of the answer here, would it make sense to do some/all of the
> > checkpoint processing in the helper subprocess anyway and only signal
> > the eventual failover up to the libxl process?
>
> It might do.  Mind you, the code in libxc is tangled enough as it is
> and is due for a rewrite.  Perhaps this could be done in the helper
> executable, although there isn't currently any way to easily
> intersperse code in there.
>
>
> This async op is potentially quite long running I think compared to a
> > normal one i.e. if the guest doesn't die it is expected that the ao
> > lives "forever". Since the associated gc's persist until the ao ends
> > this might end up accumulating lots of allocations? Ian had a similar
> > concern about Roger's hotplug daemon series and suggested creating a per
> > iteration gc or something.
>
Yes, this is indeed a problem.  Well spotted.
>
Which of the xc_domain_save (and _restore) callbacks are called each
> remus iteration ?
>
>
Almost all of them on the xc_domain_save side. (suspend, resume,
save_qemu state, checkpoint).
xc_domain_restore doesn't have any callbacks AFAIK. And remus as of now
does not have a component on the restore side. It piggybacks on live
migration's
restore framework.


> I think introducing a per-iteration gc here is going to involve taking
> some care, since we need to be sure not to put
> per-iteraton-gc-allocated objects into data structures which are used
> by subsequent iterations.
>
>
FWIW, the remus related code that executes per iteration does not allocate
anything.
All allocations happen only during setup and I was under the impression
that no other
allocations are taking place everytime xc_domain_save calls back into libxl.

However, it may be possible that other parts of the AO machinery (and there
are a lot of them) are allocating stuff per iteration. And if that is the
case, it could
easily lead to OOMs since Remus technically runs as long as the domain
lives.


Shriram writes:
> > Fair enough. My question is what is the overhead of setting up, firing
> > and tearing down a timeout event using the event gen framework, if I
> > wish to checkpoint the VM, say every 20ms ?
>
> The ultimate cost of going back into the event loop to wait for a
> timeout will depend on what else the process is doing.  If the process
> is doing nothing else, it's about two calls to gettimeofday and one to
> poll.  Plus a bit of in-process computation, but that's going to be
> swamped by system call overhead.
>
> Having said that, libxl is not performance-optimised.  Indeed the
> callback mechanism involves context switching, and IPC, between the
> save/restore helper and libxl proper.  Probably not too much to be
> doing every 20ms for a single domain, but if you have a lot of these
> it's going to end up taking a lot of dom0 cpu etc.
>
>
Yes and that is a problem. Xend+Remus avoided this by linking
the libcheckpoint library that interfaced with both the python & libxc code.


> I assume you're not doing this for HVM domains, which involve saving
> the qemu state each time too.
>
>
It includes HVM domains too. Although in that case, xenstore based suspend
takes about 5ms. So the checkpoint interval is typically 50ms or so.

If there is a latency sensitive task running inside
the VM, lower checkpoint interval leads to better performance.

[-- Attachment #1.2: Type: text/html, Size: 5435 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel