Domains not being destroyed properly

Domains not being destroyed properly

[Anthony Wright]

If I create a domain with 'xl create -e', and then shut the domain down
with 'xl shutdown' according to 'xl list' it gets stuck in state
'--ps-d', with a name of '(none)' and 0 ram, ('xm list' doesn't show the
domain).

If I destroy the domain with 'xl destroy' the domain is destroyed properly.

If I create a domain with 'xl create' (without the '-e' option) and then
use 'xl shutdown', the domain is destroyed properly.

Since the 'xl shutdown' & 'xl destroy' give different results I presume
this is a bug.


As an extra question... Is there a way to be notified when a domain is
destroyed other than leaving the 'xl create' process lying around? I'd
like to know when any domain is destroyed, and leaving a large number of
processes lying around just to be able to do this seems rather ugly. In
the past I've editted some of the python code to achieve this, but my
patch doesn't work with 4.1, so I'm seeing if there's an official way to
do this before I work out a new patch.

thanks,

Anthony.

Re: Domains not being destroyed properly

[Nathan March]

On 6/17/2011 4:29 AM, Anthony Wright wrote:
> As an extra question... Is there a way to be notified when a domain is
> destroyed other than leaving the 'xl create' process lying around? I'd
> like to know when any domain is destroyed, and leaving a large number of
> processes lying around just to be able to do this seems rather ugly. In
> the past I've editted some of the python code to achieve this, but my
> patch doesn't work with 4.1, so I'm seeing if there's an official way to
> do this before I work out a new patch.

Not ideal, but my approach was to add hook scripts into the block device
script and consider a vm down if xvda1 has been removed:

Line 229 (below the FRONTEND_UUID):
   /path/to/block add ${XENBUS_PATH}

Line 324 (below "remove)"):
   /path/to/block remove ${XENBUS_PATH}


xen1 scripts # cat block
#!/bin/bash
set -x -e

STATE="$1"
PATH="$2"

VMNAME=$(/usr/bin/xenstore-read ${PATH}/domain);
DEV=$(/usr/bin/xenstore-read ${PATH}/dev);

if [[ "$DEV" == "xvda1" ]]; then
    # Do stuff
fi;

exit 0


- Nathan

Re: Domains not being destroyed properly

[Ian Campbell]

On Fri, 2011-06-17 at 12:29 +0100, Anthony Wright wrote:
> If I create a domain with 'xl create -e', and then shut the domain down
> with 'xl shutdown' according to 'xl list' it gets stuck in state
> '--ps-d', with a name of '(none)' and 0 ram, ('xm list' doesn't show the
> domain).

> 
> If I destroy the domain with 'xl destroy' the domain is destroyed properly.
> 
> If I create a domain with 'xl create' (without the '-e' option) and then
> use 'xl shutdown', the domain is destroyed properly.

> 
> Since the 'xl shutdown' & 'xl destroy' give different results I presume
> this is a bug.

The -e option to xl create means don't daemonize to babysit this domain.
One of the key bits of functionality of the daemon is to destroy the
domain after it is shutdown. So if you use -e you need to do the destroy
manually. So effectively you have gotten what you asked for ;-)

> As an extra question... Is there a way to be notified when a domain is
> destroyed other than leaving the 'xl create' process lying around? I'd
> like to know when any domain is destroyed, and leaving a large number of
> processes lying around just to be able to do this seems rather ugly. In
> the past I've editted some of the python code to achieve this, but my
> patch doesn't work with 4.1, so I'm seeing if there's an official way to
> do this before I work out a new patch.

You can take a xenstore watch on the @releaseDomain pseudo node, does
that do what you want?

Ian.

> 
> thanks,
> 
> Anthony.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Domains not being destroyed properly

[Anthony Wright]

On 21/06/2011 12:51, Ian Campbell wrote:
> On Fri, 2011-06-17 at 12:29 +0100, Anthony Wright wrote:
>> If I create a domain with 'xl create -e', and then shut the domain down
>> with 'xl shutdown' according to 'xl list' it gets stuck in state
>> '--ps-d', with a name of '(none)' and 0 ram, ('xm list' doesn't show the
>> domain).
>> If I destroy the domain with 'xl destroy' the domain is destroyed properly.
>>
>> If I create a domain with 'xl create' (without the '-e' option) and then
>> use 'xl shutdown', the domain is destroyed properly.
>> Since the 'xl shutdown' & 'xl destroy' give different results I presume
>> this is a bug.
> The -e option to xl create means don't daemonize to babysit this domain.
> One of the key bits of functionality of the daemon is to destroy the
> domain after it is shutdown. So if you use -e you need to do the destroy
> manually. So effectively you have gotten what you asked for ;-)
>
However I can't destroy the domain with 'xl destroy'. The command runs,
but the domain is still there afterwards.
>> As an extra question... Is there a way to be notified when a domain is
>> destroyed other than leaving the 'xl create' process lying around? I'd
>> like to know when any domain is destroyed, and leaving a large number of
>> processes lying around just to be able to do this seems rather ugly. In
>> the past I've editted some of the python code to achieve this, but my
>> patch doesn't work with 4.1, so I'm seeing if there's an official way to
>> do this before I work out a new patch.
> You can take a xenstore watch on the @releaseDomain pseudo node, does
> that do what you want?
>
I found the @releaseDomain xenstore watch, and have modified my code to
use it. I was wondering if there's a mechanism to find out which domain
is dying. The only mechanism I can find at the moment is to scan through
all the domains to see which one isn't there any more. Assuming 'xl
destroy' works, it would make things a little easier. Is there a better
way to get the domain states other than doing a 'xl list' and parsing
the result?

thanks,

Anthony.

Re: Domains not being destroyed properly

[Ian Jackson]

Anthony Wright writes ("Re: [Xen-devel] Domains not being destroyed properly"):
> On 21/06/2011 12:51, Ian Campbell wrote:
> > On Fri, 2011-06-17 at 12:29 +0100, Anthony Wright wrote:
> >> If I destroy the domain with 'xl destroy' the domain is destroyed
> >> properly.
...
> However I can't destroy the domain with 'xl destroy'. The command runs,
> but the domain is still there afterwards.

Which of the above statements is true ?  Or, alternatively, what is
the difference between the two situations ?

Ian.

Re: Domains not being destroyed properly

[Ian Jackson]

Anthony Wright writes ("[Xen-devel] Domains not being destroyed properly"):
> As an extra question... Is there a way to be notified when a domain is
> destroyed other than leaving the 'xl create' process lying around? I'd
> like to know when any domain is destroyed, and leaving a large number of
> processes lying around just to be able to do this seems rather ugly. In
> the past I've editted some of the python code to achieve this, but my
> patch doesn't work with 4.1, so I'm seeing if there's an official way to
> do this before I work out a new patch.

Processes are cheap.  But, the facilities should be there in libxl to
code this up as part of "xl" perhaps, or as a separate ad-hoc utility.

Ian.

Re: Domains not being destroyed properly

[Konrad Rzeszutek Wilk]

On Tue, Jun 21, 2011 at 01:57:20PM +0100, Anthony Wright wrote:
> On 21/06/2011 12:51, Ian Campbell wrote:
> > On Fri, 2011-06-17 at 12:29 +0100, Anthony Wright wrote:
> >> If I create a domain with 'xl create -e', and then shut the domain down
> >> with 'xl shutdown' according to 'xl list' it gets stuck in state
> >> '--ps-d', with a name of '(none)' and 0 ram, ('xm list' doesn't show the
> >> domain).
> >> If I destroy the domain with 'xl destroy' the domain is destroyed properly.
> >>
> >> If I create a domain with 'xl create' (without the '-e' option) and then
> >> use 'xl shutdown', the domain is destroyed properly.
> >> Since the 'xl shutdown' & 'xl destroy' give different results I presume
> >> this is a bug.
> > The -e option to xl create means don't daemonize to babysit this domain.
> > One of the key bits of functionality of the daemon is to destroy the
> > domain after it is shutdown. So if you use -e you need to do the destroy
> > manually. So effectively you have gotten what you asked for ;-)
> >
> However I can't destroy the domain with 'xl destroy'. The command runs,
> but the domain is still there afterwards.

Is qemu-dm running for that domain? what happens if you kill it?

Re: Domains not being destroyed properly

[Anthony Wright]

On 21/06/2011 14:39, Konrad Rzeszutek Wilk wrote:
> On Tue, Jun 21, 2011 at 01:57:20PM +0100, Anthony Wright wrote:
>> On 21/06/2011 12:51, Ian Campbell wrote:
>>> On Fri, 2011-06-17 at 12:29 +0100, Anthony Wright wrote:
>>>> If I create a domain with 'xl create -e', and then shut the domain down
>>>> with 'xl shutdown' according to 'xl list' it gets stuck in state
>>>> '--ps-d', with a name of '(none)' and 0 ram, ('xm list' doesn't show the
>>>> domain).
>>>> If I destroy the domain with 'xl destroy' the domain is destroyed properly.
>>>>
>>>> If I create a domain with 'xl create' (without the '-e' option) and then
>>>> use 'xl shutdown', the domain is destroyed properly.
>>>> Since the 'xl shutdown' & 'xl destroy' give different results I presume
>>>> this is a bug.
>>> The -e option to xl create means don't daemonize to babysit this domain.
>>> One of the key bits of functionality of the daemon is to destroy the
>>> domain after it is shutdown. So if you use -e you need to do the destroy
>>> manually. So effectively you have gotten what you asked for ;-)
>>>
>> However I can't destroy the domain with 'xl destroy'. The command runs,
>> but the domain is still there afterwards.
> Is qemu-dm running for that domain? what happens if you kill it?
Ah ha.... that makes the domain go away.

Ok, to be clear what I did....

xl create -e domain-details
xl shutdown <domain>
(End up with a domain called (null) in state --ps-d according to xl list
kill the qemu-dm process associated with the domain
the domain goes away

note that I didn't need to do an 'xl destroy' the kill of qemu-dm was
sufficient

If I do....

xl create -e domain-details
xl destroy <domain>
the domain goes away

Re: Domains not being destroyed properly

[Ian Campbell]

On Tue, 2011-06-21 at 15:52 +0100, Anthony Wright wrote:
> On 21/06/2011 14:39, Konrad Rzeszutek Wilk wrote:
> > On Tue, Jun 21, 2011 at 01:57:20PM +0100, Anthony Wright wrote:
> >> On 21/06/2011 12:51, Ian Campbell wrote:
> >>> On Fri, 2011-06-17 at 12:29 +0100, Anthony Wright wrote:
> >>>> If I create a domain with 'xl create -e', and then shut the domain down
> >>>> with 'xl shutdown' according to 'xl list' it gets stuck in state
> >>>> '--ps-d', with a name of '(none)' and 0 ram, ('xm list' doesn't show the
> >>>> domain).
> >>>> If I destroy the domain with 'xl destroy' the domain is destroyed properly.
> >>>>
> >>>> If I create a domain with 'xl create' (without the '-e' option) and then
> >>>> use 'xl shutdown', the domain is destroyed properly.
> >>>> Since the 'xl shutdown' & 'xl destroy' give different results I presume
> >>>> this is a bug.
> >>> The -e option to xl create means don't daemonize to babysit this domain.
> >>> One of the key bits of functionality of the daemon is to destroy the
> >>> domain after it is shutdown. So if you use -e you need to do the destroy
> >>> manually. So effectively you have gotten what you asked for ;-)
> >>>
> >> However I can't destroy the domain with 'xl destroy'. The command runs,
> >> but the domain is still there afterwards.
> > Is qemu-dm running for that domain? what happens if you kill it?
> Ah ha.... that makes the domain go away.
> 
> Ok, to be clear what I did....
> 
> xl create -e domain-details
> xl shutdown <domain>
> (End up with a domain called (null) in state --ps-d according to xl list
> kill the qemu-dm process associated with the domain
> the domain goes away
> 
> note that I didn't need to do an 'xl destroy' the kill of qemu-dm was
> sufficient
> 
> If I do....
> 
> xl create -e domain-details
> xl destroy <domain>
> the domain goes away

The difference is that shutdown is only a request to the guest to shut
itself down, while destroy is the big hammer which kills the guest with
prejudice and tidies up any detritus (or at least that is the intention
IIRC).

I suspect that you will probably find that only the hypervisor parts of
the domain are being cleaned up by killing qemu -- e.g. the xenstore
entries and some other stuff (e.g. libxl user data) are not being
cleaned up.

If you do:
xl create -e domain-details
xl shutdown <domain>
xl destroy <domain>

Does everything get cleaned up?

If you don't want to run an xl daemon for each domain (although I agree
with IanJ that they should be pretty cheap) I think a new xl sub-command
which caused it to behave as a single daemon reaping domains would be a
fine thing to have in our toolkit.

Ian.

Re: Domains not being destroyed properly

[Anthony Wright]

On 21/06/2011 16:44, Ian Campbell wrote:
> On Tue, 2011-06-21 at 15:52 +0100, Anthony Wright wrote:
>> Ok, to be clear what I did....
>> xl create -e domain-details
>> xl shutdown <domain>
>> (End up with a domain called (null) in state --ps-d according to xl list
>> kill the qemu-dm process associated with the domain
>> the domain goes away
>>
>> note that I didn't need to do an 'xl destroy' the kill of qemu-dm was
>> sufficient
>>
>> If I do....
>>
>> xl create -e domain-details
>> xl destroy <domain>
>> the domain goes away
> The difference is that shutdown is only a request to the guest to shut
> itself down, while destroy is the big hammer which kills the guest with
> prejudice and tidies up any detritus (or at least that is the intention
> IIRC).
>
> I suspect that you will probably find that only the hypervisor parts of
> the domain are being cleaned up by killing qemu -- e.g. the xenstore
> entries and some other stuff (e.g. libxl user data) are not being
> cleaned up.
>
> If you do:
> xl create -e domain-details
> xl shutdown <domain>
> xl destroy <domain>
>
> Does everything get cleaned up?
If I do:
xl create -e domain-details
xl shutdown <domain>
[ Wait for the domain to go to state --ps-d ]
xl destroy <domain>

The 'xl destroy' has no effect. The only thing that has an effect is
killing the qemu-dm processes which destroys the domain even without
running 'xl destroy'.

> If you don't want to run an xl daemon for each domain (although I agree
> with IanJ that they should be pretty cheap) I think a new xl sub-command
> which caused it to behave as a single daemon reaping domains would be a
> fine thing to have in our toolkit.
That would be really helpful, something that would inform me when a
domain died supplying it's domain id & domain name - the name is really
important because that's my reference, and I can't convert from an
domain id to a domain name after the domain has died. I used to have a
script that sat in /etc/xen/scripts/domain-destroyed which was called on
domain death with the dom-id and dom-name as arguments.

Anthony

Re: Domains not being destroyed properly

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] Domains not being destroyed properly"):
> I suspect that you will probably find that only the hypervisor parts of
> the domain are being cleaned up by killing qemu -- e.g. the xenstore
> entries and some other stuff (e.g. libxl user data) are not being
> cleaned up.

It is puzzling that the hypervisor domain gets destroyed at all.  What
is making the destroy domain hypercall ?

Ian.

Re: Domains not being destroyed properly

[Ian Campbell]

On Tue, 2011-06-21 at 17:26 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] Domains not being destroyed properly"):
> > I suspect that you will probably find that only the hypervisor parts of
> > the domain are being cleaned up by killing qemu -- e.g. the xenstore
> > entries and some other stuff (e.g. libxl user data) are not being
> > cleaned up.
> 
> It is puzzling that the hypervisor domain gets destroyed at all.  What
> is making the destroy domain hypercall ?

I wondered that and handwaved it to myself as being the
HLT-with-interrupt-disabled logic kicking in and shutting down the guest
but now that I look it seems that this just triggers a domain shutdown
event to dom0 and doesn't actually destroy the domain (which is actually
as expected before I handwaved). So I'm a bit surprised again now too.

Ian.

Re: Domains not being destroyed properly

[Keir Fraser]

On 21/06/2011 17:42, "Ian Campbell" <Ian.Campbell@eu.citrix.com> wrote:

> On Tue, 2011-06-21 at 17:26 +0100, Ian Jackson wrote:
>> Ian Campbell writes ("Re: [Xen-devel] Domains not being destroyed properly"):
>>> I suspect that you will probably find that only the hypervisor parts of
>>> the domain are being cleaned up by killing qemu -- e.g. the xenstore
>>> entries and some other stuff (e.g. libxl user data) are not being
>>> cleaned up.
>> 
>> It is puzzling that the hypervisor domain gets destroyed at all.  What
>> is making the destroy domain hypercall ?
> 
> I wondered that and handwaved it to myself as being the
> HLT-with-interrupt-disabled logic kicking in and shutting down the guest
> but now that I look it seems that this just triggers a domain shutdown
> event to dom0 and doesn't actually destroy the domain (which is actually
> as expected before I handwaved). So I'm a bit surprised again now too.

A domain can only be destroyed via the destroydomain domctl from dom0.

 -- Keir

> Ian.
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Domains not being destroyed properly

[Tim Deegan]

Hi, 

At 17:26 +0100 on 21 Jun (1308677212), Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] Domains not being destroyed properly"):
> > I suspect that you will probably find that only the hypervisor parts of
> > the domain are being cleaned up by killing qemu -- e.g. the xenstore
> > entries and some other stuff (e.g. libxl user data) are not being
> > cleaned up.
> 
> It is puzzling that the hypervisor domain gets destroyed at all.  What
> is making the destroy domain hypercall ?

I suspect that xl destroy is doing it, but the domain can't actually
disappear until the last references to ist memory are dropped by
qemu-dm.

Tim.

-- 
Tim Deegan <tim@xen.org>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: Domains not being destroyed properly

[Ian Campbell]

On Tue, 2011-06-21 at 17:04 +0100, Anthony Wright wrote:
> On 21/06/2011 16:44, Ian Campbell wrote:
> > On Tue, 2011-06-21 at 15:52 +0100, Anthony Wright wrote:
> >> Ok, to be clear what I did....
> >> xl create -e domain-details
> >> xl shutdown <domain>
> >> (End up with a domain called (null) in state --ps-d according to xl list
> >> kill the qemu-dm process associated with the domain
> >> the domain goes away
> >>
> >> note that I didn't need to do an 'xl destroy' the kill of qemu-dm was
> >> sufficient
> >>
> >> If I do....
> >>
> >> xl create -e domain-details
> >> xl destroy <domain>
> >> the domain goes away
> > The difference is that shutdown is only a request to the guest to shut
> > itself down, while destroy is the big hammer which kills the guest with
> > prejudice and tidies up any detritus (or at least that is the intention
> > IIRC).
> >
> > I suspect that you will probably find that only the hypervisor parts of
> > the domain are being cleaned up by killing qemu -- e.g. the xenstore
> > entries and some other stuff (e.g. libxl user data) are not being
> > cleaned up.
> >
> > If you do:
> > xl create -e domain-details
> > xl shutdown <domain>
> > xl destroy <domain>
> >
> > Does everything get cleaned up?
> If I do:
> xl create -e domain-details
> xl shutdown <domain>
> [ Wait for the domain to go to state --ps-d ]
> xl destroy <domain>
> 
> The 'xl destroy' has no effect. The only thing that has an effect is
> killing the qemu-dm processes which destroys the domain even without
> running 'xl destroy'.

This sounds like a bug in "xl destroy" to me.

> 
> > If you don't want to run an xl daemon for each domain (although I agree
> > with IanJ that they should be pretty cheap) I think a new xl sub-command
> > which caused it to behave as a single daemon reaping domains would be a
> > fine thing to have in our toolkit.
> That would be really helpful, something that would inform me when a
> domain died supplying it's domain id & domain name - the name is really
> important because that's my reference, and I can't convert from an
> domain id to a domain name after the domain has died.

In this case I think you can do so before the call to "xl destroy" since
that is when all the xenstore gubbins needed to do the translation is
called. In the past xend probably got their before you could and nuked
it all.

BTW, I was trying to imply that we would accept a patch which
implemented this new subcommand ;-)

>  I used to have a
> script that sat in /etc/xen/scripts/domain-destroyed which was called on
> domain death with the dom-id and dom-name as arguments.

Including some sort of callout mechanism on shutdown but before destroy
would be a reasonable feature for a daemon which did this kind of global
babysitting to have.

Ian.

Re: Domains not being destroyed properly

[Ian Campbell]

On Tue, 2011-06-21 at 20:35 +0100, Tim Deegan wrote:
> Hi, 
> 
> At 17:26 +0100 on 21 Jun (1308677212), Ian Jackson wrote:
> > Ian Campbell writes ("Re: [Xen-devel] Domains not being destroyed properly"):
> > > I suspect that you will probably find that only the hypervisor parts of
> > > the domain are being cleaned up by killing qemu -- e.g. the xenstore
> > > entries and some other stuff (e.g. libxl user data) are not being
> > > cleaned up.
> > 
> > It is puzzling that the hypervisor domain gets destroyed at all.  What
> > is making the destroy domain hypercall ?
> 
> I suspect that xl destroy is doing it, but the domain can't actually
> disappear until the last references to ist memory are dropped by
> qemu-dm.

Anthony reckons that just killing the qemu after an "xl shutdown" is
sufficient, without an "xl destroy". I can't see anywhere that "xl
shutdown" would issue the destroydomain hypercall though (which is
expected, since it shouldn't be doing it).

Ian.

Re: Domains not being destroyed properly

[Anthony Wright]

On 22/06/2011 08:56, Ian Campbell wrote:
> On Tue, 2011-06-21 at 17:04 +0100, Anthony Wright wrote:
>> On 21/06/2011 16:44, Ian Campbell wrote:
>>> If you don't want to run an xl daemon for each domain (although I agree
>>> with IanJ that they should be pretty cheap) I think a new xl sub-command
>>> which caused it to behave as a single daemon reaping domains would be a
>>> fine thing to have in our toolkit.
>> That would be really helpful, something that would inform me when a
>> domain died supplying it's domain id & domain name - the name is really
>> important because that's my reference, and I can't convert from an
>> domain id to a domain name after the domain has died.
> In this case I think you can do so before the call to "xl destroy" since
> that is when all the xenstore gubbins needed to do the translation is
> called. In the past xend probably got their before you could and nuked
> it all.
>
> BTW, I was trying to imply that we would accept a patch which
> implemented this new subcommand ;-)
The problem is that I don't think there's enough information available
from watching @releaseDomain to implement a reaper well. The
xenstore-watch on @releaseDomain only tells you that a domain has died
(to make things more complicated I actually get two notifications when a
domain get's shutdown), it doesn't tell you which domain has died. A
reaper would have to maintain a list of domains as they were before the
notification to compare against the list after the notification to be
able to issue a notification for the domain that has died. If that list
got out of sync it would start issuing incorrect domain death notifications.

Looking at the watch code while the code in xenstored_domain has the
domain id & name available, there doesn't seem to be the ability to be
able to add arguments to @releaseDomain to let it tell the watchers
which domain has been released to avoid having to maintain this list.

Therefore the simplest change would seem to be to modify
xenstored_domain.c so that if a script existed (e.g.
/etc/xen/scripts/domain-destroyed) at domain destruction this script was
called with the domain id & name as arguments. Would it be acceptable?

Anthony.

Re: Domains not being destroyed properly

[Tim Deegan]

Hi, 

At 13:54 +0100 on 24 Jun (1308923697), Anthony Wright wrote:
> The problem is that I don't think there's enough information available
> from watching @releaseDomain to implement a reaper well. The
> xenstore-watch on @releaseDomain only tells you that a domain has died
> (to make things more complicated I actually get two notifications when a
> domain get's shutdown), it doesn't tell you which domain has died. A
> reaper would have to maintain a list of domains as they were before the
> notification to compare against the list after the notification to be
> able to issue a notification for the domain that has died.

That sounds awfully fragile.  It could get a list of all living domains
and reap qemus/xenstore data for any domain not in the list.  (Or, more
safely, enumerate all qemus and xenstore entries and check for each
whether the domain is alive). 

> Therefore the simplest change would seem to be to modify
> xenstored_domain.c so that if a script existed (e.g.
> /etc/xen/scripts/domain-destroyed) at domain destruction this script was
> called with the domain id & name as arguments. Would it be acceptable?

I don't think so.  At the moment it's possible (with a bit of effort) to
run xenstored in a chroot jail or in its own VM; plumbing it into the
rest of the tollstack that way would break that.

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: Domains not being destroyed properly

[Anthony Wright]

On 24/06/2011 14:01, Tim Deegan wrote:
> Hi, 
>
> At 13:54 +0100 on 24 Jun (1308923697), Anthony Wright wrote:
>> The problem is that I don't think there's enough information available
>> from watching @releaseDomain to implement a reaper well. The
>> xenstore-watch on @releaseDomain only tells you that a domain has died
>> (to make things more complicated I actually get two notifications when a
>> domain get's shutdown), it doesn't tell you which domain has died. A
>> reaper would have to maintain a list of domains as they were before the
>> notification to compare against the list after the notification to be
>> able to issue a notification for the domain that has died.
> That sounds awfully fragile.  It could get a list of all living domains
> and reap qemus/xenstore data for any domain not in the list.  (Or, more
> safely, enumerate all qemus and xenstore entries and check for each
> whether the domain is alive). 
Is this information still available after a domain has been destroyed? I
was expecting that the watcher of @releaseDomain was only notified of a
domain destruction after the domain had been completely destroyed so
there would be no information amount the domain available any more. Are
you saying the would still be qemus/xenstore information available for
the domain, and if so how does it get tidied up?

Anthony.

Re: Domains not being destroyed properly

[Ian Campbell]

On Fri, 2011-06-24 at 13:54 +0100, Anthony Wright wrote:
> The problem is that I don't think there's enough information available
> from watching @releaseDomain to implement a reaper well. The
> xenstore-watch on @releaseDomain only tells you that a domain has died
> (to make things more complicated I actually get two notifications when a
> domain get's shutdown), it doesn't tell you which domain has died.

This has proven to be sufficient for all toolstacks I know of.

>  A
> reaper would have to maintain a list of domains as they were before the
> notification to compare against the list after the notification to be
> able to issue a notification for the domain that has died. If that list
> got out of sync it would start issuing incorrect domain death notifications.

You don't need to maintain a list, simply call libxl_list_domain() and
for each returned domain examine the flags and shutdown reason to
determine if it is dead (and why), dying or still running.

Since you are explicitly arranging that nothing else will destroy
domains once they die so they will always be in that list when you get
to them.

> Looking at the watch code while the code in xenstored_domain has the
> domain id & name available, there doesn't seem to be the ability to be
> able to add arguments to @releaseDomain to let it tell the watchers
> which domain has been released to avoid having to maintain this list.

It shouldn't matter due to the above but it is possible to add extra
data to the vector included in the watch other than XS_WATCH_PATH and
XS_WATCH_TOKEN). IOW you can add more stuff to the "data" vector in
add_event. Linux's drivers/xen/xenfs/xenbus.c:watch_fired() even
supports exporting the extra to userspace.

> Therefore the simplest change would seem to be to modify
> xenstored_domain.c so that if a script existed (e.g.
> /etc/xen/scripts/domain-destroyed) at domain destruction this script was
> called with the domain id & name as arguments. Would it be acceptable?
> 
> Anthony.

Re: Domains not being destroyed properly

[Anthony Wright]

On 24/06/2011 14:45, Ian Campbell wrote:
> On Fri, 2011-06-24 at 13:54 +0100, Anthony Wright wrote:
>> A reaper would have to maintain a list of domains as they were before the
>> notification to compare against the list after the notification to be
>> able to issue a notification for the domain that has died. If that list
>> got out of sync it would start issuing incorrect domain death notifications.
> You don't need to maintain a list, simply call libxl_list_domain() and
> for each returned domain examine the flags and shutdown reason to
> determine if it is dead (and why), dying or still running.
>
> Since you are explicitly arranging that nothing else will destroy
> domains once they die so they will always be in that list when you get
> to them.
My original thought on reaper was that it was only responsible for
helping notify user space applications of a domain's death,  but you
seem to be suggesting that it should also be responsible for tidying up
after a domain death as well. Isn't some other part of the system
already fulfilling this role?

It seems that for 'xl create' this tidy up is already being done, and I
had presumed the code for 'xl create -e' tidy up already exists but
wasn't working correctly. If you're saying the 'xl create -e' tidy up
code needs to be written, that's probably beyond my knowledge of xen,
and in any case I'm not sure it should be in reaper. Reaper should
notify user space of any domain destruction whether created by 'xl
create -e' or 'xl create'. If reaper did the tidy up for 'xl create -e'
it would have access to information needed to pass to the user space
applications, however for an 'xl create' domain death this information
may no longer be available since 'xl create' may have already tidied up.

>> Looking at the watch code while the code in xenstored_domain has the
>> domain id & name available, there doesn't seem to be the ability to be
>> able to add arguments to @releaseDomain to let it tell the watchers
>> which domain has been released to avoid having to maintain this list.
> It shouldn't matter due to the above but it is possible to add extra
> data to the vector included in the watch other than XS_WATCH_PATH and
> XS_WATCH_TOKEN). IOW you can add more stuff to the "data" vector in
> add_event. Linux's drivers/xen/xenfs/xenbus.c:watch_fired() even
> supports exporting the extra to userspace.

Re: Domains not being destroyed properly

[Ian Campbell]

On Fri, 2011-06-24 at 14:36 +0100, Anthony Wright wrote:
> On 24/06/2011 14:01, Tim Deegan wrote:
> > Hi, 
> >
> > At 13:54 +0100 on 24 Jun (1308923697), Anthony Wright wrote:
> >> The problem is that I don't think there's enough information available
> >> from watching @releaseDomain to implement a reaper well. The
> >> xenstore-watch on @releaseDomain only tells you that a domain has died
> >> (to make things more complicated I actually get two notifications when a
> >> domain get's shutdown), it doesn't tell you which domain has died. A
> >> reaper would have to maintain a list of domains as they were before the
> >> notification to compare against the list after the notification to be
> >> able to issue a notification for the domain that has died.
> > That sounds awfully fragile.  It could get a list of all living domains
> > and reap qemus/xenstore data for any domain not in the list.  (Or, more
> > safely, enumerate all qemus and xenstore entries and check for each
> > whether the domain is alive). 
> Is this information still available after a domain has been destroyed? I
> was expecting that the watcher of @releaseDomain was only notified of a
> domain destruction after the domain had been completely destroyed

@releaseDomain fires after the domain has shutdown but _before_ the
domain is destroyed. This is necessary because @releaseDomain is the
signal to the toolstack that it is time to destroy the domain.

>  so
> there would be no information amount the domain available any more. Are
> you saying the would still be qemus/xenstore information available for
> the domain, and if so how does it get tidied up?
> 
> Anthony.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Domains not being destroyed properly

[Ian Campbell]

On Fri, 2011-06-24 at 15:15 +0100, Anthony Wright wrote:
> On 24/06/2011 14:45, Ian Campbell wrote:
> > On Fri, 2011-06-24 at 13:54 +0100, Anthony Wright wrote:
> >> A reaper would have to maintain a list of domains as they were before the
> >> notification to compare against the list after the notification to be
> >> able to issue a notification for the domain that has died. If that list
> >> got out of sync it would start issuing incorrect domain death notifications.
> > You don't need to maintain a list, simply call libxl_list_domain() and
> > for each returned domain examine the flags and shutdown reason to
> > determine if it is dead (and why), dying or still running.
> >
> > Since you are explicitly arranging that nothing else will destroy
> > domains once they die so they will always be in that list when you get
> > to them.
> My original thought on reaper was that it was only responsible for
> helping notify user space applications of a domain's death,  but you
> seem to be suggesting that it should also be responsible for tidying up
> after a domain death as well. Isn't some other part of the system
> already fulfilling this role?

It would be xl, but you are using xl create -e which explicitly turns
off that behaviour.

> It seems that for 'xl create' this tidy up is already being done, and I
> had presumed the code for 'xl create -e' tidy up already exists but
> wasn't working correctly.

No, "xl create -e" means "don't babysit this domain waiting for it to
die. I, the user, will take care of cleaning it up manually when it
dies".

You said you were using -e because you didn't want a load of processes
sitting around to fulfil this requirement and so I suggested you might
like a single reaper daemon which takes case of it instead.

Ian.


>  If you're saying the 'xl create -e' tidy up
> code needs to be written, that's probably beyond my knowledge of xen,
> and in any case I'm not sure it should be in reaper. Reaper should
> notify user space of any domain destruction whether created by 'xl
> create -e' or 'xl create'. If reaper did the tidy up for 'xl create -e'
> it would have access to information needed to pass to the user space
> applications, however for an 'xl create' domain death this information
> may no longer be available since 'xl create' may have already tidied up.