xl shutdown --wait "racy"

xl shutdown --wait "racy"

[Sander Eikelenboom]

Hi Ian (C|J) Konrad,

I'm currently trying to workaround the pci-(detach|assignable-remove) issues i 
reported earlier. 

The workaround i thought of was:
- shutting down the guest
- starting it without 1 of the original devices passed through
- use xl pci-assignable-remove and bind the device to the dom0 driver.

But during this i noticed that a "xl shutdown --wait" does wait .. but returns:
- Before the domain is removed from for instance "xl list", it still listed there in 
"--ps--" state.
- before pciback has done it's restore config space magic.

So it seems the wait loop is exiting somewhat prematurely, is this expected ? 

--
Sander

Re: xl shutdown --wait "racy"

[Ian Campbell]

On Wed, 2014-04-16 at 16:08 +0200, Sander Eikelenboom wrote:
> Hi Ian (C|J) Konrad,
> 
> I'm currently trying to workaround the pci-(detach|assignable-remove) issues i 
> reported earlier. 
> 
> The workaround i thought of was:
> - shutting down the guest
> - starting it without 1 of the original devices passed through
> - use xl pci-assignable-remove and bind the device to the dom0 driver.
> 
> But during this i noticed that a "xl shutdown --wait" does wait .. but returns:
> - Before the domain is removed from for instance "xl list", it still listed there in 
> "--ps--" state.
> - before pciback has done it's restore config space magic.
> 
> So it seems the wait loop is exiting somewhat prematurely, is this expected ? 

It is waiting for the domain to be shutdown (state 's') not for the
domain to be destroyed. So it's doing what it said it would (I
appreciate you might not find this distinction helpful under the
circumstances...)

Ian.

Re: xl shutdown --wait "racy"

[Sander Eikelenboom]

Wednesday, April 16, 2014, 4:13:59 PM, you wrote:

> On Wed, 2014-04-16 at 16:08 +0200, Sander Eikelenboom wrote:
>> Hi Ian (C|J) Konrad,
>> 
>> I'm currently trying to workaround the pci-(detach|assignable-remove) issues i 
>> reported earlier. 
>> 
>> The workaround i thought of was:
>> - shutting down the guest
>> - starting it without 1 of the original devices passed through
>> - use xl pci-assignable-remove and bind the device to the dom0 driver.
>> 
>> But during this i noticed that a "xl shutdown --wait" does wait .. but returns:
>> - Before the domain is removed from for instance "xl list", it still listed there in 
>> "--ps--" state.
>> - before pciback has done it's restore config space magic.
>> 
>> So it seems the wait loop is exiting somewhat prematurely, is this expected ? 

> It is waiting for the domain to be shutdown (state 's') not for the
> domain to be destroyed. So it's doing what it said it would (I
> appreciate you might not find this distinction helpful under the
> circumstances...)

It's at least not entirely what i expected ;-)

Is it because there can be different "follow-up actions" due to the 
"on_poweroff=" config option ?

I can see the shutdown state would be the thing to wait for if it's configured 
with "preserve", but if it's configured with the default "destroy" i would 
expect it to wait until the guest is actually destroyed. The "restart" is perhaps 
somewhat more difficult, same goes for the coredump variants.


> Ian.

Re: xl shutdown --wait "racy"

[George Dunlap]

On Wed, Apr 16, 2014 at 3:13 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Wed, 2014-04-16 at 16:08 +0200, Sander Eikelenboom wrote:
>> Hi Ian (C|J) Konrad,
>>
>> I'm currently trying to workaround the pci-(detach|assignable-remove) issues i
>> reported earlier.
>>
>> The workaround i thought of was:
>> - shutting down the guest
>> - starting it without 1 of the original devices passed through
>> - use xl pci-assignable-remove and bind the device to the dom0 driver.
>>
>> But during this i noticed that a "xl shutdown --wait" does wait .. but returns:
>> - Before the domain is removed from for instance "xl list", it still listed there in
>> "--ps--" state.
>> - before pciback has done it's restore config space magic.
>>
>> So it seems the wait loop is exiting somewhat prematurely, is this expected ?
>
> It is waiting for the domain to be shutdown (state 's') not for the
> domain to be destroyed. So it's doing what it said it would (I
> appreciate you might not find this distinction helpful under the
> circumstances...)

For any reasonable person's definition of "shutdown", it does *not*
wait until it's shutdown.  "In the shutdown state" is not something
anyone outside of Xen cares about: what they care about is being able
to, for example, start the domain again (or start a domain that
depends on resources currently held by the shutting down domain).

 -George

Re: xl shutdown --wait "racy"

[Ian Campbell]

On Wed, 2014-04-16 at 16:26 +0200, Sander Eikelenboom wrote:
> Wednesday, April 16, 2014, 4:13:59 PM, you wrote:
> 
> > On Wed, 2014-04-16 at 16:08 +0200, Sander Eikelenboom wrote:
> >> Hi Ian (C|J) Konrad,
> >> 
> >> I'm currently trying to workaround the pci-(detach|assignable-remove) issues i 
> >> reported earlier. 
> >> 
> >> The workaround i thought of was:
> >> - shutting down the guest
> >> - starting it without 1 of the original devices passed through
> >> - use xl pci-assignable-remove and bind the device to the dom0 driver.
> >> 
> >> But during this i noticed that a "xl shutdown --wait" does wait .. but returns:
> >> - Before the domain is removed from for instance "xl list", it still listed there in 
> >> "--ps--" state.
> >> - before pciback has done it's restore config space magic.
> >> 
> >> So it seems the wait loop is exiting somewhat prematurely, is this expected ? 
> 
> > It is waiting for the domain to be shutdown (state 's') not for the
> > domain to be destroyed. So it's doing what it said it would (I
> > appreciate you might not find this distinction helpful under the
> > circumstances...)
> 
> It's at least not entirely what i expected ;-)
> 
> Is it because there can be different "follow-up actions" due to the 
> "on_poweroff=" config option ?

Not really, those are somewhat unrelated.

shutdown and destroy are two distinct events. Once a domain has shutdown
(called the shutdown hypercall etc) it goes into state "shutdown" and an
event is generated from the hypervisor to the toolstack. The toolstack's
response to this is to actually destroy the domain, that is to tear down
the resources it is using etc.

on_* only matter for the destroy phase since they tell the toolstack
what it should do (restart, preserve, really destroy etc).

> I can see the shutdown state would be the thing to wait for if it's configured 
> with "preserve", but if it's configured with the default "destroy" i would 
> expect it to wait until the guest is actually destroyed. The "restart" is perhaps 
> somewhat more difficult, same goes for the coredump variants.

Re: xl shutdown --wait "racy"

[Sander Eikelenboom]

Wednesday, April 16, 2014, 4:33:30 PM, you wrote:

> On Wed, 2014-04-16 at 16:26 +0200, Sander Eikelenboom wrote:
>> Wednesday, April 16, 2014, 4:13:59 PM, you wrote:
>> 
>> > On Wed, 2014-04-16 at 16:08 +0200, Sander Eikelenboom wrote:
>> >> Hi Ian (C|J) Konrad,
>> >> 
>> >> I'm currently trying to workaround the pci-(detach|assignable-remove) issues i 
>> >> reported earlier. 
>> >> 
>> >> The workaround i thought of was:
>> >> - shutting down the guest
>> >> - starting it without 1 of the original devices passed through
>> >> - use xl pci-assignable-remove and bind the device to the dom0 driver.
>> >> 
>> >> But during this i noticed that a "xl shutdown --wait" does wait .. but returns:
>> >> - Before the domain is removed from for instance "xl list", it still listed there in 
>> >> "--ps--" state.
>> >> - before pciback has done it's restore config space magic.
>> >> 
>> >> So it seems the wait loop is exiting somewhat prematurely, is this expected ? 
>> 
>> > It is waiting for the domain to be shutdown (state 's') not for the
>> > domain to be destroyed. So it's doing what it said it would (I
>> > appreciate you might not find this distinction helpful under the
>> > circumstances...)
>> 
>> It's at least not entirely what i expected ;-)
>> 
>> Is it because there can be different "follow-up actions" due to the 
>> "on_poweroff=" config option ?

> Not really, those are somewhat unrelated.

> shutdown and destroy are two distinct events. Once a domain has shutdown
> (called the shutdown hypercall etc) it goes into state "shutdown" and an
> event is generated from the hypervisor to the toolstack. The toolstack's
> response to this is to actually destroy the domain, that is to tear down
> the resources it is using etc.

> on_* only matter for the destroy phase since they tell the toolstack
> what it should do (restart, preserve, really destroy etc).

Hmm ok, it should be called "--wait_until_halfway" then ;-)

On the more serious side .. would patches be accepted that:

a) differentiate when it returns from waiting based on the on_*

        preserve: this could probably stay as is .. after the shutdown event
        destroy:
        restart:
        rename-restart:
        coredump-destroy:
        coredump-restart:

        for the other ones .. i don't know if there actually are events in libxl 
        that could be 'easily' coupled ?

b) make it possible for the xl commandline to overrule the on_* from the configfile
c) also introduce a -w/--wait for xl destroy

(with b and c, i could emulate "a" by doing a "xl shutdown -w preserve" followed 
by a "xl destroy -w".

>> I can see the shutdown state would be the thing to wait for if it's configured 
>> with "preserve", but if it's configured with the default "destroy" i would 
>> expect it to wait until the guest is actually destroyed. The "restart" is perhaps 
>> somewhat more difficult, same goes for the coredump variants.

Re: xl shutdown --wait "racy"

[Ian Campbell]

On Wed, 2014-04-16 at 16:55 +0200, Sander Eikelenboom wrote:
> Wednesday, April 16, 2014, 4:33:30 PM, you wrote:
> 
> > On Wed, 2014-04-16 at 16:26 +0200, Sander Eikelenboom wrote:
> >> Wednesday, April 16, 2014, 4:13:59 PM, you wrote:
> >> 
> >> > On Wed, 2014-04-16 at 16:08 +0200, Sander Eikelenboom wrote:
> >> >> Hi Ian (C|J) Konrad,
> >> >> 
> >> >> I'm currently trying to workaround the pci-(detach|assignable-remove) issues i 
> >> >> reported earlier. 
> >> >> 
> >> >> The workaround i thought of was:
> >> >> - shutting down the guest
> >> >> - starting it without 1 of the original devices passed through
> >> >> - use xl pci-assignable-remove and bind the device to the dom0 driver.
> >> >> 
> >> >> But during this i noticed that a "xl shutdown --wait" does wait .. but returns:
> >> >> - Before the domain is removed from for instance "xl list", it still listed there in 
> >> >> "--ps--" state.
> >> >> - before pciback has done it's restore config space magic.
> >> >> 
> >> >> So it seems the wait loop is exiting somewhat prematurely, is this expected ? 
> >> 
> >> > It is waiting for the domain to be shutdown (state 's') not for the
> >> > domain to be destroyed. So it's doing what it said it would (I
> >> > appreciate you might not find this distinction helpful under the
> >> > circumstances...)
> >> 
> >> It's at least not entirely what i expected ;-)
> >> 
> >> Is it because there can be different "follow-up actions" due to the 
> >> "on_poweroff=" config option ?
> 
> > Not really, those are somewhat unrelated.
> 
> > shutdown and destroy are two distinct events. Once a domain has shutdown
> > (called the shutdown hypercall etc) it goes into state "shutdown" and an
> > event is generated from the hypervisor to the toolstack. The toolstack's
> > response to this is to actually destroy the domain, that is to tear down
> > the resources it is using etc.
> 
> > on_* only matter for the destroy phase since they tell the toolstack
> > what it should do (restart, preserve, really destroy etc).
> 
> Hmm ok, it should be called "--wait_until_halfway" then ;-)

;-)

> On the more serious side .. would patches be accepted that:
> 
> a) differentiate when it returns from waiting based on the on_*
> 
>         preserve: this could probably stay as is .. after the shutdown event
>         destroy:
>         restart:
>         rename-restart:
>         coredump-destroy:
>         coredump-restart:
> 
>         for the other ones .. i don't know if there actually are events in libxl 
>         that could be 'easily' coupled ?

Might be tricky, since on_* is processed by the daemonised xl which is
monitoring the domain, not the xl shutdown process.

> b) make it possible for the xl commandline to overrule the on_* from the configfile

I guess you mean the xl shutdown command. This will also be tricky, for
the same reasons as a.

> c) also introduce a -w/--wait for xl destroy

Yes.

I'll add:

d) Make "xl shutdown --wait" actually wait for the domain to be
destroyed.

Probably, assuming that is possible (I'm concerned about races in the
implementation of this...). Might also interact weirdly with on_* I
suppose.

Ian.

Re: xl shutdown --wait "racy"

[George Dunlap]

On 04/16/2014 04:02 PM, Ian Campbell wrote:
> On Wed, 2014-04-16 at 16:55 +0200, Sander Eikelenboom wrote:
>> Wednesday, April 16, 2014, 4:33:30 PM, you wrote:
>>
>>> On Wed, 2014-04-16 at 16:26 +0200, Sander Eikelenboom wrote:
>>>> Wednesday, April 16, 2014, 4:13:59 PM, you wrote:
>>>>
>>>>> On Wed, 2014-04-16 at 16:08 +0200, Sander Eikelenboom wrote:
>>>>>> Hi Ian (C|J) Konrad,
>>>>>>
>>>>>> I'm currently trying to workaround the pci-(detach|assignable-remove) issues i
>>>>>> reported earlier.
>>>>>>
>>>>>> The workaround i thought of was:
>>>>>> - shutting down the guest
>>>>>> - starting it without 1 of the original devices passed through
>>>>>> - use xl pci-assignable-remove and bind the device to the dom0 driver.
>>>>>>
>>>>>> But during this i noticed that a "xl shutdown --wait" does wait .. but returns:
>>>>>> - Before the domain is removed from for instance "xl list", it still listed there in
>>>>>> "--ps--" state.
>>>>>> - before pciback has done it's restore config space magic.
>>>>>>
>>>>>> So it seems the wait loop is exiting somewhat prematurely, is this expected ?
>>>>> It is waiting for the domain to be shutdown (state 's') not for the
>>>>> domain to be destroyed. So it's doing what it said it would (I
>>>>> appreciate you might not find this distinction helpful under the
>>>>> circumstances...)
>>>> It's at least not entirely what i expected ;-)
>>>>
>>>> Is it because there can be different "follow-up actions" due to the
>>>> "on_poweroff=" config option ?
>>> Not really, those are somewhat unrelated.
>>> shutdown and destroy are two distinct events. Once a domain has shutdown
>>> (called the shutdown hypercall etc) it goes into state "shutdown" and an
>>> event is generated from the hypervisor to the toolstack. The toolstack's
>>> response to this is to actually destroy the domain, that is to tear down
>>> the resources it is using etc.
>>> on_* only matter for the destroy phase since they tell the toolstack
>>> what it should do (restart, preserve, really destroy etc).
>> Hmm ok, it should be called "--wait_until_halfway" then ;-)
> ;-)
>
>> On the more serious side .. would patches be accepted that:
>>
>> a) differentiate when it returns from waiting based on the on_*
>>
>>          preserve: this could probably stay as is .. after the shutdown event
>>          destroy:
>>          restart:
>>          rename-restart:
>>          coredump-destroy:
>>          coredump-restart:
>>
>>          for the other ones .. i don't know if there actually are events in libxl
>>          that could be 'easily' coupled ?
> Might be tricky, since on_* is processed by the daemonised xl which is
> monitoring the domain, not the xl shutdown process.

Store it in xenstore somewhere?

>
>> b) make it possible for the xl commandline to overrule the on_* from the configfile
> I guess you mean the xl shutdown command. This will also be tricky, for
> the same reasons as a.
>
>> c) also introduce a -w/--wait for xl destroy
> Yes.
>
> I'll add:
>
> d) Make "xl shutdown --wait" actually wait for the domain to be
> destroyed.

This seems like the most reasonable thing to do.

  -George

Re: xl shutdown --wait "racy"

[Konrad Rzeszutek Wilk]

On Wed, Apr 16, 2014 at 04:08:21PM +0200, Sander Eikelenboom wrote:
> Hi Ian (C|J) Konrad,
> 
> I'm currently trying to workaround the pci-(detach|assignable-remove) issues i 
> reported earlier. 
> 
> The workaround i thought of was:
> - shutting down the guest
> - starting it without 1 of the original devices passed through
> - use xl pci-assignable-remove and bind the device to the dom0 driver.
> 
> But during this i noticed that a "xl shutdown --wait" does wait .. but returns:
> - Before the domain is removed from for instance "xl list", it still listed there in 
> "--ps--" state.
> - before pciback has done it's restore config space magic.

Which reminds me - when you do this - are you (or any processes) in the /sysfs directory?
I've been having a hard-time reproducing the issue. Sometimes I see it, but then if I try
again the problem does not show up again.

> 
> So it seems the wait loop is exiting somewhat prematurely, is this expected ? 
> 
> --
> Sander
> 

Re: xl shutdown --wait "racy"

[Sander Eikelenboom]

Wednesday, April 16, 2014, 5:02:50 PM, you wrote:

> On Wed, 2014-04-16 at 16:55 +0200, Sander Eikelenboom wrote:
>> Wednesday, April 16, 2014, 4:33:30 PM, you wrote:
>> 
>> > On Wed, 2014-04-16 at 16:26 +0200, Sander Eikelenboom wrote:
>> >> Wednesday, April 16, 2014, 4:13:59 PM, you wrote:
>> >> 
>> >> > On Wed, 2014-04-16 at 16:08 +0200, Sander Eikelenboom wrote:
>> >> >> Hi Ian (C|J) Konrad,
>> >> >> 
>> >> >> I'm currently trying to workaround the pci-(detach|assignable-remove) issues i 
>> >> >> reported earlier. 
>> >> >> 
>> >> >> The workaround i thought of was:
>> >> >> - shutting down the guest
>> >> >> - starting it without 1 of the original devices passed through
>> >> >> - use xl pci-assignable-remove and bind the device to the dom0 driver.
>> >> >> 
>> >> >> But during this i noticed that a "xl shutdown --wait" does wait .. but returns:
>> >> >> - Before the domain is removed from for instance "xl list", it still listed there in 
>> >> >> "--ps--" state.
>> >> >> - before pciback has done it's restore config space magic.
>> >> >> 
>> >> >> So it seems the wait loop is exiting somewhat prematurely, is this expected ? 
>> >> 
>> >> > It is waiting for the domain to be shutdown (state 's') not for the
>> >> > domain to be destroyed. So it's doing what it said it would (I
>> >> > appreciate you might not find this distinction helpful under the
>> >> > circumstances...)
>> >> 
>> >> It's at least not entirely what i expected ;-)
>> >> 
>> >> Is it because there can be different "follow-up actions" due to the 
>> >> "on_poweroff=" config option ?
>> 
>> > Not really, those are somewhat unrelated.
>> 
>> > shutdown and destroy are two distinct events. Once a domain has shutdown
>> > (called the shutdown hypercall etc) it goes into state "shutdown" and an
>> > event is generated from the hypervisor to the toolstack. The toolstack's
>> > response to this is to actually destroy the domain, that is to tear down
>> > the resources it is using etc.
>> 
>> > on_* only matter for the destroy phase since they tell the toolstack
>> > what it should do (restart, preserve, really destroy etc).
>> 
>> Hmm ok, it should be called "--wait_until_halfway" then ;-)

> ;-)

>> On the more serious side .. would patches be accepted that:
>> 
>> a) differentiate when it returns from waiting based on the on_*
>> 
>>         preserve: this could probably stay as is .. after the shutdown event
>>         destroy:
>>         restart:
>>         rename-restart:
>>         coredump-destroy:
>>         coredump-restart:
>> 
>>         for the other ones .. i don't know if there actually are events in libxl 
>>         that could be 'easily' coupled ?

> Might be tricky, since on_* is processed by the daemonised xl which is
> monitoring the domain, not the xl shutdown process.

>> b) make it possible for the xl commandline to overrule the on_* from the configfile

> I guess you mean the xl shutdown command. This will also be tricky, for
> the same reasons as a.

>> c) also introduce a -w/--wait for xl destroy

> Yes.

> I'll add:

> d) Make "xl shutdown --wait" actually wait for the domain to be
> destroyed.

> Probably, assuming that is possible (I'm concerned about races in the
> implementation of this...). Might also interact weirdly with on_* I
> suppose.

Well if we could pass down the events that "wait_for_domain_deaths" is allowed 
to return on ... now it seems to return on *any* event .. and only print 
something different on both shutdown and complete death ... 

Is there a special event that's triggered on timeout (as defined in 
/etc/defaults/xendomains: XENDOMAINS_STOP_MAXWAIT=300 ?

The the solution seems to be to let the caller of "wait_for_domain_deaths" be 
able the specify the events it should return on. (always return on timeout ... 
return on any unless specified ... only return on specified when specific events 
are specified)  

Could you elaborate on how you think this would get "racy" ?

static void wait_for_domain_deaths(libxl_evgen_domain_death **deathws, int nr)
{
    int rc, count = 0;
    LOG("Waiting for %d domains", nr);
    while(1 && count < nr) {
        libxl_event *event;
        rc = libxl_event_wait(ctx, &event, LIBXL_EVENTMASK_ALL, 0,0);
        if (rc) {
            LOG("Failed to get event, quitting (rc=%d)", rc);
            exit(-1);
        }

        switch (event->type) {
        case LIBXL_EVENT_TYPE_DOMAIN_DEATH:
            LOG("Domain %d has been destroyed", event->domid);
            libxl_evdisable_domain_death(ctx, deathws[event->for_user]);
            count++;
            break;
        case LIBXL_EVENT_TYPE_DOMAIN_SHUTDOWN:
            LOG("Domain %d has been shut down, reason code %d",
                event->domid, event->u.domain_shutdown.shutdown_reason);
            libxl_evdisable_domain_death(ctx, deathws[event->for_user]);
            count++;
            break;
        default:
            LOG("Unexpected event type %d", event->type);
            break;
        }
        libxl_event_free(ctx, event);
    }
}


> Ian.

Re: xl shutdown --wait "racy"

[Sander Eikelenboom]

Wednesday, April 16, 2014, 5:19:34 PM, you wrote:

> On Wed, Apr 16, 2014 at 04:08:21PM +0200, Sander Eikelenboom wrote:
>> Hi Ian (C|J) Konrad,
>> 
>> I'm currently trying to workaround the pci-(detach|assignable-remove) issues i 
>> reported earlier. 
>> 
>> The workaround i thought of was:
>> - shutting down the guest
>> - starting it without 1 of the original devices passed through
>> - use xl pci-assignable-remove and bind the device to the dom0 driver.
>> 
>> But during this i noticed that a "xl shutdown --wait" does wait .. but returns:
>> - Before the domain is removed from for instance "xl list", it still listed there in 
>> "--ps--" state.
>> - before pciback has done it's restore config space magic.

> Which reminds me - when you do this - are you (or any processes) in the /sysfs directory?
> I've been having a hard-time reproducing the issue. Sometimes I see it, but then if I try
> again the problem does not show up again.

When i do *what* ? (it's not exactly clear if you are referring to the problem 
i'm trying to workaround .. or the problems i'm finding when trying to 
workaround :-) )

>> 
>> So it seems the wait loop is exiting somewhat prematurely, is this expected ? 
>> 
>> --
>> Sander
>> 

Re: xl shutdown --wait "racy"

[Ian Campbell]

On Wed, 2014-04-16 at 17:20 +0200, Sander Eikelenboom wrote:
> Wednesday, April 16, 2014, 5:02:50 PM, you wrote:
> 
> > On Wed, 2014-04-16 at 16:55 +0200, Sander Eikelenboom wrote:
> >> Wednesday, April 16, 2014, 4:33:30 PM, you wrote:
> >> 
> >> > On Wed, 2014-04-16 at 16:26 +0200, Sander Eikelenboom wrote:
> >> >> Wednesday, April 16, 2014, 4:13:59 PM, you wrote:
> >> >> 
> >> >> > On Wed, 2014-04-16 at 16:08 +0200, Sander Eikelenboom wrote:
> >> >> >> Hi Ian (C|J) Konrad,
> >> >> >> 
> >> >> >> I'm currently trying to workaround the pci-(detach|assignable-remove) issues i 
> >> >> >> reported earlier. 
> >> >> >> 
> >> >> >> The workaround i thought of was:
> >> >> >> - shutting down the guest
> >> >> >> - starting it without 1 of the original devices passed through
> >> >> >> - use xl pci-assignable-remove and bind the device to the dom0 driver.
> >> >> >> 
> >> >> >> But during this i noticed that a "xl shutdown --wait" does wait .. but returns:
> >> >> >> - Before the domain is removed from for instance "xl list", it still listed there in 
> >> >> >> "--ps--" state.
> >> >> >> - before pciback has done it's restore config space magic.
> >> >> >> 
> >> >> >> So it seems the wait loop is exiting somewhat prematurely, is this expected ? 
> >> >> 
> >> >> > It is waiting for the domain to be shutdown (state 's') not for the
> >> >> > domain to be destroyed. So it's doing what it said it would (I
> >> >> > appreciate you might not find this distinction helpful under the
> >> >> > circumstances...)
> >> >> 
> >> >> It's at least not entirely what i expected ;-)
> >> >> 
> >> >> Is it because there can be different "follow-up actions" due to the 
> >> >> "on_poweroff=" config option ?
> >> 
> >> > Not really, those are somewhat unrelated.
> >> 
> >> > shutdown and destroy are two distinct events. Once a domain has shutdown
> >> > (called the shutdown hypercall etc) it goes into state "shutdown" and an
> >> > event is generated from the hypervisor to the toolstack. The toolstack's
> >> > response to this is to actually destroy the domain, that is to tear down
> >> > the resources it is using etc.
> >> 
> >> > on_* only matter for the destroy phase since they tell the toolstack
> >> > what it should do (restart, preserve, really destroy etc).
> >> 
> >> Hmm ok, it should be called "--wait_until_halfway" then ;-)
> 
> > ;-)
> 
> >> On the more serious side .. would patches be accepted that:
> >> 
> >> a) differentiate when it returns from waiting based on the on_*
> >> 
> >>         preserve: this could probably stay as is .. after the shutdown event
> >>         destroy:
> >>         restart:
> >>         rename-restart:
> >>         coredump-destroy:
> >>         coredump-restart:
> >> 
> >>         for the other ones .. i don't know if there actually are events in libxl 
> >>         that could be 'easily' coupled ?
> 
> > Might be tricky, since on_* is processed by the daemonised xl which is
> > monitoring the domain, not the xl shutdown process.
> 
> >> b) make it possible for the xl commandline to overrule the on_* from the configfile
> 
> > I guess you mean the xl shutdown command. This will also be tricky, for
> > the same reasons as a.
> 
> >> c) also introduce a -w/--wait for xl destroy
> 
> > Yes.
> 
> > I'll add:
> 
> > d) Make "xl shutdown --wait" actually wait for the domain to be
> > destroyed.
> 
> > Probably, assuming that is possible (I'm concerned about races in the
> > implementation of this...). Might also interact weirdly with on_* I
> > suppose.
> 
> Well if we could pass down the events that "wait_for_domain_deaths" is allowed 
> to return on ... now it seems to return on *any* event .. and only print 
> something different on both shutdown and complete death ... 

Any other even would be unexpected I think, since the corresponding
libxl_evenable_* would never have been called in the xl shutdown path.

> Is there a special event that's triggered on timeout (as defined in 
> /etc/defaults/xendomains: XENDOMAINS_STOP_MAXWAIT=300 ?

I don't see a timeout in the libxl_event_wait prototype so I suppose
not.

> The the solution seems to be to let the caller of "wait_for_domain_deaths" be 
> able the specify the events it should return on.

They already can -- by only enabling those events.

>  (always return on timeout ... 
> return on any unless specified ... only return on specified when specific events 
> are specified)  
> 
> Could you elaborate on how you think this would get "racy" ?

Oh, it looks like libxl already solved it and has an event, never mind.
(I was concerned that a new domain with the same domid might appear
right after the domain was gone)

Ian.

Re: xl shutdown --wait "racy"

[Sander Eikelenboom]

Wednesday, April 16, 2014, 6:27:10 PM, you wrote:

> On Wed, 2014-04-16 at 17:20 +0200, Sander Eikelenboom wrote:
>> Wednesday, April 16, 2014, 5:02:50 PM, you wrote:
>> 
>> > On Wed, 2014-04-16 at 16:55 +0200, Sander Eikelenboom wrote:
>> >> Wednesday, April 16, 2014, 4:33:30 PM, you wrote:
>> >> 
>> >> > On Wed, 2014-04-16 at 16:26 +0200, Sander Eikelenboom wrote:
>> >> >> Wednesday, April 16, 2014, 4:13:59 PM, you wrote:
>> >> >> 
>> >> >> > On Wed, 2014-04-16 at 16:08 +0200, Sander Eikelenboom wrote:
>> >> >> >> Hi Ian (C|J) Konrad,
>> >> >> >> 
>> >> >> >> I'm currently trying to workaround the pci-(detach|assignable-remove) issues i 
>> >> >> >> reported earlier. 
>> >> >> >> 
>> >> >> >> The workaround i thought of was:
>> >> >> >> - shutting down the guest
>> >> >> >> - starting it without 1 of the original devices passed through
>> >> >> >> - use xl pci-assignable-remove and bind the device to the dom0 driver.
>> >> >> >> 
>> >> >> >> But during this i noticed that a "xl shutdown --wait" does wait .. but returns:
>> >> >> >> - Before the domain is removed from for instance "xl list", it still listed there in 
>> >> >> >> "--ps--" state.
>> >> >> >> - before pciback has done it's restore config space magic.
>> >> >> >> 
>> >> >> >> So it seems the wait loop is exiting somewhat prematurely, is this expected ? 
>> >> >> 
>> >> >> > It is waiting for the domain to be shutdown (state 's') not for the
>> >> >> > domain to be destroyed. So it's doing what it said it would (I
>> >> >> > appreciate you might not find this distinction helpful under the
>> >> >> > circumstances...)
>> >> >> 
>> >> >> It's at least not entirely what i expected ;-)
>> >> >> 
>> >> >> Is it because there can be different "follow-up actions" due to the 
>> >> >> "on_poweroff=" config option ?
>> >> 
>> >> > Not really, those are somewhat unrelated.
>> >> 
>> >> > shutdown and destroy are two distinct events. Once a domain has shutdown
>> >> > (called the shutdown hypercall etc) it goes into state "shutdown" and an
>> >> > event is generated from the hypervisor to the toolstack. The toolstack's
>> >> > response to this is to actually destroy the domain, that is to tear down
>> >> > the resources it is using etc.
>> >> 
>> >> > on_* only matter for the destroy phase since they tell the toolstack
>> >> > what it should do (restart, preserve, really destroy etc).
>> >> 
>> >> Hmm ok, it should be called "--wait_until_halfway" then ;-)
>> 
>> > ;-)
>> 
>> >> On the more serious side .. would patches be accepted that:
>> >> 
>> >> a) differentiate when it returns from waiting based on the on_*
>> >> 
>> >>         preserve: this could probably stay as is .. after the shutdown event
>> >>         destroy:
>> >>         restart:
>> >>         rename-restart:
>> >>         coredump-destroy:
>> >>         coredump-restart:
>> >> 
>> >>         for the other ones .. i don't know if there actually are events in libxl 
>> >>         that could be 'easily' coupled ?
>> 
>> > Might be tricky, since on_* is processed by the daemonised xl which is
>> > monitoring the domain, not the xl shutdown process.
>> 
>> >> b) make it possible for the xl commandline to overrule the on_* from the configfile
>> 
>> > I guess you mean the xl shutdown command. This will also be tricky, for
>> > the same reasons as a.
>> 
>> >> c) also introduce a -w/--wait for xl destroy
>> 
>> > Yes.
>> 
>> > I'll add:
>> 
>> > d) Make "xl shutdown --wait" actually wait for the domain to be
>> > destroyed.
>> 
>> > Probably, assuming that is possible (I'm concerned about races in the
>> > implementation of this...). Might also interact weirdly with on_* I
>> > suppose.
>> 
>> Well if we could pass down the events that "wait_for_domain_deaths" is allowed 
>> to return on ... now it seems to return on *any* event .. and only print 
>> something different on both shutdown and complete death ... 

> Any other even would be unexpected I think, since the corresponding
> libxl_evenable_* would never have been called in the xl shutdown path.

>> Is there a special event that's triggered on timeout (as defined in 
>> /etc/defaults/xendomains: XENDOMAINS_STOP_MAXWAIT=300 ?

> I don't see a timeout in the libxl_event_wait prototype so I suppose
> not.

>> The the solution seems to be to let the caller of "wait_for_domain_deaths" be 
>> able the specify the events it should return on.

> They already can -- by only enabling those events.

>>  (always return on timeout ... 
>> return on any unless specified ... only return on specified when specific events 
>> are specified)  
>> 
>> Could you elaborate on how you think this would get "racy" ?

> Oh, it looks like libxl already solved it and has an event, never mind.
> (I was concerned that a new domain with the same domid might appear
> right after the domain was gone)

Hrmm yes well .. i only see a way to enable both from libxl_event.h:

typedef struct libxl__evgen_domain_death libxl_evgen_domain_death;
int libxl_evenable_domain_death(libxl_ctx *ctx, uint32_t domid,
                         libxl_ev_user, libxl_evgen_domain_death **evgen_out);
void libxl_evdisable_domain_death(libxl_ctx *ctx, libxl_evgen_domain_death*);
  /* Arranges for the generation of DOMAIN_SHUTDOWN and DOMAIN_DESTROY
   * events.  A domain which is destroyed before it shuts down
   * may generate only a DESTROY event.
   */


Although from libxl_internal.h it looks like there once has been a thought about 
differentiating:

struct libxl__evgen_domain_death {
    uint32_t domid;
    unsigned shutdown_reported:1, death_reported:1;
    LIBXL_TAILQ_ENTRY(libxl_evgen_domain_death) entry;
        /* on list .death_reported ? CTX->death_list : CTX->death_reported */
    libxl_ev_user user;
};
_hidden void
libxl__evdisable_domain_death(libxl__gc*, libxl_evgen_domain_death*);



> Ian.

Re: xl shutdown --wait "racy"

[Konrad Rzeszutek Wilk]

On Wed, Apr 16, 2014 at 05:21:57PM +0200, Sander Eikelenboom wrote:
> 
> Wednesday, April 16, 2014, 5:19:34 PM, you wrote:
> 
> > On Wed, Apr 16, 2014 at 04:08:21PM +0200, Sander Eikelenboom wrote:
> >> Hi Ian (C|J) Konrad,
> >> 
> >> I'm currently trying to workaround the pci-(detach|assignable-remove) issues i 
> >> reported earlier. 
> >> 
> >> The workaround i thought of was:
> >> - shutting down the guest
> >> - starting it without 1 of the original devices passed through
> >> - use xl pci-assignable-remove and bind the device to the dom0 driver.
> >> 
> >> But during this i noticed that a "xl shutdown --wait" does wait .. but returns:
> >> - Before the domain is removed from for instance "xl list", it still listed there in 
> >> "--ps--" state.
> >> - before pciback has done it's restore config space magic.
> 
> > Which reminds me - when you do this - are you (or any processes) in the /sysfs directory?
> > I've been having a hard-time reproducing the issue. Sometimes I see it, but then if I try
> > again the problem does not show up again.
> 
> When i do *what* ? (it's not exactly clear if you are referring to the problem 
> i'm trying to workaround .. or the problems i'm finding when trying to 
> workaround :-) )

Whoa. Looking back at my response it is pretty vague. I was refering to the
http://lists.xen.org/archives/html/xen-devel/2014-04/msg02237.html issue.

But that is unrelated to this problem you are reporting so lets just skip
my question.

Re: xl shutdown --wait "racy"

[Ian Campbell]

On Wed, 2014-04-16 at 19:04 +0200, Sander Eikelenboom wrote:

> i only see a way to enable both from libxl_event.h:

You could always ignore the one you don't want...

Re: xl shutdown --wait "racy"

[Ian Jackson]

George Dunlap writes ("Re: [Xen-devel] xl shutdown --wait "racy""):
> On Wed, Apr 16, 2014 at 3:13 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > It is waiting for the domain to be shutdown (state 's') not for the
> > domain to be destroyed. So it's doing what it said it would (I
> > appreciate you might not find this distinction helpful under the
> > circumstances...)
> 
> For any reasonable person's definition of "shutdown", it does *not*
> wait until it's shutdown.  "In the shutdown state" is not something
> anyone outside of Xen cares about: what they care about is being able
> to, for example, start the domain again (or start a domain that
> depends on resources currently held by the shutting down domain).

Quite.  I think this is simply a bug and it should wait for the domain
to be destroyed.

It's IMO a tolerable side-effect if this means that when the domain
shuts down in a way that causes it to be preserved (ie the daemonic xl
doesn't reap it) xl shutdown -w gets stuck.  (There should be an
option to restore the former behaviour but it should not be the
default.)

Ideally xl would record something somewhere so that it would know
what's going on and could make "xl shutdown -w" fail in the default
case if it's going to wait "forever".

Ian.

Re: xl shutdown --wait "racy"

[Sander Eikelenboom]

Thursday, April 17, 2014, 8:09:39 PM, you wrote:

> George Dunlap writes ("Re: [Xen-devel] xl shutdown --wait "racy""):
>> On Wed, Apr 16, 2014 at 3:13 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> > It is waiting for the domain to be shutdown (state 's') not for the
>> > domain to be destroyed. So it's doing what it said it would (I
>> > appreciate you might not find this distinction helpful under the
>> > circumstances...)
>> 
>> For any reasonable person's definition of "shutdown", it does *not*
>> wait until it's shutdown.  "In the shutdown state" is not something
>> anyone outside of Xen cares about: what they care about is being able
>> to, for example, start the domain again (or start a domain that
>> depends on resources currently held by the shutting down domain).

> Quite.  I think this is simply a bug and it should wait for the domain
> to be destroyed.

It seems the shutdown is racy in another aspect as well, the code in "wait_for_domain_deaths" 
on receiving a event doesn't actually check if it's a event related to the 
domain it should wait for. It only checks a count (1 if you shutdown only one 
domain, nb_domains -1 if you shutdown all).

To be less error prone it should ideally pass an array of domids it needs to 
wait for through to wait_for_domain_deaths and check on it.

> It's IMO a tolerable side-effect if this means that when the domain
> shuts down in a way that causes it to be preserved (ie the daemonic xl
> doesn't reap it) xl shutdown -w gets stuck.  (There should be an
> option to restore the former behaviour but it should not be the
> default.)

> Ideally xl would record something somewhere so that it would know
> what's going on and could make "xl shutdown -w" fail in the default
> case if it's going to wait "forever".

There isn't a generic timeout function that could raise an timeout event
when it is taking too long ?

But i'm running into my all too limited C skills here ..
(passive reading is fairly ok, actively building something somewhat more complex 
in a sane way is not :-) )


> Ian.

Re: xl shutdown --wait "racy"

[Ian Campbell]

On Wed, 2014-04-23 at 15:38 +0200, Sander Eikelenboom wrote:
> Thursday, April 17, 2014, 8:09:39 PM, you wrote:
> 
> > George Dunlap writes ("Re: [Xen-devel] xl shutdown --wait "racy""):
> >> On Wed, Apr 16, 2014 at 3:13 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >> > It is waiting for the domain to be shutdown (state 's') not for the
> >> > domain to be destroyed. So it's doing what it said it would (I
> >> > appreciate you might not find this distinction helpful under the
> >> > circumstances...)
> >> 
> >> For any reasonable person's definition of "shutdown", it does *not*
> >> wait until it's shutdown.  "In the shutdown state" is not something
> >> anyone outside of Xen cares about: what they care about is being able
> >> to, for example, start the domain again (or start a domain that
> >> depends on resources currently held by the shutting down domain).
> 
> > Quite.  I think this is simply a bug and it should wait for the domain
> > to be destroyed.
> 
> It seems the shutdown is racy in another aspect as well, the code in "wait_for_domain_deaths" 
> on receiving a event doesn't actually check if it's a event related to the 
> domain it should wait for. It only checks a count (1 if you shutdown only one 
> domain, nb_domains -1 if you shutdown all).

It only calls libxl_evenable_domain_death for domains which it should
wait for, and by construction the number of deaths passed to
wait_for_domain_deaths is equal to the number of domains for which
libxl_evenable_domain_death was called.

> To be less error prone it should ideally pass an array of domids it needs to 
> wait for through to wait_for_domain_deaths and check on it.
> 
> > It's IMO a tolerable side-effect if this means that when the domain
> > shuts down in a way that causes it to be preserved (ie the daemonic xl
> > doesn't reap it) xl shutdown -w gets stuck.  (There should be an
> > option to restore the former behaviour but it should not be the
> > default.)
> 
> > Ideally xl would record something somewhere so that it would know
> > what's going on and could make "xl shutdown -w" fail in the default
> > case if it's going to wait "forever".
> 
> There isn't a generic timeout function that could raise an timeout event
> when it is taking too long ?

Nope. I suppose you could use alarm(2) or something but that would be
getting pretty hairy.

Ian.

Re: xl shutdown --wait "racy"

[Sander Eikelenboom]

Wednesday, April 23, 2014, 3:42:32 PM, you wrote:

> On Wed, 2014-04-23 at 15:38 +0200, Sander Eikelenboom wrote:
>> Thursday, April 17, 2014, 8:09:39 PM, you wrote:
>> 
>> > George Dunlap writes ("Re: [Xen-devel] xl shutdown --wait "racy""):
>> >> On Wed, Apr 16, 2014 at 3:13 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> >> > It is waiting for the domain to be shutdown (state 's') not for the
>> >> > domain to be destroyed. So it's doing what it said it would (I
>> >> > appreciate you might not find this distinction helpful under the
>> >> > circumstances...)
>> >> 
>> >> For any reasonable person's definition of "shutdown", it does *not*
>> >> wait until it's shutdown.  "In the shutdown state" is not something
>> >> anyone outside of Xen cares about: what they care about is being able
>> >> to, for example, start the domain again (or start a domain that
>> >> depends on resources currently held by the shutting down domain).
>> 
>> > Quite.  I think this is simply a bug and it should wait for the domain
>> > to be destroyed.
>> 
>> It seems the shutdown is racy in another aspect as well, the code in "wait_for_domain_deaths" 
>> on receiving a event doesn't actually check if it's a event related to the 
>> domain it should wait for. It only checks a count (1 if you shutdown only one 
>> domain, nb_domains -1 if you shutdown all).

> It only calls libxl_evenable_domain_death for domains which it should
> wait for, and by construction the number of deaths passed to
> wait_for_domain_deaths is equal to the number of domains for which
> libxl_evenable_domain_death was called.

Also if i do two times "xl shutdown --wait" with a script in parallel ?

That would give a event for the guest that shuts down the fastest .. but wouldn't 
the "wait_for_domain_deaths" for the other guest also respond to that (and 
return too early) ?


>> To be less error prone it should ideally pass an array of domids it needs to 
>> wait for through to wait_for_domain_deaths and check on it.
>> 
>> > It's IMO a tolerable side-effect if this means that when the domain
>> > shuts down in a way that causes it to be preserved (ie the daemonic xl
>> > doesn't reap it) xl shutdown -w gets stuck.  (There should be an
>> > option to restore the former behaviour but it should not be the
>> > default.)
>> 
>> > Ideally xl would record something somewhere so that it would know
>> > what's going on and could make "xl shutdown -w" fail in the default
>> > case if it's going to wait "forever".
>> 
>> There isn't a generic timeout function that could raise an timeout event
>> when it is taking too long ?

> Nope. I suppose you could use alarm(2) or something but that would be
> getting pretty hairy.

> Ian.

Re: xl shutdown --wait "racy"

[Ian Campbell]

On Wed, 2014-04-23 at 15:49 +0200, Sander Eikelenboom wrote:
> Wednesday, April 23, 2014, 3:42:32 PM, you wrote:
> 
> > On Wed, 2014-04-23 at 15:38 +0200, Sander Eikelenboom wrote:
> >> Thursday, April 17, 2014, 8:09:39 PM, you wrote:
> >> 
> >> > George Dunlap writes ("Re: [Xen-devel] xl shutdown --wait "racy""):
> >> >> On Wed, Apr 16, 2014 at 3:13 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >> >> > It is waiting for the domain to be shutdown (state 's') not for the
> >> >> > domain to be destroyed. So it's doing what it said it would (I
> >> >> > appreciate you might not find this distinction helpful under the
> >> >> > circumstances...)
> >> >> 
> >> >> For any reasonable person's definition of "shutdown", it does *not*
> >> >> wait until it's shutdown.  "In the shutdown state" is not something
> >> >> anyone outside of Xen cares about: what they care about is being able
> >> >> to, for example, start the domain again (or start a domain that
> >> >> depends on resources currently held by the shutting down domain).
> >> 
> >> > Quite.  I think this is simply a bug and it should wait for the domain
> >> > to be destroyed.
> >> 
> >> It seems the shutdown is racy in another aspect as well, the code in "wait_for_domain_deaths" 
> >> on receiving a event doesn't actually check if it's a event related to the 
> >> domain it should wait for. It only checks a count (1 if you shutdown only one 
> >> domain, nb_domains -1 if you shutdown all).
> 
> > It only calls libxl_evenable_domain_death for domains which it should
> > wait for, and by construction the number of deaths passed to
> > wait_for_domain_deaths is equal to the number of domains for which
> > libxl_evenable_domain_death was called.
> 
> Also if i do two times "xl shutdown --wait" with a script in parallel ?
> 
> That would give a event for the guest that shuts down the fastest ..

Both instance of xl are independent and will see the events for the
domain they are interested in, irrespecitve of whether some other xl is
also waiting for hte same domain to die.

>  but wouldn't 
> the "wait_for_domain_deaths" for the other guest also respond to that (and 
> return too early) ?

I don't think libxl will fire events for domains which the particular xl
instance didn't express an interest in.

Ian.