No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping

No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping

[Sander Eikelenboom]

I'm encountering the following problem:

When trying to shutdown a PV domU with more than 1 vpcu, the shutdown fails.
The domain keeps running with 100% cpu, and it's still possible to get the console of this domU with xm console.
When only 1 vcpu is assigned the domain does shutdown.

Last lines of the PV domU console:

Debian GNU/Linux 5.0 tv hvc0

INIT: Switching to runlevel: 0
INIT: Sending processes the TERM signal
Stopping web server: apache2 ... waiting .
Stopping MythTV server: mythbackend No /usr/bin/mythbackend found running; none killed.
.
Stopping MTA: exim4_listener.
Stopping rsync daemon: rsync.
Stopping MySQL database server: mysqld.
Saving the system clock.
Cannot access the Hardware Clock via any known method.
Use the --debug option to see the details of our search for an access method.
Stopping enhanced syslogd: rsyslogd.
Asking all remaining processes to terminate...done.
All processes ended within 2 seconds....done.
Deconfiguring network interfaces...done.
Cleaning up ifupdown....
Deactivating swap...done.
Unmounting local filesystems...done.
Will now halt.
[ 4336.046876] md: stopping all md devices.
[ 4337.047171] xenbus_dev_shutdown:  trying shutdown of device/console/0: Initialising
[ 4337.047194] xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping
[ 4337.047200] xenbus_dev_shutdown: result of shutdown of device/console/0: Initialising
[ 4337.047205] xenbus_dev_shutdown:  trying shutdown of device/vif/0: Connected
[ 4337.110869] xenbus_dev_shutdown: result of shutdown of device/vif/0: Closed
[ 4337.110883] xenbus_dev_shutdown:  trying shutdown of device/vbd/51714: Connected
[ 4337.161975] xenbus_dev_shutdown: result of shutdown of device/vbd/51714: Closed
[ 4337.161989] xenbus_dev_shutdown:  trying shutdown of device/vbd/51713: Connected
[ 4337.217136] xenbus_dev_shutdown: result of shutdown of device/vbd/51713: Closed
[ 4337.217167] System halted.


But when using xenstore-ls .. i see that for every domain (but 1 and multiple vcpu's):
    - All devices have state=4
    - Except all backend = "/local/domain/0/backend/console/*/0" entries, those have state=1
    - Although xenstore is saying the state is initializing .. xm console works perfectly for all domains.
    - Perhaps this also explains the high event/0 load in dom0, related to tty and xenconsoled ?

DomU kernels vary from debian 2.6.26-xen kernels, to 2.6.37-rc1 mainline.
Xen-unstable-tip and xen-next-2.6.32 dom0


--
Sander

Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping

[Sander Eikelenboom]

Hmm .. i haven't received any response, is there anyone who could point me to the functions involved in communicating the state of the console from "initializing" to "connected" ?
That way i could at some additional printk's to find out why the state of domU consoles stays "1" instead of "4" in xenstore.

--
Sander

Friday, November 12, 2010, 6:30:10 PM, you wrote:

> I'm encountering the following problem:

> When trying to shutdown a PV domU with more than 1 vpcu, the shutdown fails.
> The domain keeps running with 100% cpu, and it's still possible to get the console of this domU with xm console.
> When only 1 vcpu is assigned the domain does shutdown.

> Last lines of the PV domU console:

> Debian GNU/Linux 5.0 tv hvc0

> INIT: Switching to runlevel: 0
> INIT: Sending processes the TERM signal
> Stopping web server: apache2 ... waiting .
> Stopping MythTV server: mythbackend No /usr/bin/mythbackend found running; none killed.
> .
> Stopping MTA: exim4_listener.
> Stopping rsync daemon: rsync.
> Stopping MySQL database server: mysqld.
> Saving the system clock.
> Cannot access the Hardware Clock via any known method.
> Use the --debug option to see the details of our search for an access method.
> Stopping enhanced syslogd: rsyslogd.
> Asking all remaining processes to terminate...done.
> All processes ended within 2 seconds....done.
> Deconfiguring network interfaces...done.
> Cleaning up ifupdown....
> Deactivating swap...done.
> Unmounting local filesystems...done.
> Will now halt.
> [ 4336.046876] md: stopping all md devices.
> [ 4337.047171] xenbus_dev_shutdown:  trying shutdown of device/console/0: Initialising
> [ 4337.047194] xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping
> [ 4337.047200] xenbus_dev_shutdown: result of shutdown of device/console/0: Initialising
> [ 4337.047205] xenbus_dev_shutdown:  trying shutdown of device/vif/0: Connected
> [ 4337.110869] xenbus_dev_shutdown: result of shutdown of device/vif/0: Closed
> [ 4337.110883] xenbus_dev_shutdown:  trying shutdown of device/vbd/51714: Connected
> [ 4337.161975] xenbus_dev_shutdown: result of shutdown of device/vbd/51714: Closed
> [ 4337.161989] xenbus_dev_shutdown:  trying shutdown of device/vbd/51713: Connected
> [ 4337.217136] xenbus_dev_shutdown: result of shutdown of device/vbd/51713: Closed
> [ 4337.217167] System halted.


> But when using xenstore-ls .. i see that for every domain (but 1 and multiple vcpu's):
>     - All devices have state=4
>     - Except all backend = "/local/domain/0/backend/console/*/0" entries, those have state=1
>     - Although xenstore is saying the state is initializing .. xm console works perfectly for all domains.
>     - Perhaps this also explains the high event/0 load in dom0, related to tty and xenconsoled ?

> DomU kernels vary from debian 2.6.26-xen kernels, to 2.6.37-rc1 mainline.
> Xen-unstable-tip and xen-next-2.6.32 dom0


> --
> Sander



-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping

[Keir Fraser]

Consoles do not have a connection handshake. If there is a state field in
xenstore, it is only unused detritus written by the toolstack
(xend/libxl/whatever).

 -- Keir

On 17/11/2010 11:58, "Sander Eikelenboom" <linux@eikelenboom.it> wrote:

> Hmm .. i haven't received any response, is there anyone who could point me to
> the functions involved in communicating the state of the console from
> "initializing" to "connected" ?
> That way i could at some additional printk's to find out why the state of domU
> consoles stays "1" instead of "4" in xenstore.
> 
> --
> Sander
> 
> Friday, November 12, 2010, 6:30:10 PM, you wrote:
> 
>> I'm encountering the following problem:
> 
>> When trying to shutdown a PV domU with more than 1 vpcu, the shutdown fails.
>> The domain keeps running with 100% cpu, and it's still possible to get the
>> console of this domU with xm console.
>> When only 1 vcpu is assigned the domain does shutdown.
> 
>> Last lines of the PV domU console:
> 
>> Debian GNU/Linux 5.0 tv hvc0
> 
>> INIT: Switching to runlevel: 0
>> INIT: Sending processes the TERM signal
>> Stopping web server: apache2 ... waiting .
>> Stopping MythTV server: mythbackend No /usr/bin/mythbackend found running;
>> none killed.
>> .
>> Stopping MTA: exim4_listener.
>> Stopping rsync daemon: rsync.
>> Stopping MySQL database server: mysqld.
>> Saving the system clock.
>> Cannot access the Hardware Clock via any known method.
>> Use the --debug option to see the details of our search for an access method.
>> Stopping enhanced syslogd: rsyslogd.
>> Asking all remaining processes to terminate...done.
>> All processes ended within 2 seconds....done.
>> Deconfiguring network interfaces...done.
>> Cleaning up ifupdown....
>> Deactivating swap...done.
>> Unmounting local filesystems...done.
>> Will now halt.
>> [ 4336.046876] md: stopping all md devices.
>> [ 4337.047171] xenbus_dev_shutdown:  trying shutdown of device/console/0:
>> Initialising
>> [ 4337.047194] xenbus_dev_shutdown: device/console/0: Initialising !=
>> Connected, skipping
>> [ 4337.047200] xenbus_dev_shutdown: result of shutdown of device/console/0:
>> Initialising
>> [ 4337.047205] xenbus_dev_shutdown:  trying shutdown of device/vif/0:
>> Connected
>> [ 4337.110869] xenbus_dev_shutdown: result of shutdown of device/vif/0:
>> Closed
>> [ 4337.110883] xenbus_dev_shutdown:  trying shutdown of device/vbd/51714:
>> Connected
>> [ 4337.161975] xenbus_dev_shutdown: result of shutdown of device/vbd/51714:
>> Closed
>> [ 4337.161989] xenbus_dev_shutdown:  trying shutdown of device/vbd/51713:
>> Connected
>> [ 4337.217136] xenbus_dev_shutdown: result of shutdown of device/vbd/51713:
>> Closed
>> [ 4337.217167] System halted.
> 
> 
>> But when using xenstore-ls .. i see that for every domain (but 1 and multiple
>> vcpu's):
>>     - All devices have state=4
>>     - Except all backend = "/local/domain/0/backend/console/*/0" entries,
>> those have state=1
>>     - Although xenstore is saying the state is initializing .. xm console
>> works perfectly for all domains.
>>     - Perhaps this also explains the high event/0 load in dom0, related to
>> tty and xenconsoled ?
> 
>> DomU kernels vary from debian 2.6.26-xen kernels, to 2.6.37-rc1 mainline.
>> Xen-unstable-tip and xen-next-2.6.32 dom0
> 
> 
>> --
>> Sander
> 
> 

Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping

[Sander Eikelenboom]

OK ... so when a pv domU shutdown, in xenbus_dev_shutdown, what does the call dev->state return for a console ?
and why does it only seems to cause a problem for domains with multiple vcpu's assigned ?

--
Sander


Wednesday, November 17, 2010, 1:06:04 PM, you wrote:

> Consoles do not have a connection handshake. If there is a state field in
> xenstore, it is only unused detritus written by the toolstack
> (xend/libxl/whatever).

>  -- Keir

> On 17/11/2010 11:58, "Sander Eikelenboom" <linux@eikelenboom.it> wrote:

>> Hmm .. i haven't received any response, is there anyone who could point me to
>> the functions involved in communicating the state of the console from
>> "initializing" to "connected" ?
>> That way i could at some additional printk's to find out why the state of domU
>> consoles stays "1" instead of "4" in xenstore.
>> 
>> --
>> Sander
>> 
>> Friday, November 12, 2010, 6:30:10 PM, you wrote:
>> 
>>> I'm encountering the following problem:
>> 
>>> When trying to shutdown a PV domU with more than 1 vpcu, the shutdown fails.
>>> The domain keeps running with 100% cpu, and it's still possible to get the
>>> console of this domU with xm console.
>>> When only 1 vcpu is assigned the domain does shutdown.
>> 
>>> Last lines of the PV domU console:
>> 
>>> Debian GNU/Linux 5.0 tv hvc0
>> 
>>> INIT: Switching to runlevel: 0
>>> INIT: Sending processes the TERM signal
>>> Stopping web server: apache2 ... waiting .
>>> Stopping MythTV server: mythbackend No /usr/bin/mythbackend found running;
>>> none killed.
>>> .
>>> Stopping MTA: exim4_listener.
>>> Stopping rsync daemon: rsync.
>>> Stopping MySQL database server: mysqld.
>>> Saving the system clock.
>>> Cannot access the Hardware Clock via any known method.
>>> Use the --debug option to see the details of our search for an access method.
>>> Stopping enhanced syslogd: rsyslogd.
>>> Asking all remaining processes to terminate...done.
>>> All processes ended within 2 seconds....done.
>>> Deconfiguring network interfaces...done.
>>> Cleaning up ifupdown....
>>> Deactivating swap...done.
>>> Unmounting local filesystems...done.
>>> Will now halt.
>>> [ 4336.046876] md: stopping all md devices.
>>> [ 4337.047171] xenbus_dev_shutdown:  trying shutdown of device/console/0:
>>> Initialising
>>> [ 4337.047194] xenbus_dev_shutdown: device/console/0: Initialising !=
>>> Connected, skipping
>>> [ 4337.047200] xenbus_dev_shutdown: result of shutdown of device/console/0:
>>> Initialising
>>> [ 4337.047205] xenbus_dev_shutdown:  trying shutdown of device/vif/0:
>>> Connected
>>> [ 4337.110869] xenbus_dev_shutdown: result of shutdown of device/vif/0:
>>> Closed
>>> [ 4337.110883] xenbus_dev_shutdown:  trying shutdown of device/vbd/51714:
>>> Connected
>>> [ 4337.161975] xenbus_dev_shutdown: result of shutdown of device/vbd/51714:
>>> Closed
>>> [ 4337.161989] xenbus_dev_shutdown:  trying shutdown of device/vbd/51713:
>>> Connected
>>> [ 4337.217136] xenbus_dev_shutdown: result of shutdown of device/vbd/51713:
>>> Closed
>>> [ 4337.217167] System halted.
>> 
>> 
>>> But when using xenstore-ls .. i see that for every domain (but 1 and multiple
>>> vcpu's):
>>>     - All devices have state=4
>>>     - Except all backend = "/local/domain/0/backend/console/*/0" entries,
>>> those have state=1
>>>     - Although xenstore is saying the state is initializing .. xm console
>>> works perfectly for all domains.
>>>     - Perhaps this also explains the high event/0 load in dom0, related to
>>> tty and xenconsoled ?
>> 
>>> DomU kernels vary from debian 2.6.26-xen kernels, to 2.6.37-rc1 mainline.
>>> Xen-unstable-tip and xen-next-2.6.32 dom0
>> 
>> 
>>> --
>>> Sander
>> 
>> 





-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping

[Sander Eikelenboom]

Ahh the difference between console on the other devices seems to be "XENBUS: Device with no driver: device/console/0"
That makes it states "initializing" for ever ...
It then probably goes wrong with the "out:" path in xenbus_dev_shutdown() doing the put_device ?

I only fail to see why it seems to cause a problem with multiple vpcu's assigned and not with just one.

--
Sander


Wednesday, November 17, 2010, 1:06:04 PM, you wrote:

> Consoles do not have a connection handshake. If there is a state field in
> xenstore, it is only unused detritus written by the toolstack
> (xend/libxl/whatever).

>  -- Keir

> On 17/11/2010 11:58, "Sander Eikelenboom" <linux@eikelenboom.it> wrote:

>> Hmm .. i haven't received any response, is there anyone who could point me to
>> the functions involved in communicating the state of the console from
>> "initializing" to "connected" ?
>> That way i could at some additional printk's to find out why the state of domU
>> consoles stays "1" instead of "4" in xenstore.
>> 
>> --
>> Sander
>> 
>> Friday, November 12, 2010, 6:30:10 PM, you wrote:
>> 
>>> I'm encountering the following problem:
>> 
>>> When trying to shutdown a PV domU with more than 1 vpcu, the shutdown fails.
>>> The domain keeps running with 100% cpu, and it's still possible to get the
>>> console of this domU with xm console.
>>> When only 1 vcpu is assigned the domain does shutdown.
>> 
>>> Last lines of the PV domU console:
>> 
>>> Debian GNU/Linux 5.0 tv hvc0
>> 
>>> INIT: Switching to runlevel: 0
>>> INIT: Sending processes the TERM signal
>>> Stopping web server: apache2 ... waiting .
>>> Stopping MythTV server: mythbackend No /usr/bin/mythbackend found running;
>>> none killed.
>>> .
>>> Stopping MTA: exim4_listener.
>>> Stopping rsync daemon: rsync.
>>> Stopping MySQL database server: mysqld.
>>> Saving the system clock.
>>> Cannot access the Hardware Clock via any known method.
>>> Use the --debug option to see the details of our search for an access method.
>>> Stopping enhanced syslogd: rsyslogd.
>>> Asking all remaining processes to terminate...done.
>>> All processes ended within 2 seconds....done.
>>> Deconfiguring network interfaces...done.
>>> Cleaning up ifupdown....
>>> Deactivating swap...done.
>>> Unmounting local filesystems...done.
>>> Will now halt.
>>> [ 4336.046876] md: stopping all md devices.
>>> [ 4337.047171] xenbus_dev_shutdown:  trying shutdown of device/console/0:
>>> Initialising
>>> [ 4337.047194] xenbus_dev_shutdown: device/console/0: Initialising !=
>>> Connected, skipping
>>> [ 4337.047200] xenbus_dev_shutdown: result of shutdown of device/console/0:
>>> Initialising
>>> [ 4337.047205] xenbus_dev_shutdown:  trying shutdown of device/vif/0:
>>> Connected
>>> [ 4337.110869] xenbus_dev_shutdown: result of shutdown of device/vif/0:
>>> Closed
>>> [ 4337.110883] xenbus_dev_shutdown:  trying shutdown of device/vbd/51714:
>>> Connected
>>> [ 4337.161975] xenbus_dev_shutdown: result of shutdown of device/vbd/51714:
>>> Closed
>>> [ 4337.161989] xenbus_dev_shutdown:  trying shutdown of device/vbd/51713:
>>> Connected
>>> [ 4337.217136] xenbus_dev_shutdown: result of shutdown of device/vbd/51713:
>>> Closed
>>> [ 4337.217167] System halted.
>> 
>> 
>>> But when using xenstore-ls .. i see that for every domain (but 1 and multiple
>>> vcpu's):
>>>     - All devices have state=4
>>>     - Except all backend = "/local/domain/0/backend/console/*/0" entries,
>>> those have state=1
>>>     - Although xenstore is saying the state is initializing .. xm console
>>> works perfectly for all domains.
>>>     - Perhaps this also explains the high event/0 load in dom0, related to
>>> tty and xenconsoled ?
>> 
>>> DomU kernels vary from debian 2.6.26-xen kernels, to 2.6.37-rc1 mainline.
>>> Xen-unstable-tip and xen-next-2.6.32 dom0
>> 
>> 
>>> --
>>> Sander
>> 
>> 





-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping

[Keir Fraser]

Our 2.6.18 tree explicitly ignores console devices in
drivers/xen/xenbus/xenbus_probe.c:xenbus_probe_frontend(). More modern
kernels should be doing similar -- I think it's a bug for guest's xenbus
driver to attempt to manage console devices. One of our kernel guys will be
able to say more, no doubt.

 K.

On 17/11/2010 12:28, "Sander Eikelenboom" <linux@eikelenboom.it> wrote:

> Ahh the difference between console on the other devices seems to be "XENBUS:
> Device with no driver: device/console/0"
> That makes it states "initializing" for ever ...
> It then probably goes wrong with the "out:" path in xenbus_dev_shutdown()
> doing the put_device ?
> 
> I only fail to see why it seems to cause a problem with multiple vpcu's
> assigned and not with just one.
> 
> --
> Sander
> 
> 
> Wednesday, November 17, 2010, 1:06:04 PM, you wrote:
> 
>> Consoles do not have a connection handshake. If there is a state field in
>> xenstore, it is only unused detritus written by the toolstack
>> (xend/libxl/whatever).
> 
>>  -- Keir
> 
>> On 17/11/2010 11:58, "Sander Eikelenboom" <linux@eikelenboom.it> wrote:
> 
>>> Hmm .. i haven't received any response, is there anyone who could point me
>>> to
>>> the functions involved in communicating the state of the console from
>>> "initializing" to "connected" ?
>>> That way i could at some additional printk's to find out why the state of
>>> domU
>>> consoles stays "1" instead of "4" in xenstore.
>>> 
>>> --
>>> Sander
>>> 
>>> Friday, November 12, 2010, 6:30:10 PM, you wrote:
>>> 
>>>> I'm encountering the following problem:
>>> 
>>>> When trying to shutdown a PV domU with more than 1 vpcu, the shutdown
>>>> fails.
>>>> The domain keeps running with 100% cpu, and it's still possible to get the
>>>> console of this domU with xm console.
>>>> When only 1 vcpu is assigned the domain does shutdown.
>>> 
>>>> Last lines of the PV domU console:
>>> 
>>>> Debian GNU/Linux 5.0 tv hvc0
>>> 
>>>> INIT: Switching to runlevel: 0
>>>> INIT: Sending processes the TERM signal
>>>> Stopping web server: apache2 ... waiting .
>>>> Stopping MythTV server: mythbackend No /usr/bin/mythbackend found running;
>>>> none killed.
>>>> .
>>>> Stopping MTA: exim4_listener.
>>>> Stopping rsync daemon: rsync.
>>>> Stopping MySQL database server: mysqld.
>>>> Saving the system clock.
>>>> Cannot access the Hardware Clock via any known method.
>>>> Use the --debug option to see the details of our search for an access
>>>> method.
>>>> Stopping enhanced syslogd: rsyslogd.
>>>> Asking all remaining processes to terminate...done.
>>>> All processes ended within 2 seconds....done.
>>>> Deconfiguring network interfaces...done.
>>>> Cleaning up ifupdown....
>>>> Deactivating swap...done.
>>>> Unmounting local filesystems...done.
>>>> Will now halt.
>>>> [ 4336.046876] md: stopping all md devices.
>>>> [ 4337.047171] xenbus_dev_shutdown:  trying shutdown of device/console/0:
>>>> Initialising
>>>> [ 4337.047194] xenbus_dev_shutdown: device/console/0: Initialising !=
>>>> Connected, skipping
>>>> [ 4337.047200] xenbus_dev_shutdown: result of shutdown of device/console/0:
>>>> Initialising
>>>> [ 4337.047205] xenbus_dev_shutdown:  trying shutdown of device/vif/0:
>>>> Connected
>>>> [ 4337.110869] xenbus_dev_shutdown: result of shutdown of device/vif/0:
>>>> Closed
>>>> [ 4337.110883] xenbus_dev_shutdown:  trying shutdown of device/vbd/51714:
>>>> Connected
>>>> [ 4337.161975] xenbus_dev_shutdown: result of shutdown of device/vbd/51714:
>>>> Closed
>>>> [ 4337.161989] xenbus_dev_shutdown:  trying shutdown of device/vbd/51713:
>>>> Connected
>>>> [ 4337.217136] xenbus_dev_shutdown: result of shutdown of device/vbd/51713:
>>>> Closed
>>>> [ 4337.217167] System halted.
>>> 
>>> 
>>>> But when using xenstore-ls .. i see that for every domain (but 1 and
>>>> multiple
>>>> vcpu's):
>>>>     - All devices have state=4
>>>>     - Except all backend = "/local/domain/0/backend/console/*/0" entries,
>>>> those have state=1
>>>>     - Although xenstore is saying the state is initializing .. xm console
>>>> works perfectly for all domains.
>>>>     - Perhaps this also explains the high event/0 load in dom0, related to
>>>> tty and xenconsoled ?
>>> 
>>>> DomU kernels vary from debian 2.6.26-xen kernels, to 2.6.37-rc1 mainline.
>>>> Xen-unstable-tip and xen-next-2.6.32 dom0
>>> 
>>> 
>>>> --
>>>> Sander
>>> 
>>> 
> 
> 
> 
> 

Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping

[Sander Eikelenboom]

Ah yes, 2.6.18 does contain the following line  in xenbus_probe.c and 2.6.37-rc2 not:

-       if (!strcmp(type, "console"))
-               return 0;
It seems to be changed,


There seems to have been a patch http://lists.xensource.com/archives/html/xen-devel/2009-11/msg01072.html but it seems it's not applied to xenbus in linux upstream.

--
Sander




Wednesday, November 17, 2010, 4:36:11 PM, you wrote:

> Our 2.6.18 tree explicitly ignores console devices in
> drivers/xen/xenbus/xenbus_probe.c:xenbus_probe_frontend(). More modern
> kernels should be doing similar -- I think it's a bug for guest's xenbus
> driver to attempt to manage console devices. One of our kernel guys will be
> able to say more, no doubt.

>  K.

> On 17/11/2010 12:28, "Sander Eikelenboom" <linux@eikelenboom.it> wrote:

>> Ahh the difference between console on the other devices seems to be "XENBUS:
>> Device with no driver: device/console/0"
>> That makes it states "initializing" for ever ...
>> It then probably goes wrong with the "out:" path in xenbus_dev_shutdown()
>> doing the put_device ?
>> 
>> I only fail to see why it seems to cause a problem with multiple vpcu's
>> assigned and not with just one.
>> 
>> --
>> Sander
>> 
>> 
>> Wednesday, November 17, 2010, 1:06:04 PM, you wrote:
>> 
>>> Consoles do not have a connection handshake. If there is a state field in
>>> xenstore, it is only unused detritus written by the toolstack
>>> (xend/libxl/whatever).
>> 
>>>  -- Keir
>> 
>>> On 17/11/2010 11:58, "Sander Eikelenboom" <linux@eikelenboom.it> wrote:
>> 
>>>> Hmm .. i haven't received any response, is there anyone who could point me
>>>> to
>>>> the functions involved in communicating the state of the console from
>>>> "initializing" to "connected" ?
>>>> That way i could at some additional printk's to find out why the state of
>>>> domU
>>>> consoles stays "1" instead of "4" in xenstore.
>>>> 
>>>> --
>>>> Sander
>>>> 
>>>> Friday, November 12, 2010, 6:30:10 PM, you wrote:
>>>> 
>>>>> I'm encountering the following problem:
>>>> 
>>>>> When trying to shutdown a PV domU with more than 1 vpcu, the shutdown
>>>>> fails.
>>>>> The domain keeps running with 100% cpu, and it's still possible to get the
>>>>> console of this domU with xm console.
>>>>> When only 1 vcpu is assigned the domain does shutdown.
>>>> 
>>>>> Last lines of the PV domU console:
>>>> 
>>>>> Debian GNU/Linux 5.0 tv hvc0
>>>> 
>>>>> INIT: Switching to runlevel: 0
>>>>> INIT: Sending processes the TERM signal
>>>>> Stopping web server: apache2 ... waiting .
>>>>> Stopping MythTV server: mythbackend No /usr/bin/mythbackend found running;
>>>>> none killed.
>>>>> .
>>>>> Stopping MTA: exim4_listener.
>>>>> Stopping rsync daemon: rsync.
>>>>> Stopping MySQL database server: mysqld.
>>>>> Saving the system clock.
>>>>> Cannot access the Hardware Clock via any known method.
>>>>> Use the --debug option to see the details of our search for an access
>>>>> method.
>>>>> Stopping enhanced syslogd: rsyslogd.
>>>>> Asking all remaining processes to terminate...done.
>>>>> All processes ended within 2 seconds....done.
>>>>> Deconfiguring network interfaces...done.
>>>>> Cleaning up ifupdown....
>>>>> Deactivating swap...done.
>>>>> Unmounting local filesystems...done.
>>>>> Will now halt.
>>>>> [ 4336.046876] md: stopping all md devices.
>>>>> [ 4337.047171] xenbus_dev_shutdown:  trying shutdown of device/console/0:
>>>>> Initialising
>>>>> [ 4337.047194] xenbus_dev_shutdown: device/console/0: Initialising !=
>>>>> Connected, skipping
>>>>> [ 4337.047200] xenbus_dev_shutdown: result of shutdown of device/console/0:
>>>>> Initialising
>>>>> [ 4337.047205] xenbus_dev_shutdown:  trying shutdown of device/vif/0:
>>>>> Connected
>>>>> [ 4337.110869] xenbus_dev_shutdown: result of shutdown of device/vif/0:
>>>>> Closed
>>>>> [ 4337.110883] xenbus_dev_shutdown:  trying shutdown of device/vbd/51714:
>>>>> Connected
>>>>> [ 4337.161975] xenbus_dev_shutdown: result of shutdown of device/vbd/51714:
>>>>> Closed
>>>>> [ 4337.161989] xenbus_dev_shutdown:  trying shutdown of device/vbd/51713:
>>>>> Connected
>>>>> [ 4337.217136] xenbus_dev_shutdown: result of shutdown of device/vbd/51713:
>>>>> Closed
>>>>> [ 4337.217167] System halted.
>>>> 
>>>> 
>>>>> But when using xenstore-ls .. i see that for every domain (but 1 and
>>>>> multiple
>>>>> vcpu's):
>>>>>     - All devices have state=4
>>>>>     - Except all backend = "/local/domain/0/backend/console/*/0" entries,
>>>>> those have state=1
>>>>>     - Although xenstore is saying the state is initializing .. xm console
>>>>> works perfectly for all domains.
>>>>>     - Perhaps this also explains the high event/0 load in dom0, related to
>>>>> tty and xenconsoled ?
>>>> 
>>>>> DomU kernels vary from debian 2.6.26-xen kernels, to 2.6.37-rc1 mainline.
>>>>> Xen-unstable-tip and xen-next-2.6.32 dom0
>>>> 
>>>> 
>>>>> --
>>>>> Sander
>>>> 
>>>> 
>> 
>> 
>> 
>> 





-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping

[Jeremy Fitzhardinge]

On 11/17/2010 12:22 PM, Sander Eikelenboom wrote:
> Ah yes, 2.6.18 does contain the following line  in xenbus_probe.c and 2.6.37-rc2 not:
>
> -       if (!strcmp(type, "console"))
> -               return 0;
> It seems to be changed,
>
>
> There seems to have been a patch http://lists.xensource.com/archives/html/xen-devel/2009-11/msg01072.html but it seems it's not applied to xenbus in linux upstream.

And does this patch fix things for you?

But I have to say that's pretty gross.  Is it really the right thing to do?

    J

Re: Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping

[Sander Eikelenboom]

[-- Attachment #1: Type: text/plain, Size: 2834 bytes --]

Hi Jeremy/Keir,

I have tried to add the following lines:
       if (!strcmp(type, "console"))
               return 0;

But it seems to be a red herring .. although the "the xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping" warning is gone.
The PV domU guest still doesn't get completely halted:
- It stays in a running state
- It uses 100% cpu according to xentop
- I can still connect to it's console

So probably something is stuck in a waiting loop, which doesn't sleep in between and consumes 100% cpu.

Last lines of domU's console(with some additional printk's in xenbus_dev_shutdown):

Cannot access the Hardware Clock via any known method.
Use the --debug option to see the details of our search for an access method.
Stopping enhanced syslogd: rsyslogd.
Asking all remaining processes to terminate...done.
All processes ended within 2 seconds....done.
Deconfiguring network interfaces...done.
Cleaning up ifupdown....
Deactivating swap...done.
Unmounting local filesystems...done.
Will now halt.
[   46.643035] md: stopping all md devices.
[   47.643320] xenbus_dev_shutdown:  trying shutdown of device/vif/0: Connected
[   47.716448] xenbus_dev_shutdown: result of shutdown of device/vif/0: Closed
[   47.716461] xenbus_dev_shutdown:  trying shutdown of device/vbd/51714: Connected
[   47.772293] xenbus_dev_shutdown: result of shutdown of device/vbd/51714: Closed
[   47.772306] xenbus_dev_shutdown:  trying shutdown of device/vbd/51713: Connected
[   47.829384] xenbus_dev_shutdown: result of shutdown of device/vbd/51713: Closed
[   47.829415] System halted.



it's a domU using a 2.6.37-rc2 kernel, file: based disk access, nothing very special.

the only difference that triggers it is:
vcpus=1 works fine ..
vcpus>1 symptoms above ..

I have also added xend.log, with first a boot and shutdown with vpcus=1 and then a boot and shutdown with vpcus=4
I left the domain(with 4vcpus) running with 100% cpu for 10 minutes, but doesn't seem to hit any timeout, and keeps running.
On 20:40 it ends up stuck, on 20:51 i'm shooting the domain off with xm destroy.


Any pointers for adding extra debug info ?

--
Sander




Wednesday, November 17, 2010, 10:57:47 PM, you wrote:

> On 11/17/2010 12:22 PM, Sander Eikelenboom wrote:
>> Ah yes, 2.6.18 does contain the following line  in xenbus_probe.c and 2.6.37-rc2 not:
>>
>> -       if (!strcmp(type, "console"))
>> -               return 0;
>> It seems to be changed,
>>
>>
>> There seems to have been a patch http://lists.xensource.com/archives/html/xen-devel/2009-11/msg01072.html but it seems it's not applied to xenbus in linux upstream.

> And does this patch fix things for you?

> But I have to say that's pretty gross.  Is it really the right thing to do?

>     J

[-- Attachment #2: xend.log --]
[-- Type: application/octet-stream, Size: 22770 bytes --]


[2010-11-18 20:36:41 8088] DEBUG (XendDomainInfo:103) XendDomainInfo.create(['vm', ['name', 'tv'], ['memory', '512'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['vcpus', 1], ['cpu_weight', 512], ['oos', 1], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6.37-rc2-usb-msi+'], ['ramdisk', '/boot/initrd.img-2.6.37-rc2-usb-msi+'], ['root', '/dev/xvda2 ro'], ['videoram', 4], ['tsc_mode', 0], ['nomigrate', 0]]], ['s3_integrity', 1], ['device', ['vbd', ['uname', 'file:/mnt/xen_images/domains/production/tv/swap.img'], ['dev', 'xvda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'file:/mnt/xen_images/domains/production/tv/disk.img'], ['dev', 'xvda2'], ['mode', 'w']]], ['device', ['vif', ['ip', '192.168.1.19'], ['mac', '00:16:3E:FC:1B:70'], ['bridge', 'xen_bridge']]]])
[2010-11-18 20:36:41 8088] DEBUG (XendDomainInfo:2498) XendDomainInfo.constructDomain
[2010-11-18 20:36:41 8088] DEBUG (balloon:187) Balloon: 3573972 KiB free; need 16384; done.
[2010-11-18 20:36:41 8088] DEBUG (XendDomain:476) Adding Domain: 28
[2010-11-18 20:36:41 8088] DEBUG (XendDomainInfo:2836) XendDomainInfo.initDomain: 28 512
[2010-11-18 20:36:41 8088] DEBUG (XendDomainInfo:2863) _initDomain:shadow_memory=0x0, memory_static_max=0x20000000, memory_static_min=0x0.
[2010-11-18 20:36:41 8088] INFO (image:182) buildDomain os=linux dom=28 vcpus=1
[2010-11-18 20:36:41 8088] DEBUG (image:721) domid          = 28
[2010-11-18 20:36:41 8088] DEBUG (image:722) memsize        = 512
[2010-11-18 20:36:41 8088] DEBUG (image:723) image          = /boot/vmlinuz-2.6.37-rc2-usb-msi+
[2010-11-18 20:36:41 8088] DEBUG (image:724) store_evtchn   = 1
[2010-11-18 20:36:41 8088] DEBUG (image:725) console_evtchn = 2
[2010-11-18 20:36:41 8088] DEBUG (image:726) cmdline        = root=/dev/xvda2 ro 
[2010-11-18 20:36:41 8088] DEBUG (image:727) ramdisk        = /boot/initrd.img-2.6.37-rc2-usb-msi+
[2010-11-18 20:36:41 8088] DEBUG (image:728) vcpus          = 1
[2010-11-18 20:36:41 8088] DEBUG (image:729) features       = 
[2010-11-18 20:36:41 8088] DEBUG (image:730) flags          = 0
[2010-11-18 20:36:41 8088] DEBUG (image:731) superpages     = 0
[2010-11-18 20:36:42 8088] INFO (XendDomainInfo:2357) createDevice: vbd : {'uuid': '97021dbe-97ed-5aec-9c96-db8ff642d556', 'bootable': 1, 'driver': 'paravirtualised', 'dev': 'xvda1', 'uname': 'file:/mnt/xen_images/domains/production/tv/swap.img', 'mode': 'w'}
[2010-11-18 20:36:42 8088] DEBUG (DevController:95) DevController: writing {'virtual-device': '51713', 'device-type': 'disk', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/28/51713'} to /local/domain/28/device/vbd/51713.
[2010-11-18 20:36:42 8088] DEBUG (DevController:97) DevController: writing {'domain': 'tv', 'frontend': '/local/domain/28/device/vbd/51713', 'uuid': '97021dbe-97ed-5aec-9c96-db8ff642d556', 'bootable': '1', 'dev': 'xvda1', 'state': '1', 'params': '/mnt/xen_images/domains/production/tv/swap.img', 'mode': 'w', 'online': '1', 'frontend-id': '28', 'type': 'file'} to /local/domain/0/backend/vbd/28/51713.
[2010-11-18 20:36:42 8088] INFO (XendDomainInfo:2357) createDevice: vbd : {'uuid': '4a5cf55f-6a71-a01e-accd-cd134e20a477', 'bootable': 0, 'driver': 'paravirtualised', 'dev': 'xvda2', 'uname': 'file:/mnt/xen_images/domains/production/tv/disk.img', 'mode': 'w'}
[2010-11-18 20:36:42 8088] DEBUG (DevController:95) DevController: writing {'virtual-device': '51714', 'device-type': 'disk', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/28/51714'} to /local/domain/28/device/vbd/51714.
[2010-11-18 20:36:42 8088] DEBUG (DevController:97) DevController: writing {'domain': 'tv', 'frontend': '/local/domain/28/device/vbd/51714', 'uuid': '4a5cf55f-6a71-a01e-accd-cd134e20a477', 'bootable': '0', 'dev': 'xvda2', 'state': '1', 'params': '/mnt/xen_images/domains/production/tv/disk.img', 'mode': 'w', 'online': '1', 'frontend-id': '28', 'type': 'file'} to /local/domain/0/backend/vbd/28/51714.
[2010-11-18 20:36:42 8088] INFO (XendDomainInfo:2357) createDevice: vif : {'ip': '192.168.1.19', 'mac': '00:16:3E:FC:1B:70', 'uuid': 'aedb3b76-1836-db64-2adc-5d8112402631', 'bridge': 'xen_bridge'}
[2010-11-18 20:36:42 8088] DEBUG (DevController:95) DevController: writing {'mac': '00:16:3E:FC:1B:70', 'handle': '0', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/28/0'} to /local/domain/28/device/vif/0.
[2010-11-18 20:36:42 8088] DEBUG (DevController:97) DevController: writing {'bridge': 'xen_bridge', 'domain': 'tv', 'handle': '0', 'uuid': 'aedb3b76-1836-db64-2adc-5d8112402631', 'script': '/etc/xen/scripts/vif-bridge', 'ip': '192.168.1.19', 'mac': '00:16:3E:FC:1B:70', 'frontend-id': '28', 'state': '1', 'online': '1', 'frontend': '/local/domain/28/device/vif/0'} to /local/domain/0/backend/vif/28/0.
[2010-11-18 20:36:43 8088] DEBUG (XendDomainInfo:3421) Storing VM details: {'on_xend_stop': 'ignore', 'pool_name': 'Pool-0', 'shadow_memory': '0', 'uuid': 'f69c83a2-1a9d-32a3-9091-dcbdb3b201ca', 'on_reboot': 'restart', 'start_time': '1290109003.26', 'on_poweroff': 'destroy', 'bootloader_args': '', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'bootloader': '', 'image': "(linux (kernel '/boot/vmlinuz-2.6.37-rc2-usb-msi+') (ramdisk '/boot/initrd.img-2.6.37-rc2-usb-msi+') (args 'root=/dev/xvda2 ro ') (superpages 0) (tsc_mode 0) (videoram 4) (pci ()) (nomigrate 0) (notes (HV_START_LOW 18446603336221196288) (FEATURES '!writable_page_tables|pae_pgdir_above_4gb') (VIRT_BASE 18446744071562067968) (GUEST_VERSION 2.6) (PADDR_OFFSET 0) (GUEST_OS linux) (HYPERCALL_PAGE 18446744071578849280) (LOADER generic) (SUSPEND_CANCEL 1) (PAE_MODE yes) (ENTRY 18446744071592509952) (XEN_VERSION xen-3.0)))", 'name': 'tv'}
[2010-11-18 20:36:43 8088] DEBUG (XendDomainInfo:1794) Storing domain details: {'console/ring-ref': '2023316', 'image/entry': '18446744071592509952', 'console/port': '2', 'store/ring-ref': '2023317', 'image/loader': 'generic', 'vm': '/vm/f69c83a2-1a9d-32a3-9091-dcbdb3b201ca', 'control/platform-feature-multiprocessor-suspend': '1', 'image/hv-start-low': '18446603336221196288', 'image/guest-os': 'linux', 'image/virt-base': '18446744071562067968', 'memory/target': '524288', 'image/guest-version': '2.6', 'image/pae-mode': 'yes', 'description': '', 'console/limit': '1048576', 'image/paddr-offset': '0', 'image/hypercall-page': '18446744071578849280', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', 'image/features/writable-page-tables': '0', 'console/type': 'xenconsoled', 'name': 'tv', 'domid': '28', 'image/xen-version': 'xen-3.0', 'store/port': '1'}
[2010-11-18 20:36:44 8088] DEBUG (DevController:95) DevController: writing {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/console/28/0'} to /local/domain/28/device/console/0.
[2010-11-18 20:36:44 8088] DEBUG (DevController:97) DevController: writing {'domain': 'tv', 'frontend': '/local/domain/28/device/console/0', 'uuid': '50f10a66-c57e-7432-871d-5ecfa86ae286', 'frontend-id': '28', 'state': '1', 'location': '2', 'online': '1', 'protocol': 'vt100'} to /local/domain/0/backend/console/28/0.
[2010-11-18 20:36:44 8088] DEBUG (XendDomainInfo:1881) XendDomainInfo.handleShutdownWatch
[2010-11-18 20:36:44 8088] DEBUG (DevController:139) Waiting for devices vif2.
[2010-11-18 20:36:44 8088] DEBUG (DevController:139) Waiting for devices vif.
[2010-11-18 20:36:44 8088] DEBUG (DevController:144) Waiting for 0.
[2010-11-18 20:36:44 8088] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vif/28/0/hotplug-status.
[2010-11-18 20:36:44 8088] DEBUG (DevController:642) hotplugStatusCallback 1.
[2010-11-18 20:36:44 8088] DEBUG (DevController:139) Waiting for devices vscsi.
[2010-11-18 20:36:44 8088] DEBUG (DevController:139) Waiting for devices vbd.
[2010-11-18 20:36:44 8088] DEBUG (DevController:144) Waiting for 51713.
[2010-11-18 20:36:44 8088] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vbd/28/51713/hotplug-status.
[2010-11-18 20:36:44 8088] DEBUG (DevController:642) hotplugStatusCallback 1.
[2010-11-18 20:36:44 8088] DEBUG (DevController:144) Waiting for 51714.
[2010-11-18 20:36:44 8088] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vbd/28/51714/hotplug-status.
[2010-11-18 20:36:44 8088] DEBUG (DevController:642) hotplugStatusCallback 1.
[2010-11-18 20:36:45 8088] DEBUG (DevController:139) Waiting for devices ioports.
[2010-11-18 20:36:45 8088] DEBUG (DevController:139) Waiting for devices irq.
[2010-11-18 20:36:45 8088] DEBUG (DevController:139) Waiting for devices vkbd.
[2010-11-18 20:36:45 8088] DEBUG (DevController:139) Waiting for devices vfb.
[2010-11-18 20:36:45 8088] DEBUG (DevController:139) Waiting for devices console.
[2010-11-18 20:36:45 8088] DEBUG (DevController:144) Waiting for 0.
[2010-11-18 20:36:45 8088] DEBUG (DevController:139) Waiting for devices pci.
[2010-11-18 20:36:45 8088] DEBUG (DevController:139) Waiting for devices tap2.
[2010-11-18 20:36:45 8088] DEBUG (DevController:139) Waiting for devices tap.
[2010-11-18 20:36:45 8088] DEBUG (DevController:139) Waiting for devices vusb.
[2010-11-18 20:36:45 8088] DEBUG (DevController:139) Waiting for devices vtpm.
[2010-11-18 20:36:45 8088] INFO (XendDomain:1225) Domain tv (28) unpaused.
[2010-11-18 20:37:45 8088] DEBUG (XendDomainInfo:524) XendDomainInfo.shutdown(poweroff)
[2010-11-18 20:37:45 8088] DEBUG (XendDomainInfo:1881) XendDomainInfo.handleShutdownWatch
[2010-11-18 20:37:45 8088] DEBUG (XendDomainInfo:1881) XendDomainInfo.handleShutdownWatch
[2010-11-18 20:37:52 8088] INFO (XendDomainInfo:2078) Domain has shutdown: name=tv id=28 reason=poweroff.
[2010-11-18 20:37:53 8088] DEBUG (XendDomainInfo:3071) XendDomainInfo.destroy: domid=28
[2010-11-18 20:37:53 8088] DEBUG (XendDomainInfo:2401) Destroying device model
[2010-11-18 20:37:53 8088] DEBUG (XendDomainInfo:2408) Releasing devices
[2010-11-18 20:37:53 8088] DEBUG (XendDomainInfo:2414) Removing vif/0
[2010-11-18 20:37:53 8088] DEBUG (XendDomainInfo:1276) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0
[2010-11-18 20:37:53 8088] DEBUG (XendDomainInfo:2414) Removing vbd/51713
[2010-11-18 20:37:53 8088] DEBUG (XendDomainInfo:1276) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51713
[2010-11-18 20:37:53 8088] DEBUG (XendDomainInfo:2414) Removing vbd/51714
[2010-11-18 20:37:53 8088] DEBUG (XendDomainInfo:1276) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51714
[2010-11-18 20:37:53 8088] DEBUG (XendDomainInfo:2414) Removing console/0
[2010-11-18 20:37:53 8088] DEBUG (XendDomainInfo:1276) XendDomainInfo.destroyDevice: deviceClass = console, device = console/0
######################################
[2010-11-18 20:38:35 8088] DEBUG (XendDomainInfo:103) XendDomainInfo.create(['vm', ['name', 'tv'], ['memory', '512'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['vcpus', 4], ['cpu_weight', 512], ['oos', 1], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6.37-rc2-usb-msi+'], ['ramdisk', '/boot/initrd.img-2.6.37-rc2-usb-msi+'], ['root', '/dev/xvda2 ro'], ['videoram', 4], ['tsc_mode', 0], ['nomigrate', 0]]], ['s3_integrity', 1], ['device', ['vbd', ['uname', 'file:/mnt/xen_images/domains/production/tv/swap.img'], ['dev', 'xvda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'file:/mnt/xen_images/domains/production/tv/disk.img'], ['dev', 'xvda2'], ['mode', 'w']]], ['device', ['vif', ['ip', '192.168.1.19'], ['mac', '00:16:3E:FC:1B:70'], ['bridge', 'xen_bridge']]]])
[2010-11-18 20:38:35 8088] DEBUG (XendDomainInfo:2498) XendDomainInfo.constructDomain
[2010-11-18 20:38:35 8088] DEBUG (balloon:187) Balloon: 3573972 KiB free; need 16384; done.
[2010-11-18 20:38:35 8088] DEBUG (XendDomain:476) Adding Domain: 29
[2010-11-18 20:38:35 8088] DEBUG (XendDomainInfo:2836) XendDomainInfo.initDomain: 29 512
[2010-11-18 20:38:35 8088] DEBUG (XendDomainInfo:2863) _initDomain:shadow_memory=0x0, memory_static_max=0x20000000, memory_static_min=0x0.
[2010-11-18 20:38:35 8088] INFO (image:182) buildDomain os=linux dom=29 vcpus=4
[2010-11-18 20:38:35 8088] DEBUG (image:721) domid          = 29
[2010-11-18 20:38:35 8088] DEBUG (image:722) memsize        = 512
[2010-11-18 20:38:35 8088] DEBUG (image:723) image          = /boot/vmlinuz-2.6.37-rc2-usb-msi+
[2010-11-18 20:38:35 8088] DEBUG (image:724) store_evtchn   = 1
[2010-11-18 20:38:35 8088] DEBUG (image:725) console_evtchn = 2
[2010-11-18 20:38:35 8088] DEBUG (image:726) cmdline        = root=/dev/xvda2 ro 
[2010-11-18 20:38:35 8088] DEBUG (image:727) ramdisk        = /boot/initrd.img-2.6.37-rc2-usb-msi+
[2010-11-18 20:38:35 8088] DEBUG (image:728) vcpus          = 4
[2010-11-18 20:38:35 8088] DEBUG (image:729) features       = 
[2010-11-18 20:38:35 8088] DEBUG (image:730) flags          = 0
[2010-11-18 20:38:35 8088] DEBUG (image:731) superpages     = 0
[2010-11-18 20:38:36 8088] INFO (XendDomainInfo:2357) createDevice: vbd : {'uuid': '14420801-6116-4cd9-b8a7-e2147c56348a', 'bootable': 1, 'driver': 'paravirtualised', 'dev': 'xvda1', 'uname': 'file:/mnt/xen_images/domains/production/tv/swap.img', 'mode': 'w'}
[2010-11-18 20:38:36 8088] DEBUG (DevController:95) DevController: writing {'virtual-device': '51713', 'device-type': 'disk', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/29/51713'} to /local/domain/29/device/vbd/51713.
[2010-11-18 20:38:36 8088] DEBUG (DevController:97) DevController: writing {'domain': 'tv', 'frontend': '/local/domain/29/device/vbd/51713', 'uuid': '14420801-6116-4cd9-b8a7-e2147c56348a', 'bootable': '1', 'dev': 'xvda1', 'state': '1', 'params': '/mnt/xen_images/domains/production/tv/swap.img', 'mode': 'w', 'online': '1', 'frontend-id': '29', 'type': 'file'} to /local/domain/0/backend/vbd/29/51713.
[2010-11-18 20:38:36 8088] INFO (XendDomainInfo:2357) createDevice: vbd : {'uuid': '2869c086-04eb-8e5e-2f62-c48b3ce4f75b', 'bootable': 0, 'driver': 'paravirtualised', 'dev': 'xvda2', 'uname': 'file:/mnt/xen_images/domains/production/tv/disk.img', 'mode': 'w'}
[2010-11-18 20:38:36 8088] DEBUG (DevController:95) DevController: writing {'virtual-device': '51714', 'device-type': 'disk', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/29/51714'} to /local/domain/29/device/vbd/51714.
[2010-11-18 20:38:36 8088] DEBUG (DevController:97) DevController: writing {'domain': 'tv', 'frontend': '/local/domain/29/device/vbd/51714', 'uuid': '2869c086-04eb-8e5e-2f62-c48b3ce4f75b', 'bootable': '0', 'dev': 'xvda2', 'state': '1', 'params': '/mnt/xen_images/domains/production/tv/disk.img', 'mode': 'w', 'online': '1', 'frontend-id': '29', 'type': 'file'} to /local/domain/0/backend/vbd/29/51714.
[2010-11-18 20:38:36 8088] INFO (XendDomainInfo:2357) createDevice: vif : {'ip': '192.168.1.19', 'mac': '00:16:3E:FC:1B:70', 'uuid': 'cb1440db-166e-6eaf-ca5c-8817e98a025d', 'bridge': 'xen_bridge'}
[2010-11-18 20:38:36 8088] DEBUG (DevController:95) DevController: writing {'mac': '00:16:3E:FC:1B:70', 'handle': '0', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/29/0'} to /local/domain/29/device/vif/0.
[2010-11-18 20:38:36 8088] DEBUG (DevController:97) DevController: writing {'bridge': 'xen_bridge', 'domain': 'tv', 'handle': '0', 'uuid': 'cb1440db-166e-6eaf-ca5c-8817e98a025d', 'script': '/etc/xen/scripts/vif-bridge', 'ip': '192.168.1.19', 'mac': '00:16:3E:FC:1B:70', 'frontend-id': '29', 'state': '1', 'online': '1', 'frontend': '/local/domain/29/device/vif/0'} to /local/domain/0/backend/vif/29/0.
[2010-11-18 20:38:37 8088] DEBUG (XendDomainInfo:3421) Storing VM details: {'on_xend_stop': 'ignore', 'pool_name': 'Pool-0', 'shadow_memory': '0', 'uuid': 'fd9fd9f2-8a01-7993-b06d-d6a3e9e233cd', 'on_reboot': 'restart', 'start_time': '1290109117.32', 'on_poweroff': 'destroy', 'bootloader_args': '', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15', 'bootloader': '', 'image': "(linux (kernel '/boot/vmlinuz-2.6.37-rc2-usb-msi+') (ramdisk '/boot/initrd.img-2.6.37-rc2-usb-msi+') (args 'root=/dev/xvda2 ro ') (superpages 0) (tsc_mode 0) (videoram 4) (pci ()) (nomigrate 0) (notes (HV_START_LOW 18446603336221196288) (FEATURES '!writable_page_tables|pae_pgdir_above_4gb') (VIRT_BASE 18446744071562067968) (GUEST_VERSION 2.6) (PADDR_OFFSET 0) (GUEST_OS linux) (HYPERCALL_PAGE 18446744071578849280) (LOADER generic) (SUSPEND_CANCEL 1) (PAE_MODE yes) (ENTRY 18446744071592509952) (XEN_VERSION xen-3.0)))", 'name': 'tv'}
[2010-11-18 20:38:37 8088] DEBUG (XendDomainInfo:1794) Storing domain details: {'console/ring-ref': '1986452', 'image/entry': '18446744071592509952', 'console/port': '2', 'cpu/3/availability': 'online', 'store/ring-ref': '1986453', 'image/loader': 'generic', 'vm': '/vm/fd9fd9f2-8a01-7993-b06d-d6a3e9e233cd', 'control/platform-feature-multiprocessor-suspend': '1', 'image/hv-start-low': '18446603336221196288', 'description': '', 'cpu/2/availability': 'online', 'cpu/1/availability': 'online', 'image/virt-base': '18446744071562067968', 'memory/target': '524288', 'image/guest-version': '2.6', 'image/pae-mode': 'yes', 'image/guest-os': 'linux', 'console/limit': '1048576', 'image/paddr-offset': '0', 'image/hypercall-page': '18446744071578849280', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', 'image/features/writable-page-tables': '0', 'console/type': 'xenconsoled', 'name': 'tv', 'domid': '29', 'image/xen-version': 'xen-3.0', 'store/port': '1'}
[2010-11-18 20:38:38 8088] DEBUG (DevController:95) DevController: writing {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/console/29/0'} to /local/domain/29/device/console/0.
[2010-11-18 20:38:38 8088] DEBUG (DevController:97) DevController: writing {'domain': 'tv', 'frontend': '/local/domain/29/device/console/0', 'uuid': '52ad36ea-7d3b-5e54-c9ee-0bb0605d7dca', 'frontend-id': '29', 'state': '1', 'location': '2', 'online': '1', 'protocol': 'vt100'} to /local/domain/0/backend/console/29/0.
[2010-11-18 20:38:39 8088] DEBUG (XendDomainInfo:1881) XendDomainInfo.handleShutdownWatch
[2010-11-18 20:38:39 8088] DEBUG (DevController:139) Waiting for devices vif2.
[2010-11-18 20:38:39 8088] DEBUG (DevController:139) Waiting for devices vif.
[2010-11-18 20:38:39 8088] DEBUG (DevController:144) Waiting for 0.
[2010-11-18 20:38:39 8088] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vif/29/0/hotplug-status.
[2010-11-18 20:38:39 8088] DEBUG (DevController:642) hotplugStatusCallback 1.
[2010-11-18 20:38:39 8088] DEBUG (DevController:139) Waiting for devices vscsi.
[2010-11-18 20:38:39 8088] DEBUG (DevController:139) Waiting for devices vbd.
[2010-11-18 20:38:39 8088] DEBUG (DevController:144) Waiting for 51713.
[2010-11-18 20:38:39 8088] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vbd/29/51713/hotplug-status.
[2010-11-18 20:38:39 8088] DEBUG (DevController:642) hotplugStatusCallback 1.
[2010-11-18 20:38:39 8088] DEBUG (DevController:144) Waiting for 51714.
[2010-11-18 20:38:39 8088] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vbd/29/51714/hotplug-status.
[2010-11-18 20:38:39 8088] DEBUG (DevController:642) hotplugStatusCallback 1.
[2010-11-18 20:38:39 8088] DEBUG (DevController:139) Waiting for devices ioports.
[2010-11-18 20:38:39 8088] DEBUG (DevController:139) Waiting for devices irq.
[2010-11-18 20:38:39 8088] DEBUG (DevController:139) Waiting for devices vkbd.
[2010-11-18 20:38:39 8088] DEBUG (DevController:139) Waiting for devices vfb.
[2010-11-18 20:38:39 8088] DEBUG (DevController:139) Waiting for devices console.
[2010-11-18 20:38:39 8088] DEBUG (DevController:144) Waiting for 0.
[2010-11-18 20:38:39 8088] DEBUG (DevController:139) Waiting for devices pci.
[2010-11-18 20:38:39 8088] DEBUG (DevController:139) Waiting for devices tap2.
[2010-11-18 20:38:39 8088] DEBUG (DevController:139) Waiting for devices tap.
[2010-11-18 20:38:39 8088] DEBUG (DevController:139) Waiting for devices vusb.
[2010-11-18 20:38:39 8088] DEBUG (DevController:139) Waiting for devices vtpm.
[2010-11-18 20:38:39 8088] INFO (XendDomain:1225) Domain tv (29) unpaused.
[2010-11-18 20:39:18 8088] DEBUG (XendDomainInfo:524) XendDomainInfo.shutdown(poweroff)
[2010-11-18 20:39:18 8088] DEBUG (XendDomainInfo:1881) XendDomainInfo.handleShutdownWatch
[2010-11-18 20:39:18 8088] DEBUG (XendDomainInfo:1881) XendDomainInfo.handleShutdownWatch
[2010-11-18 20:40:48 8088] DEBUG (XendDomainInfo:2406) No device model
[2010-11-18 20:40:48 8088] DEBUG (XendDomainInfo:2408) Releasing devices
[2010-11-18 20:51:37 8088] DEBUG (XendDomainInfo:3071) XendDomainInfo.destroy: domid=29
[2010-11-18 20:51:37 8088] DEBUG (XendDomainInfo:2401) Destroying device model
[2010-11-18 20:51:37 8088] DEBUG (XendDomainInfo:2408) Releasing devices
[2010-11-18 20:51:37 8088] DEBUG (XendDomainInfo:2414) Removing vif/0
[2010-11-18 20:51:37 8088] DEBUG (XendDomainInfo:1276) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0
[2010-11-18 20:51:37 8088] DEBUG (XendDomainInfo:2414) Removing vbd/51713
[2010-11-18 20:51:37 8088] DEBUG (XendDomainInfo:1276) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51713
[2010-11-18 20:51:37 8088] DEBUG (XendDomainInfo:2414) Removing vbd/51714
[2010-11-18 20:51:37 8088] DEBUG (XendDomainInfo:1276) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51714
[2010-11-18 20:51:37 8088] DEBUG (XendDomainInfo:2414) Removing console/0
[2010-11-18 20:51:37 8088] DEBUG (XendDomainInfo:1276) XendDomainInfo.destroyDevice: deviceClass = console, device = console/0
[2010-11-18 20:51:37 8088] DEBUG (XendDomainInfo:2406) No device model
[2010-11-18 20:51:37 8088] DEBUG (XendDomainInfo:2408) Releasing devices
[2010-11-18 20:51:37 8088] DEBUG (XendDomainInfo:2414) Removing vif/0
[2010-11-18 20:51:37 8088] DEBUG (XendDomainInfo:1276) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0
[2010-11-18 20:51:37 8088] DEBUG (XendDomainInfo:2414) Removing vbd/51713
[2010-11-18 20:51:37 8088] DEBUG (XendDomainInfo:1276) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51713
[2010-11-18 20:51:37 8088] DEBUG (XendDomainInfo:2414) Removing vbd/51714
[2010-11-18 20:51:37 8088] DEBUG (XendDomainInfo:1276) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51714

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping

[Jeremy Fitzhardinge]

On 11/18/2010 11:57 AM, Sander Eikelenboom wrote:
> Hi Jeremy/Keir,
>
> I have tried to add the following lines:
>        if (!strcmp(type, "console"))
>                return 0;
>
> But it seems to be a red herring .. although the "the xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping" warning is gone.
> The PV domU guest still doesn't get completely halted:
> - It stays in a running state
> - It uses 100% cpu according to xentop
> - I can still connect to it's console
>
> So probably something is stuck in a waiting loop, which doesn't sleep in between and consumes 100% cpu.

Hm, I generally see clean shutdowns of my multi-VCPU domains.

Can you do

# gdbsx -a <domid> <arch-size> 9999        (arch-size is 32 or 64)

then

$ gdb vmlinux
(gdb) tar rem :9999
(gdb) thread apply all bt

to see what's going on?  (You may need to enable CONFIG_DEBUG_INFO to
get useful info if you haven't already.)

    J


> Last lines of domU's console(with some additional printk's in xenbus_dev_shutdown):
>
> Cannot access the Hardware Clock via any known method.
> Use the --debug option to see the details of our search for an access method.
> Stopping enhanced syslogd: rsyslogd.
> Asking all remaining processes to terminate...done.
> All processes ended within 2 seconds....done.
> Deconfiguring network interfaces...done.
> Cleaning up ifupdown....
> Deactivating swap...done.
> Unmounting local filesystems...done.
> Will now halt.
> [   46.643035] md: stopping all md devices.
> [   47.643320] xenbus_dev_shutdown:  trying shutdown of device/vif/0: Connected
> [   47.716448] xenbus_dev_shutdown: result of shutdown of device/vif/0: Closed
> [   47.716461] xenbus_dev_shutdown:  trying shutdown of device/vbd/51714: Connected
> [   47.772293] xenbus_dev_shutdown: result of shutdown of device/vbd/51714: Closed
> [   47.772306] xenbus_dev_shutdown:  trying shutdown of device/vbd/51713: Connected
> [   47.829384] xenbus_dev_shutdown: result of shutdown of device/vbd/51713: Closed
> [   47.829415] System halted.
>
>
>
> it's a domU using a 2.6.37-rc2 kernel, file: based disk access, nothing very special.
>
> the only difference that triggers it is:
> vcpus=1 works fine ..
> vcpus>1 symptoms above ..
>
> I have also added xend.log, with first a boot and shutdown with vpcus=1 and then a boot and shutdown with vpcus=4
> I left the domain(with 4vcpus) running with 100% cpu for 10 minutes, but doesn't seem to hit any timeout, and keeps running.
> On 20:40 it ends up stuck, on 20:51 i'm shooting the domain off with xm destroy.
>
>
> Any pointers for adding extra debug info ?
>
> --
> Sander
>
>
>
>
> Wednesday, November 17, 2010, 10:57:47 PM, you wrote:
>
>> On 11/17/2010 12:22 PM, Sander Eikelenboom wrote:
>>> Ah yes, 2.6.18 does contain the following line  in xenbus_probe.c and 2.6.37-rc2 not:
>>>
>>> -       if (!strcmp(type, "console"))
>>> -               return 0;
>>> It seems to be changed,
>>>
>>>
>>> There seems to have been a patch http://lists.xensource.com/archives/html/xen-devel/2009-11/msg01072.html but it seems it's not applied to xenbus in linux upstream.
>> And does this patch fix things for you?
>> But I have to say that's pretty gross.  Is it really the right thing to do?
>>     J
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel

Re: Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping

[Sander Eikelenboom]

Sorry, something went wrong with copy and pasting, here is the complete output:

(gdb) tar rem :9999
Remote debugging using :9999
[New Thread 0]
[Switching to Thread 0]
smp_call_function_many (mask=0xffffffff81ce8e70, func=0xffffffff81007c4d <stop_self>, info=0x0, wait=true) at kernel/smp.c:104
104             while (data->flags & CSD_FLAG_LOCK)
(gdb) thread apply all bt
[New Thread 1]
[New Thread 2]
[New Thread 3]

Thread 4 (Thread 3):
#0  0xffffffff8100130a in hypercall_page ()
#1  0xffffffffffff02dd in ?? ()
#2  0x0000000000000080 in irq_stack_union ()
#3  0xffffffff81007c9a in stop_self (v=<value optimized out>) at /usr/src/new/linux-2.6-usb-msi/arch/x86/include/asm/xen/hypercall.h:413
#4  0xffffffff81077adf in generic_smp_call_function_interrupt () at kernel/smp.c:200
#5  0xffffffff81007eb2 in xen_call_function_interrupt (irq=<value optimized out>, dev_id=<value optimized out>) at arch/x86/xen/smp.c:472
#6  0xffffffff8109907b in handle_IRQ_event (irq=286, action=0xffff88001e8257e0) at kernel/irq/handle.c:68
#7  0xffffffff8109ae9c in handle_percpu_irq (irq=286, desc=0xffff88001ea05500) at kernel/irq/chip.c:684
#8  0xffffffff8127a3b0 in __xen_evtchn_do_upcall () at include/linux/irqdesc.h:119
#9  0xffffffff8127ab80 in xen_evtchn_do_upcall (regs=<value optimized out>) at drivers/xen/events.c:1143
#10 0xffffffff8100bc2e in xen_do_hypervisor_callback () at arch/x86/kernel/entry_64.S:1233
#11 0xffff88001e9e3e28 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 3 (Thread 2):
#0  0xffffffff8100130a in hypercall_page ()
#1  0xffffffff81006fbd in xen_force_evtchn_callback () at /usr/src/new/linux-2.6-usb-msi/arch/x86/include/asm/xen/hypercall.h:362
#2  0xffffffff81077adf in generic_smp_call_function_interrupt () at kernel/smp.c:200
#3  0xffffffff81007eb2 in xen_call_function_interrupt (irq=<value optimized out>, dev_id=<value optimized out>) at arch/x86/xen/smp.c:472
#4  0xffffffff8109907b in handle_IRQ_event (irq=291, action=0xffff88001e825600) at kernel/irq/handle.c:68
#5  0xffffffff8109ae9c in handle_percpu_irq (irq=291, desc=0xffff88001ea05000) at kernel/irq/chip.c:684
#6  0xffffffff8127a3b0 in __xen_evtchn_do_upcall () at include/linux/irqdesc.h:119
#7  0xffffffff8127ab80 in xen_evtchn_do_upcall (regs=<value optimized out>) at drivers/xen/events.c:1143
#8  0xffffffff8100bc2e in xen_do_hypervisor_callback () at arch/x86/kernel/entry_64.S:1233
#9  0xffff88001e9e1e28 in ?? ()
#10 0x0000000000000000 in ?? ()
Current language:  auto; currently asm

Thread 2 (Thread 1):
#0  0xffffffff8100130a in hypercall_page ()
#1  0xffffffff81ce8e70 in cpu_possible_bits ()
#2  0x0000000000000080 in irq_stack_union ()
#3  0xffffffff81007c9a in stop_self (v=<value optimized out>) at /usr/src/new/linux-2.6-usb-msi/arch/x86/include/asm/xen/hypercall.h:413
#4  0xffffffff81077adf in generic_smp_call_function_interrupt () at kernel/smp.c:200
#5  0xffffffff81007eb2 in xen_call_function_interrupt (irq=<value optimized out>, dev_id=<value optimized out>) at arch/x86/xen/smp.c:472
#6  0xffffffff8109907b in handle_IRQ_event (irq=296, action=0xffff88001e825420) at kernel/irq/handle.c:68
#7  0xffffffff8109ae9c in handle_percpu_irq (irq=296, desc=0xffff88001e824b00) at kernel/irq/chip.c:684
#8  0xffffffff8127a3b0 in __xen_evtchn_do_upcall () at include/linux/irqdesc.h:119
#9  0xffffffff8127ab80 in xen_evtchn_do_upcall (regs=<value optimized out>) at drivers/xen/events.c:1143
#10 0xffffffff8100bc2e in xen_do_hypervisor_callback () at arch/x86/kernel/entry_64.S:1233
#11 0xffff88001e9d7e28 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 1 (Thread 0):
#0  smp_call_function_many (mask=0xffffffff81ce8e70, func=0xffffffff81007c4d <stop_self>, info=0x0, wait=true) at kernel/smp.c:104
#1  0xffffffff81077a5a in smp_call_function (func=0x40 <irq_stack_union+64>, info=<value optimized out>, wait=<value optimized out>) at kernel/smp.c:506
#2  0xffffffff81007eda in xen_stop_other_cpus (wait=<value optimized out>) at arch/x86/xen/smp.c:431
#3  0xffffffff81003172 in xen_reboot () at /usr/src/new/linux-2.6-usb-msi/arch/x86/include/asm/smp.h:81
#4  0xffffffff810031b5 in xen_machine_halt () at arch/x86/xen/enlighten.c:1039
#5  0xffffffff81023795 in machine_halt () at arch/x86/kernel/reboot.c:726
#6  0xffffffff8105e9b3 in kernel_halt () at kernel/sys.c:336
#7  0xffffffff8105eb03 in sys_reboot (magic1=-18751827, magic2=672274793, cmd=3454992675, arg=0x0) at kernel/sys.c:407
#8  0xffffffff8100acc2 in system_call_fastpath () at arch/x86/kernel/entry_64.S:479
#9  0x0000000000000202 in irq_stack_union ()
#10 0x0000000000000000 in ?? ()
Current language:  auto; currently c


Friday, November 19, 2010, 6:07:43 PM, you wrote:

> On 11/18/2010 11:57 AM, Sander Eikelenboom wrote:
>> Hi Jeremy/Keir,
>>
>> I have tried to add the following lines:
>>        if (!strcmp(type, "console"))
>>                return 0;
>>
>> But it seems to be a red herring .. although the "the xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping" warning is gone.
>> The PV domU guest still doesn't get completely halted:
>> - It stays in a running state
>> - It uses 100% cpu according to xentop
>> - I can still connect to it's console
>>
>> So probably something is stuck in a waiting loop, which doesn't sleep in between and consumes 100% cpu.

> Hm, I generally see clean shutdowns of my multi-VCPU domains.

> Can you do

> # gdbsx -a <domid> <arch-size> 9999        (arch-size is 32 or 64)

> then

> $ gdb vmlinux
> (gdb) tar rem :9999
> (gdb) thread apply all bt

> to see what's going on?  (You may need to enable CONFIG_DEBUG_INFO to
> get useful info if you haven't already.)

>     J


>> Last lines of domU's console(with some additional printk's in xenbus_dev_shutdown):
>>
>> Cannot access the Hardware Clock via any known method.
>> Use the --debug option to see the details of our search for an access method.
>> Stopping enhanced syslogd: rsyslogd.
>> Asking all remaining processes to terminate...done.
>> All processes ended within 2 seconds....done.
>> Deconfiguring network interfaces...done.
>> Cleaning up ifupdown....
>> Deactivating swap...done.
>> Unmounting local filesystems...done.
>> Will now halt.
>> [   46.643035] md: stopping all md devices.
>> [   47.643320] xenbus_dev_shutdown:  trying shutdown of device/vif/0: Connected
>> [   47.716448] xenbus_dev_shutdown: result of shutdown of device/vif/0: Closed
>> [   47.716461] xenbus_dev_shutdown:  trying shutdown of device/vbd/51714: Connected
>> [   47.772293] xenbus_dev_shutdown: result of shutdown of device/vbd/51714: Closed
>> [   47.772306] xenbus_dev_shutdown:  trying shutdown of device/vbd/51713: Connected
>> [   47.829384] xenbus_dev_shutdown: result of shutdown of device/vbd/51713: Closed
>> [   47.829415] System halted.
>>
>>
>>
>> it's a domU using a 2.6.37-rc2 kernel, file: based disk access, nothing very special.
>>
>> the only difference that triggers it is:
>> vcpus=1 works fine ..
>> vcpus>1 symptoms above ..
>>
>> I have also added xend.log, with first a boot and shutdown with vpcus=1 and then a boot and shutdown with vpcus=4
>> I left the domain(with 4vcpus) running with 100% cpu for 10 minutes, but doesn't seem to hit any timeout, and keeps running.
>> On 20:40 it ends up stuck, on 20:51 i'm shooting the domain off with xm destroy.
>>
>>
>> Any pointers for adding extra debug info ?
>>
>> --
>> Sander
>>
>>
>>
>>
>> Wednesday, November 17, 2010, 10:57:47 PM, you wrote:
>>
>>> On 11/17/2010 12:22 PM, Sander Eikelenboom wrote:
>>>> Ah yes, 2.6.18 does contain the following line  in xenbus_probe.c and 2.6.37-rc2 not:
>>>>
>>>> -       if (!strcmp(type, "console"))
>>>> -               return 0;
>>>> It seems to be changed,
>>>>
>>>>
>>>> There seems to have been a patch http://lists.xensource.com/archives/html/xen-devel/2009-11/msg01072.html but it seems it's not applied to xenbus in linux upstream.
>>> And does this patch fix things for you?
>>> But I have to say that's pretty gross.  Is it really the right thing to do?
>>>     J
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel




-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it