Re: More Xen troubles (with xend this time)

[Mike Wray <mike.wray@hpl.hp.com>]

Jérôme Petazzoni wrote:
> 
>>> Okay, and once we "lost" xend, there is no way to do anything ? 
>>> (create, destroy, shutdown, get a console...)
>>>   
>> You could try removing xend's database in /var/xen/xend-db -- you will
>> lose your domain's names but I've found that sometimes this allows
>> restarting xend when it's in the state where it doesn't want to start
>> anymore.
>>  
> Interesting ! I did that, and then :
> 
> # xm list
> Name              Id  Mem(MB)  CPU  State  Time(s)  Console
> Domain-0           0       59    0  r----   5915.7       
> Domain-16         16       63    0  -b---    328.7       
> Domain-17         17      127    0  -b---    699.8       
> Domain-21         21        0    0  ----c      0.0       
> Domain-22         22        0    0  ----c      0.0      
> I could successfully create a new domain attached to a console, and stop 
> it ; but other stuff didn't work :
> 
> # xm console 16
> Error: No console information

The problem is that if you remove xend-db xend no longer knows anything about
the running domains (like what inter-domain ports their consoles are on,
what their names are, what devices they have). So xend can no longer shut
them down properly because it doesn't know what devices to release.
This is why you get all the errors.

> And xm shutdown didn't do anything, it seems. Destroying crashed domains 
> didn't work either. It keeps spitting those messages every couple of 
> seconds in xend.log :
> 
> [2004-11-30 18:06:44 xend] DEBUG (XendDomain:244) XendDomain>reap> 
> domain died name=Domain-21 id=21
> [2004-11-30 18:06:44 xend] INFO (XendDomain:564) Destroying domain: 
> name=Domain-21
> [2004-11-30 18:06:44 xend] DEBUG (XendDomain:244) XendDomain>reap> 
> domain died name=Domain-22 id=22
> [2004-11-30 18:06:44 xend] INFO (XendDomain:564) Destroying domain: 
> name=Domain-22
> [2004-11-30 18:06:44 xend] INFO (XendRoot:91) EVENT> xend.domain.exit 
> ['Domain-21', '21', 'crash']
> [2004-11-30 18:06:44 xend] INFO (XendRoot:91) EVENT> xend.domain.destroy 
> ['Domain-21', '21']
> [2004-11-30 18:06:44 xend] INFO (XendRoot:91) EVENT> xend.domain.exit 
> ['Domain-22', '22', 'crash']
> [2004-11-30 18:06:44 xend] INFO (XendRoot:91) EVENT> xend.domain.destroy 
> ['Domain-22', '22']

Xend is trying to get rid of the domains, but because their devices aren't
freed (because you removed the info about them) the domains won't go away.
So xend keeps detecting crashed domains and trying to get rid of them.

Not the nicest behaviour - crashes don't happen very much so this path
is not well-explored.

> xend-debug.log and xm dmesg are silent.
> 
> Anything useful I can try to get more information about this before I 
> reboot the beast ? :-)

At this point you're pretty stuck. Time for a reboot.

Mike


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/