zombie domains

zombie domains

[Gerd Knorr]

   Hi,

How can I figure why some domain is still in zombie state, like these ones:

   master-xen root /vm/ttylinux# xm list
   Name                              ID Mem(MiB) VCPUs State  Time(s)
   Domain-0                           0      574     1 r-----    90.5
   Zombie-small-11                   28        0     1 ---s-d     0.9
   Zombie-small-17                   34        0     1 ---s-d     0.5
   Zombie-small-18                   35        0     1 ---s-d     0.6
   Zombie-small-19                   36        0     1 ---s-d     0.5

I've created 16 ttylinux instances with a script, then called "xm 
shutdown -a -w", then ended up with these four Zombies ...

   Gerd

Re: zombie domains

[David F Barrera]

I saved a domain and ended up with a Zombie-migrating domain:

x235:~ # xm list
Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0      250     1 r-----  5518.8
vm1                                1      127     1 ------  2564.9
Zombie-migrating-vm2               2        0     1 ---s-d  1292.4
vm3                                3      127     1 ------  1683.5
vm4                                4      127     1 ------  1783.8
vm5                                5      127     1 r-----  1615.0
vm2                                6      126     1 ------    10.9

At first I thought it was a temporary state, but after an hour or so, it 
is still there. I later restored it (vm2) successfully, but the zombie 
is still there.

Gerd Knorr wrote:

>   Hi,
>
> How can I figure why some domain is still in zombie state, like these 
> ones:
>
>   master-xen root /vm/ttylinux# xm list
>   Name                              ID Mem(MiB) VCPUs State  Time(s)
>   Domain-0                           0      574     1 r-----    90.5
>   Zombie-small-11                   28        0     1 ---s-d     0.9
>   Zombie-small-17                   34        0     1 ---s-d     0.5
>   Zombie-small-18                   35        0     1 ---s-d     0.6
>   Zombie-small-19                   36        0     1 ---s-d     0.5
>
> I've created 16 ttylinux instances with a script, then called "xm 
> shutdown -a -w", then ended up with these four Zombies ...
>
>   Gerd
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: zombie domains

[Ewan Mellor]

On Mon, Oct 31, 2005 at 03:47:59PM +0100, Gerd Knorr wrote:

>   Hi,
> 
> How can I figure why some domain is still in zombie state, like these ones:
> 
>   master-xen root /vm/ttylinux# xm list
>   Name                              ID Mem(MiB) VCPUs State  Time(s)
>   Domain-0                           0      574     1 r-----    90.5
>   Zombie-small-11                   28        0     1 ---s-d     0.9
>   Zombie-small-17                   34        0     1 ---s-d     0.5
>   Zombie-small-18                   35        0     1 ---s-d     0.6
>   Zombie-small-19                   36        0     1 ---s-d     0.5
> 
> I've created 16 ttylinux instances with a script, then called "xm 
> shutdown -a -w", then ended up with these four Zombies ...

It's not easy, Gerd, hence the delay in replying to you!

Steven Hand found a bug related to Zombies (changeset 7618:a45408260abb) which
may well have fixed this problem, so could you retest and let us know how you
got on?

Thanks,

Ewan.

Re: zombie domains

[Steven Hand]

> On Mon, Oct 31, 2005 at 03:47:59PM +0100, Gerd Knorr wrote:
> 
> >   Hi,
> > 
> > How can I figure why some domain is still in zombie state, like these ones:
> > 
> >   master-xen root /vm/ttylinux# xm list
> >   Name                              ID Mem(MiB) VCPUs State  Time(s)
> >   Domain-0                           0      574     1 r-----    90.5
> >   Zombie-small-11                   28        0     1 ---s-d     0.9
> >   Zombie-small-17                   34        0     1 ---s-d     0.5
> >   Zombie-small-18                   35        0     1 ---s-d     0.6
> >   Zombie-small-19                   36        0     1 ---s-d     0.5
> > 
> > I've created 16 ttylinux instances with a script, then called "xm 
> > shutdown -a -w", then ended up with these four Zombies ...

They're almost certainly in zombie state since there's an oustanding 
reference to a page of theirs which hasn't been dropped. 


If you have a serial console, hit 'Ctrl-A' three times (switch console
to Xen) and then hit 'q' 

This will dump out various info for each domain including something like 

(XEN) Xen: DOM N, flags=0 refcnt=BLAH nr_pages=FOO xenheap_pages=BAZ

... if 'nr_pages' is > 0 (it is probably a small integer), then this 
is why the domain is not being destroyed. 

The cset Ewan mentioned fixes this. 

Would be keen to see if your problem is something else...


cheers,


S.

Re: zombie domains

[Gerd Knorr]

> It's not easy, Gerd, hence the delay in replying to you!

Seems to be gone now, at least I can't reproduce it that easily any more.

cheers,

   Gerd

RE: zombie domains

[Yu, Ping Y]

I met this problem too and this zombie domain could not be destroyed. 
I have opened a bug on this. 
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=323


-----Original Message-----
From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of David F Barrera
Sent: 2005年11月1日 4:59
To: Gerd Knorr
Cc: xen-devel
Subject: Re: [Xen-devel] zombie domains

I saved a domain and ended up with a Zombie-migrating domain:

x235:~ # xm list
Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0      250     1 r-----  5518.8
vm1                                1      127     1 ------  2564.9
Zombie-migrating-vm2               2        0     1 ---s-d  1292.4
vm3                                3      127     1 ------  1683.5
vm4                                4      127     1 ------  1783.8
vm5                                5      127     1 r-----  1615.0
vm2                                6      126     1 ------    10.9

At first I thought it was a temporary state, but after an hour or so, it 
is still there. I later restored it (vm2) successfully, but the zombie 
is still there.

Gerd Knorr wrote:

>   Hi,
>
> How can I figure why some domain is still in zombie state, like these 
> ones:
>
>   master-xen root /vm/ttylinux# xm list
>   Name                              ID Mem(MiB) VCPUs State  Time(s)
>   Domain-0                           0      574     1 r-----    90.5
>   Zombie-small-11                   28        0     1 ---s-d     0.9
>   Zombie-small-17                   34        0     1 ---s-d     0.5
>   Zombie-small-18                   35        0     1 ---s-d     0.6
>   Zombie-small-19                   36        0     1 ---s-d     0.5
>
> I've created 16 ttylinux instances with a script, then called "xm 
> shutdown -a -w", then ended up with these four Zombies ...
>
>   Gerd
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Zombie domains

[David Lie]

I'm using the grant table to map a shared frame between two domains.  
Domain 1 shares the frame, and Domain 2 maps it into it's address space. 
I then make sure Domain 2 unmaps the frame, and releases all event 
channels, etc... before shutting down.  Domain 2 always remains as a 
zombie though when I do xm list.  If I dump the domain info in the Xen 
console, I get this information for the zombie domain:

(XEN) General information for domain 12:
(XEN)     flags=6 refcnt=1 nr_pages=0 xenheap_pages=0 dirty_cpus={}
(XEN)     handle=f4a55907-26db-d7c3-f6a7-392637013289
(XEN) Rangesets belonging to domain 12:
(XEN)     Interrupts { }
(XEN)     I/O Memory { }
(XEN)     I/O Ports  { }
(XEN) Memory pages belonging to domain 12:
(XEN) VCPU information and callbacks for domain 12:
(XEN)     VCPU0: CPU0 [has=F] flags=10 upcall_pend = 01, upcall_mask = 
00 dirty}(XEN)     Notifying guest (virq 1, port 0, stat 0/0/-1)

Unfortunately, these fields do not mean very much to me.  What does 
upcall_pend mean?  Sometimes this field is 01, sometimes it's 00.  What 
about refcnt?  Note that Domain 2 is not running XenoLinux, but 
something similar to Mini-OS.  Can anyone on this list see what might be 
stopping this domain from shutting down cleanly?  If not, is there any 
documentation as to what these fields mean?

Thanks in advance.

-DL

RE: Zombie domains

[Ian Pratt]

> I'm using the grant table to map a shared frame between two domains.
> Domain 1 shares the frame, and Domain 2 maps it into it's address
space.
> I then make sure Domain 2 unmaps the frame, and releases all event
> channels, etc... before shutting down.  Domain 2 always remains as a
> zombie though when I do xm list.  If I dump the domain info in the Xen
> console, I get this information for the zombie domain:
> 
> (XEN) General information for domain 12:
> (XEN)     flags=6 refcnt=1 nr_pages=0 xenheap_pages=0 dirty_cpus={}
> (XEN)     handle=f4a55907-26db-d7c3-f6a7-392637013289
> (XEN) Rangesets belonging to domain 12:
> (XEN)     Interrupts { }
> (XEN)     I/O Memory { }
> (XEN)     I/O Ports  { }
> (XEN) Memory pages belonging to domain 12:
> (XEN) VCPU information and callbacks for domain 12:
> (XEN)     VCPU0: CPU0 [has=F] flags=10 upcall_pend = 01, upcall_mask =
> 00 dirty}(XEN)     Notifying guest (virq 1, port 0, stat 0/0/-1)

The usual region for zombie domains is other domains having its memory
mapped, but not in this case: nr_pages=0

> Unfortunately, these fields do not mean very much to me.  What does
> upcall_pend mean?  

There's an event pending for the domain. Not a big deal.

> Sometimes this field is 01, sometimes it's 00.  What
> about refcnt?  

Something has a reference to the domain structure, hence preventing it
from being freed. This must be a xen bug. Your OS is likely provoking an
error path that is missing a 'put'.   

Have you tried this with latest -unstable?

Ian

Re: Zombie domains

[David Lie]

This was done on the testing release.  There are some small changes to
the underlying Xen hypervisor that we've made so it's not as straight
forward to just try it on the unstable release (I'm working on a port of
our OSDI work from Xen 2.0 to Xen 3.0).

I'll look into it, but it is not causing any huge problems for me right
now.  Another thing is I noticed is the flags field also seems a bit
suspicious.  Am I interpreting that the value "6" correctly to mean that
the domain is both dying and being debugged?  How would it get into the
state that it thinks it is being debugged?

-DL

Ian Pratt wrote:
>> I'm using the grant table to map a shared frame between two domains.
>> Domain 1 shares the frame, and Domain 2 maps it into it's address
> space.
>> I then make sure Domain 2 unmaps the frame, and releases all event
>> channels, etc... before shutting down.  Domain 2 always remains as a
>> zombie though when I do xm list.  If I dump the domain info in the Xen
>> console, I get this information for the zombie domain:
>>
>> (XEN) General information for domain 12:
>> (XEN)     flags=6 refcnt=1 nr_pages=0 xenheap_pages=0 dirty_cpus={}
>> (XEN)     handle=f4a55907-26db-d7c3-f6a7-392637013289
>> (XEN) Rangesets belonging to domain 12:
>> (XEN)     Interrupts { }
>> (XEN)     I/O Memory { }
>> (XEN)     I/O Ports  { }
>> (XEN) Memory pages belonging to domain 12:
>> (XEN) VCPU information and callbacks for domain 12:
>> (XEN)     VCPU0: CPU0 [has=F] flags=10 upcall_pend = 01, upcall_mask =
>> 00 dirty}(XEN)     Notifying guest (virq 1, port 0, stat 0/0/-1)
> 
> The usual region for zombie domains is other domains having its memory
> mapped, but not in this case: nr_pages=0
> 
>> Unfortunately, these fields do not mean very much to me.  What does
>> upcall_pend mean?  
> 
> There's an event pending for the domain. Not a big deal.
> 
>> Sometimes this field is 01, sometimes it's 00.  What
>> about refcnt?  
> 
> Something has a reference to the domain structure, hence preventing it
> from being freed. This must be a xen bug. Your OS is likely provoking an
> error path that is missing a 'put'.   
> 
> Have you tried this with latest -unstable?
> 
> Ian