From mboxrd@z Thu Jan  1 00:00:00 1970
From: Andrew Cooper <andrew.cooper3@citrix.com>
Subject: Re: HVMs terminating as (null)
Date: Sat, 23 Nov 2013 20:26:11 +0000
Message-ID: <52910F63.4070705@citrix.com>
References: <5290D480.1000804@crc.id.au>
	<20131123192746.GA21689@aepfle.de>	<52910427.5020507@crc.id.au>
	<52910853.1090708@crc.id.au> <52910A2F.50806@citrix.com>
	<52910B7F.6030503@crc.id.au>
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary="===============2231236130229094471=="
Return-path: <xen-devel-bounces@lists.xen.org>
In-Reply-To: <52910B7F.6030503@crc.id.au>
List-Unsubscribe: <http://lists.xen.org/cgi-bin/mailman/options/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xen.org>
List-Help: <mailto:xen-devel-request@lists.xen.org?subject=help>
List-Subscribe: <http://lists.xen.org/cgi-bin/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=subscribe>
Sender: xen-devel-bounces@lists.xen.org
Errors-To: xen-devel-bounces@lists.xen.org
To: Steven Haigh <netwiz@crc.id.au>
Cc: xen-devel@lists.xen.org
List-Id: xen-devel@lists.xenproject.org

--===============2231236130229094471==
Content-Type: multipart/alternative;
	boundary="------------050204020307060302080703"

--------------050204020307060302080703
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7bit

On 23/11/13 20:09, Steven Haigh wrote:
> On 24/11/13 07:03, Andrew Cooper wrote:
>> On 23/11/13 19:56, Steven Haigh wrote:
>>> On 24/11/13 06:38, Steven Haigh wrote:
>>>> On 24/11/13 06:27, Olaf Hering wrote:
>>>>> On Sun, Nov 24, Steven Haigh wrote:
>>>>>
>>>>>> Running Xen 4.2.3 with all the current XSA fixes.
>>>>>
>>>>> How exactly did you start the guests?
>>>>
>>>> The DomUs were started with: xl create /etc/xen/<configfile>
>>>>
>>>>> Does 'ps faxu' show qemu processes for the listed domain_ids?
>>>>> What is the 'xenstore-ls -f | sort' output?
>>>>
>>>> I'll have to check this when I manage to reproduce it. So far, I have
>>>> been unable to get a reliable way to reproduce it. I managed to get a
>>>> system to do it every time a HVM DomU was shutdown OR restarted - but
>>>> after a reboot of the Dom0 I can't get it into that state again.
>>>>
>>>> As soon as I can get a system in this state again, I'll leave it to see
>>>> what information I can extract.
>>>
>>> Ha! As always, as soon as I send this, I notice its happened on a Dom0.
>>>
>>> # xl list
>>> Name                                        ID   Mem VCPUs      State
>>> Time(s)
>>> Domain-0                                     0  1579     2     r-----
>>>  2731.3
>>> planner.vm                                   1  1013     1     -b----
>>>   189.3
>>> (null)                                       2     0     1     --psrd
>>>   301.1
>>> tracker.vm                                   3  1013     2     -b----
>>>   834.4
>>>
>>> Attached is the output of:
>>> # xl debug-keys q
>>> # xl dmesg  > xen-dmesg.log
>>> # gzip xen-dmesg.log
>>
>> Ok - from dmesg.
>>
>> (XEN) General information for domain 2:
>> (XEN)     refcnt=1 dying=2 pause_count=2
>> (XEN)     nr_pages=2 xenheap_pages=0 shared_pages=0 paged_pages=0
>> dirty_cpus={} max_pages=262400
>> (XEN)     handle=ef58ef1a-784d-4e59-8079-42bdee87f219 vm_assist=00000000
>> (XEN)     paging assistance: hap refcounts translate external
>> ...
>> (XEN) Memory pages belonging to domain 2:
>> (XEN)     DomPage 00000000000866e0: caf=00000001, taf=0000000000000000
>> (XEN)     DomPage 00000000000866e1: caf=00000001, taf=0000000000000000
>> (XEN)     PoD entries=0 cachesize=0
>>
>>
>> So there are indeed two outstanding pages causing this domain to become
>> a zombie.  They are normal pages, with 1 outstanding ref.
>>
>> Can you collect "xl debug-keys g" as well?
>
> Sure - attached.

(XEN)       -------- active --------       -------- shared --------
(XEN) [ref] localdom mfn      pin          localdom gmfn     flags
(XEN) grant-table for remote domain:    2 (v1)
(XEN) [16302]        0 0x0866e1 0x00000001          0 0x0064e1 0x19
(XEN) [16320]        0 0x0866e0 0x00000001          0 0x0064e0 0x19

Ok - so domain 2 has two outstanding grants.  This explains why it is a
zombie.

Both these grants are GFT_writing | GFT_reading | GFT_permit_access, but
seemingly unmapped.

I will have to defer to someone who knows the grant code better.  Is it
possible for a domain to be a zombie just because it has two grants it
hasn't manually invalidated?

~Andrew


--------------050204020307060302080703
Content-Type: text/html; charset="ISO-8859-1"
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    On 23/11/13 20:09, Steven Haigh wrote:<br>
    <span style="white-space: pre;">&gt; On 24/11/13 07:03, Andrew
      Cooper wrote:<br>
      &gt;&gt; On 23/11/13 19:56, Steven Haigh wrote:<br>
      &gt;&gt;&gt; On 24/11/13 06:38, Steven Haigh wrote:<br>
      &gt;&gt;&gt;&gt; On 24/11/13 06:27, Olaf Hering wrote:<br>
      &gt;&gt;&gt;&gt;&gt; On Sun, Nov 24, Steven Haigh wrote:<br>
      &gt;&gt;&gt;&gt;&gt;<br>
      &gt;&gt;&gt;&gt;&gt;&gt; Running Xen 4.2.3 with all the current
      XSA fixes.<br>
      &gt;&gt;&gt;&gt;&gt;<br>
      &gt;&gt;&gt;&gt;&gt; How exactly did you start the guests?<br>
      &gt;&gt;&gt;&gt;<br>
      &gt;&gt;&gt;&gt; The DomUs were started with: xl create
      /etc/xen/&lt;configfile&gt;<br>
      &gt;&gt;&gt;&gt;<br>
      &gt;&gt;&gt;&gt;&gt; Does 'ps faxu' show qemu processes for the
      listed domain_ids?<br>
      &gt;&gt;&gt;&gt;&gt; What is the 'xenstore-ls -f | sort' output?<br>
      &gt;&gt;&gt;&gt;<br>
      &gt;&gt;&gt;&gt; I'll have to check this when I manage to
      reproduce it. So far, I have<br>
      &gt;&gt;&gt;&gt; been unable to get a reliable way to reproduce
      it. I managed to get a<br>
      &gt;&gt;&gt;&gt; system to do it every time a HVM DomU was
      shutdown OR restarted - but<br>
      &gt;&gt;&gt;&gt; after a reboot of the Dom0 I can't get it into
      that state again.<br>
      &gt;&gt;&gt;&gt;<br>
      &gt;&gt;&gt;&gt; As soon as I can get a system in this state
      again, I'll leave it to see<br>
      &gt;&gt;&gt;&gt; what information I can extract.<br>
      &gt;&gt;&gt;<br>
      &gt;&gt;&gt; Ha! As always, as soon as I send this, I notice its
      happened on a Dom0.<br>
      &gt;&gt;&gt;<br>
      &gt;&gt;&gt; # xl list<br>
      &gt;&gt;&gt; Name&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ID&nbsp;&nbsp; Mem
      VCPUs&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; State<br>
      &gt;&gt;&gt; Time(s)<br>
      &gt;&gt;&gt; Domain-0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0&nbsp;
      1579&nbsp;&nbsp;&nbsp;&nbsp; 2&nbsp;&nbsp;&nbsp;&nbsp; r-----<br>
      &gt;&gt;&gt;&nbsp; 2731.3<br>
      &gt;&gt;&gt; planner.vm&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;
      1013&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp;&nbsp;&nbsp; -b----<br>
      &gt;&gt;&gt;&nbsp;&nbsp; 189.3<br>
      &gt;&gt;&gt; (null)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2&nbsp;&nbsp;&nbsp;&nbsp;
      0&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp;&nbsp;&nbsp;&nbsp; --psrd<br>
      &gt;&gt;&gt;&nbsp;&nbsp; 301.1<br>
      &gt;&gt;&gt; tracker.vm&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3&nbsp;
      1013&nbsp;&nbsp;&nbsp;&nbsp; 2&nbsp;&nbsp;&nbsp;&nbsp; -b----<br>
      &gt;&gt;&gt;&nbsp;&nbsp; 834.4<br>
      &gt;&gt;&gt;<br>
      &gt;&gt;&gt; Attached is the output of:<br>
      &gt;&gt;&gt; # xl debug-keys q<br>
      &gt;&gt;&gt; # xl dmesg&nbsp; &gt; xen-dmesg.log<br>
      &gt;&gt;&gt; # gzip xen-dmesg.log<br>
      &gt;&gt;<br>
      &gt;&gt; Ok - from dmesg.<br>
      &gt;&gt;<br>
      &gt;&gt; (XEN) General information for domain 2:<br>
      &gt;&gt; (XEN)&nbsp;&nbsp;&nbsp;&nbsp; refcnt=1 dying=2 pause_count=2<br>
      &gt;&gt; (XEN)&nbsp;&nbsp;&nbsp;&nbsp; nr_pages=2 xenheap_pages=0 shared_pages=0
      paged_pages=0<br>
      &gt;&gt; dirty_cpus={} max_pages=262400<br>
      &gt;&gt; (XEN)&nbsp;&nbsp;&nbsp;&nbsp; handle=ef58ef1a-784d-4e59-8079-42bdee87f219
      vm_assist=00000000<br>
      &gt;&gt; (XEN)&nbsp;&nbsp;&nbsp;&nbsp; paging assistance: hap refcounts translate
      external<br>
      &gt;&gt; ...<br>
      &gt;&gt; (XEN) Memory pages belonging to domain 2:<br>
      &gt;&gt; (XEN)&nbsp;&nbsp;&nbsp;&nbsp; DomPage 00000000000866e0: caf=00000001,
      taf=0000000000000000<br>
      &gt;&gt; (XEN)&nbsp;&nbsp;&nbsp;&nbsp; DomPage 00000000000866e1: caf=00000001,
      taf=0000000000000000<br>
      &gt;&gt; (XEN)&nbsp;&nbsp;&nbsp;&nbsp; PoD entries=0 cachesize=0<br>
      &gt;&gt;<br>
      &gt;&gt;<br>
      &gt;&gt; So there are indeed two outstanding pages causing this
      domain to become<br>
      &gt;&gt; a zombie.&nbsp; They are normal pages, with 1 outstanding ref.<br>
      &gt;&gt;<br>
      &gt;&gt; Can you collect "xl debug-keys g" as well?<br>
      &gt;<br>
      &gt; Sure - attached.</span><br>
    <br>
    (XEN)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -------- active --------&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -------- shared --------<br>
    (XEN) [ref] localdom mfn&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; pin&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; localdom gmfn&nbsp;&nbsp;&nbsp;&nbsp; flags<br>
    (XEN) grant-table for remote domain:&nbsp;&nbsp;&nbsp; 2 (v1)<br>
    (XEN) [16302]&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0 0x0866e1 0x00000001&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0 0x0064e1 0x19<br>
    (XEN) [16320]&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0 0x0866e0 0x00000001&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0 0x0064e0 0x19<br>
    <br>
    Ok - so domain 2 has two outstanding grants.&nbsp; This explains why it
    is a zombie.<br>
    <br>
    Both these grants are GFT_writing | GFT_reading | GFT_permit_access,
    but seemingly unmapped.<br>
    <br>
    I will have to defer to someone who knows the grant code better.&nbsp; Is
    it possible for a domain to be a zombie just because it has two
    grants it hasn't manually invalidated?<br>
    <br>
    ~Andrew<br>
    <br>
  </body>
</html>

--------------050204020307060302080703--


--===============2231236130229094471==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

--===============2231236130229094471==--