xen-devel.lists.xenproject.org archive mirror
 help / color / mirror / Atom feed
* [xen-unstable test] 25949: regressions - trouble: broken/fail/pass
@ 2014-04-23  1:08 xen.org
  2014-04-23  9:31 ` Jan Beulich
  0 siblings, 1 reply; 6+ messages in thread
From: xen.org @ 2014-04-23  1:08 UTC (permalink / raw)
  To: xen-devel; +Cc: ian.jackson

flight 25949 xen-unstable real [real]
http://www.chiark.greenend.org.uk/~xensrcts/logs/25949/

Regressions :-(

Tests which did not succeed and are blocking,
including tests which could not be run:
 test-amd64-amd64-xl-qemuu-ovmf-amd64  3 host-install(3) broken REGR. vs. 25945
 test-amd64-i386-xl-qemut-win7-amd64 13 guest-localmigrate/x10 fail REGR. vs. 25945
 test-amd64-i386-xl-qemuu-win7-amd64  5 xen-boot           fail REGR. vs. 25945

Regressions which are regarded as allowable (not blocking):
 test-amd64-i386-xl-qemuu-winxpsp3 11 guest-saverestore.2       fail like 25905

Tests which did not succeed, but are not blocking:
 test-amd64-i386-xl-qemuu-ovmf-amd64  7 debian-hvm-install      fail never pass
 test-armhf-armhf-xl          10 migrate-support-check        fail   never pass
 test-amd64-amd64-xl-pcipt-intel  9 guest-start                 fail never pass
 test-amd64-i386-xl-qemut-winxpsp3 14 guest-stop                fail never pass
 test-amd64-i386-xl-qemut-winxpsp3-vcpus1 14 guest-stop         fail never pass
 test-amd64-i386-xl-qemuu-winxpsp3-vcpus1 14 guest-stop         fail never pass
 test-amd64-amd64-xl-win7-amd64 14 guest-stop                   fail never pass
 test-amd64-amd64-xl-qemuu-win7-amd64 14 guest-stop             fail never pass
 test-amd64-i386-xl-win7-amd64 14 guest-stop                   fail  never pass
 test-amd64-amd64-xl-qemut-winxpsp3 14 guest-stop               fail never pass
 test-amd64-i386-xl-winxpsp3-vcpus1 14 guest-stop               fail never pass
 test-amd64-amd64-xl-winxpsp3 14 guest-stop                   fail   never pass
 test-amd64-i386-xl-winxpsp3  14 guest-stop                   fail   never pass
 test-amd64-amd64-xl-qemut-win7-amd64 14 guest-stop             fail never pass
 test-amd64-amd64-xl-qemuu-winxpsp3 14 guest-stop               fail never pass

version targeted for testing:
 xen                  816f6224823320c8452fd3af5d873a2b82f5e1c3
baseline version:
 xen                  01feb234d0cb3bff248694d99397fb63a9757490

------------------------------------------------------------
People who touched revisions under test:
  Andrew Cooper <andrew.cooper3@citrix.com>
  Boris Ostrovsky <boris.ostrovsky@oracle.com>
  Daniel De Graaf <dgdegra@tycho.nsa.gov>
  Jan Beulich <jbeulich@suse.com>
  Kevin Tian <kevin.tian@intel.com>
------------------------------------------------------------

jobs:
 build-amd64                                                  pass    
 build-armhf                                                  pass    
 build-i386                                                   pass    
 build-amd64-oldkern                                          pass    
 build-i386-oldkern                                           pass    
 build-amd64-pvops                                            pass    
 build-armhf-pvops                                            pass    
 build-i386-pvops                                             pass    
 test-amd64-amd64-xl                                          pass    
 test-armhf-armhf-xl                                          pass    
 test-amd64-i386-xl                                           pass    
 test-amd64-i386-rhel6hvm-amd                                 pass    
 test-amd64-i386-qemut-rhel6hvm-amd                           pass    
 test-amd64-i386-qemuu-rhel6hvm-amd                           pass    
 test-amd64-i386-freebsd10-amd64                              pass    
 test-amd64-amd64-xl-qemuu-ovmf-amd64                         broken  
 test-amd64-i386-xl-qemuu-ovmf-amd64                          fail    
 test-amd64-amd64-xl-qemut-win7-amd64                         fail    
 test-amd64-i386-xl-qemut-win7-amd64                          fail    
 test-amd64-amd64-xl-qemuu-win7-amd64                         fail    
 test-amd64-i386-xl-qemuu-win7-amd64                          fail    
 test-amd64-amd64-xl-win7-amd64                               fail    
 test-amd64-i386-xl-win7-amd64                                fail    
 test-amd64-i386-xl-credit2                                   pass    
 test-amd64-i386-freebsd10-i386                               pass    
 test-amd64-amd64-xl-pcipt-intel                              fail    
 test-amd64-i386-rhel6hvm-intel                               pass    
 test-amd64-i386-qemut-rhel6hvm-intel                         pass    
 test-amd64-i386-qemuu-rhel6hvm-intel                         pass    
 test-amd64-i386-xl-multivcpu                                 pass    
 test-amd64-amd64-pair                                        pass    
 test-amd64-i386-pair                                         pass    
 test-amd64-amd64-xl-sedf-pin                                 pass    
 test-amd64-amd64-xl-sedf                                     pass    
 test-amd64-i386-xl-qemut-winxpsp3-vcpus1                     fail    
 test-amd64-i386-xl-qemuu-winxpsp3-vcpus1                     fail    
 test-amd64-i386-xl-winxpsp3-vcpus1                           fail    
 test-amd64-amd64-xl-qemut-winxpsp3                           fail    
 test-amd64-i386-xl-qemut-winxpsp3                            fail    
 test-amd64-amd64-xl-qemuu-winxpsp3                           fail    
 test-amd64-i386-xl-qemuu-winxpsp3                            fail    
 test-amd64-amd64-xl-winxpsp3                                 fail    
 test-amd64-i386-xl-winxpsp3                                  fail    


------------------------------------------------------------
sg-report-flight on osstest.cam.xci-test.com
logs: /home/xc_osstest/logs
images: /home/xc_osstest/images

Logs, config files, etc. are available at
    http://www.chiark.greenend.org.uk/~xensrcts/logs

Test harness code can be found at
    http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary


Not pushing.

------------------------------------------------------------
commit 816f6224823320c8452fd3af5d873a2b82f5e1c3
Author: Daniel De Graaf <dgdegra@tycho.nsa.gov>
Date:   Tue Apr 22 12:10:13 2014 +0200

    allow hardware domain != dom0
    
    This adds a hypervisor command line option "hardware_dom=" which takes a
    domain ID.  When the domain with this ID is created, it will be used
    as the hardware domain.
    
    This is intended to be used when domain 0 is a dedicated stub domain for
    domain building, allowing the hardware domain to be de-privileged and
    act only as a driver domain.
    
    Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
    Reviewed-by: Jan Beulich <jbeulich@suse.com>

commit 4d21a95558b9c9007d8b70e63d1449a4a10f535c
Author: Andrew Cooper <andrew.cooper3@citrix.com>
Date:   Tue Apr 22 12:09:36 2014 +0200

    x86/hap: fix lack of newline in error message
    
    to avoid corrupting the following line.
    
    Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
    Acked-by: Jan Beulich <JBeulich@suse.com>

commit 88e64cb785c1de4f686c1aa1993a0003b7db9e1a
Author: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Date:   Tue Apr 22 12:08:56 2014 +0200

    x86/HVM: use fixed TSC value when saving or restoring domain
    
    When a domain is saved each VCPU's TSC value needs to be preserved. To get it we
    use hvm_get_guest_tsc(). This routine (either itself or via get_s_time() which
    it may call) calculates VCPU's TSC based on current host's TSC value (by doing a
    rdtscll()). Since this is performed for each VCPU separately we end up with
    un-synchronized TSCs.
    
    Similarly, during a restore each VCPU is assigned its TSC based on host's current
    tick, causing virtual TSCs to diverge further.
    
    With this, we can easily get into situation where a guest may see time going
    backwards.
    
    Instead of reading new TSC value for each VCPU when saving/restoring it we should
    use the same value across all VCPUs.
    
    Reported-by: Philippe Coquard <philippe.coquard@mpsa.com>
    Signed-off-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
    Reviewed-by: Jan Beulich <jbeulich@suse.com>

commit b95fd03b5f0b66384bd7c190d5861ae68eb98c85
Author: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Date:   Tue Apr 22 12:08:06 2014 +0200

    x86/svm: enable TSC scaling
    
    TSC ratio enabling logic is inverted: we want to use it when we
    are running in native tsc mode, i.e. when d->arch.vtsc is zero.
    
    Also, since now svm_set_tsc_offset()'s calculations depend
    on vtsc's value, we need to call hvm_funcs.set_tsc_offset() after
    vtsc changes in tsc_set_info().
    
    In addition, with TSC ratio enabled, svm_set_tsc_offset() will
    need to do rdtsc. With that we may end up having TSCs on guest's
    processors out of sync. d->arch.hvm_domain.sync_tsc which is set
    by the boot processor can now be used by APs as reference TSC
    value instead of host's current TSC.
    
    Signed-off-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
    Reviewed-by: Jan Beulich <jbeulich@suse.com>

commit 82713ec8d2b65d17f13e46a131e38bfe5baf8bd6
Author: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Date:   Tue Apr 22 12:07:37 2014 +0200

    x86: use native RDTSC(P) execution when guest and host frequencies are the same
    
    We should be able to continue using native RDTSC(P) execution on
    HVM/PVH guests after migration if host and guest frequencies are
    equal (this includes the case when the frequencies are made equal
    by TSC scaling feature).
    
    This also allows us to revert main part of commit 4aab59a3 (svm: Do not
    intercept RDTSC(P) when TSC scaling is supported by hardware) which
    was wrong: while RDTSC intercepts were disabled domain's vtsc could
    still be set, leading to inconsistent view of guest's TSC.
    
    Signed-off-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
    Acked-by: Jan Beulich <jbeulich@suse.com>

commit e681c0408564a2c0d1c6d56d3f683f8db079458c
Author: Jan Beulich <jbeulich@suse.com>
Date:   Tue Apr 22 12:05:44 2014 +0200

    ACPI/ERST: fix signed/unsigned type conflicts
    
    Error checks exist in the respective code path that expect negative
    values to indicate errors, yet negative values can't be communicated
    through size_t. Thus ssize_t needs to be introduced (also on ARM for
    consistency, even if the code in question isn't currently being used
    on there).
    
    The bug is theoretical only in so far as all the involved code is
    effectively dead. Reflect this by excluding that code from non-debug
    builds.
    
    Signed-off-by: Jan Beulich <jbeulich@suse.com>
    Reviewed-by: Christoph Egger <chegger@amazon.de>

commit 061eebe0e99ad45c9c3b1a778b06140de4a91f25
Author: Jan Beulich <jbeulich@suse.com>
Date:   Tue Apr 22 12:04:20 2014 +0200

    x86/MSI: drop workaround for insecure Dom0 kernels
    
    Considering that
    - the workaround is expensive (iterating through the entire P2M space
      of a domain),
    - the planned elimination of the expensiveness (by propagating the type
      change step by step to the individual P2M leaves) wouldn't address
      the IOMMU side of things (as for it to obey to the changed
      permissions the adjustments must be pushed down immediately through
      the entire tree)
    - the proper solution (PHYSDEVOP_msix_prepare) should by now be
      implemented by all security conscious Dom0 kernels
    remove the workaround, killing eventual guests that would be known to
    become a security risk instead.
    
    Signed-off-by: Jan Beulich <jbeulich@suse.com>
    Acked-by: Kevin Tian <kevin.tian@intel.com>
(qemu changes not included)

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [xen-unstable test] 25949: regressions - trouble: broken/fail/pass
  2014-04-23  1:08 [xen-unstable test] 25949: regressions - trouble: broken/fail/pass xen.org
@ 2014-04-23  9:31 ` Jan Beulich
  2014-04-23  9:45   ` Ian Campbell
  2014-04-24 11:15   ` Ian Jackson
  0 siblings, 2 replies; 6+ messages in thread
From: Jan Beulich @ 2014-04-23  9:31 UTC (permalink / raw)
  To: ian.jackson, xen-devel

>>> On 23.04.14 at 03:08, <Ian.Jackson@eu.citrix.com> wrote:
> flight 25949 xen-unstable real [real]
> http://www.chiark.greenend.org.uk/~xensrcts/logs/25949/ 
> 
> Regressions :-(
> 
> Tests which did not succeed and are blocking,
> including tests which could not be run:
>  test-amd64-amd64-xl-qemuu-ovmf-amd64  3 host-install(3) broken REGR. vs. 25945
>  test-amd64-i386-xl-qemut-win7-amd64 13 guest-localmigrate/x10 fail REGR. vs. 25945
>  test-amd64-i386-xl-qemuu-win7-amd64  5 xen-boot           fail REGR. vs. 25945

The log appears to show that Dom0 booted up fine - were there any
infrastructure problems (also taking into consideration the broken
host-install above)?

Jan

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [xen-unstable test] 25949: regressions - trouble: broken/fail/pass
  2014-04-23  9:31 ` Jan Beulich
@ 2014-04-23  9:45   ` Ian Campbell
  2014-04-23 10:02     ` Jan Beulich
  2014-04-24 11:15   ` Ian Jackson
  1 sibling, 1 reply; 6+ messages in thread
From: Ian Campbell @ 2014-04-23  9:45 UTC (permalink / raw)
  To: Jan Beulich; +Cc: xen-devel, ian.jackson

On Wed, 2014-04-23 at 10:31 +0100, Jan Beulich wrote:
> >>> On 23.04.14 at 03:08, <Ian.Jackson@eu.citrix.com> wrote:
> > flight 25949 xen-unstable real [real]
> > http://www.chiark.greenend.org.uk/~xensrcts/logs/25949/ 
> > 
> > Regressions :-(
> > 
> > Tests which did not succeed and are blocking,
> > including tests which could not be run:
> >  test-amd64-amd64-xl-qemuu-ovmf-amd64  3 host-install(3) broken REGR. vs. 25945

This seems to be on itch-mite:
        2014-04-22 21:21:43 Z executing ssh ... root@10.80.249.148
        cat /proc/partitions
        2014-04-22 21:21:43 Z | major minor  #blocks  name
        2014-04-22 21:21:43 Z | 
        2014-04-22 21:21:43 Z |   11        0    1048575 sr0
        2014-04-22 21:21:43 Z |    8        0  244140625 sda
        2014-04-22 21:21:43 Z |    8       16  244198584 sdb
        2014-04-22 21:21:43 Z executing ssh ... root@10.80.249.148 pvcreate /dev/sdb
          Device /dev/sdb not found (or ignored by filtering).
        2014-04-22 21:21:44 Z command nonzero waitstatus 1280: ssh -o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=100 -o ServerAliveInterval=100 -o PasswordAuthentication=no -o ChallengeResponseAuthentication=no -o UserKnownHostsFile=tmp/t.known_hosts_25949.test-amd64-amd64-xl-qemuu-ovmf-amd64 root@10.80.249.148 pvcreate /dev/sdb
        status 1280 at Osstest/TestSupport.pm line 389.
        + rc=5

Very odd.

> >  test-amd64-i386-xl-qemut-win7-amd64 13 guest-localmigrate/x10 fail REGR. vs. 25945

        no active lease win.guest.osstest at ./ts-guest-localmigrate line 44.

I think we see these occasionally? The screenshot shows the guest
looking OK.

> >  test-amd64-i386-xl-qemuu-win7-amd64  5 xen-boot           fail REGR. vs. 25945

This was grain weevil. Looks like exim was slow to start, which might
have been DNS related, but it was only 30s, which seems like to little
to cause it to time out.

> The log appears to show that Dom0 booted up fine - were there any
> infrastructure problems (also taking into consideration the broken
> host-install above)?

Citrix IT were messing with the firewall configuration around here
yesterday, the change has been rolled back though. But I don't think the
machines which are involved here should have been affected since they
all live on the same subnet. But DNS might have been impacted for them I
guess.

Ian.

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [xen-unstable test] 25949: regressions - trouble: broken/fail/pass
  2014-04-23  9:45   ` Ian Campbell
@ 2014-04-23 10:02     ` Jan Beulich
  0 siblings, 0 replies; 6+ messages in thread
From: Jan Beulich @ 2014-04-23 10:02 UTC (permalink / raw)
  To: Ian Campbell; +Cc: xen-devel, ian.jackson

>>> On 23.04.14 at 11:45, <Ian.Campbell@citrix.com> wrote:
> On Wed, 2014-04-23 at 10:31 +0100, Jan Beulich wrote:
>> >>> On 23.04.14 at 03:08, <Ian.Jackson@eu.citrix.com> wrote:
>> >  test-amd64-i386-xl-qemut-win7-amd64 13 guest-localmigrate/x10 fail REGR. vs. 25945
> 
>         no active lease win.guest.osstest at ./ts-guest-localmigrate line 44.
> 
> I think we see these occasionally? The screenshot shows the guest
> looking OK.

Yes, the usual situation when these (randomly) fail. I never managed
to see anything that might hint at what it actually is that is going wrong
here every couple of runs.

Jan

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [xen-unstable test] 25949: regressions - trouble: broken/fail/pass
  2014-04-23  9:31 ` Jan Beulich
  2014-04-23  9:45   ` Ian Campbell
@ 2014-04-24 11:15   ` Ian Jackson
  2014-04-24 11:36     ` David Vrabel
  1 sibling, 1 reply; 6+ messages in thread
From: Ian Jackson @ 2014-04-24 11:15 UTC (permalink / raw)
  To: Jan Beulich; +Cc: xen-devel

Jan Beulich writes ("Re: [Xen-devel] [xen-unstable test] 25949: regressions - trouble: broken/fail/pass"):
> >>> On 23.04.14 at 03:08, <Ian.Jackson@eu.citrix.com> wrote:
> > flight 25949 xen-unstable real [real]
> > http://www.chiark.greenend.org.uk/~xensrcts/logs/25949/ 
> > 
> > Regressions :-(
> > 
> > Tests which did not succeed and are blocking,
> > including tests which could not be run:
> >  test-amd64-i386-xl-qemuu-win7-amd64  5 xen-boot fail REGR. vs. 25945
> 
> The log appears to show that Dom0 booted up fine - were there any
> infrastructure problems (also taking into consideration the broken
> host-install above)?

Actually, the serial log shows a problem.

Firstly, timing.  Unfortunately there is a discrepancy in the
timestamps because the new osstest vm didn't have ntp installed (now
fixed).  I'm going to suffix osstest vm timestamps with [o] and serial
log timestamps with [l].

18:10:41[o] is when osstest timed out waiting for
the system's boot scripts to complete.  It had been waiting since
18:08:59[o], which is when the sshd started accepting connections on
port 22 (all of this is from 5.ts-host-reboot.log).

At 18:10:42[o] osstest sends the first debug keys, requesting serial
input to go to xen (this is from 6.ts-logs-capture.log).  You can see
this in the serial log at 18:10:19[l] (in serial-grain-weevil.log).

In the serial log you can see that the host has in fact not finished
booting by this time.  osstest requests a whole bunch of debug keys
from Xen until 18:11:07[l], at which point it sends a debug key
request that is responded to by the dom0 kernel.  About 11 seconds
later, at 18:11:18[l], the dom0 finally prints its login prompt.

So I think in fact the dom0 kernel was stuck somehow and being prodded
woke it up.  Either that or it was running unconscionably slowly.

A network problem isn't a very good explanation for this failure
because by this point dom0 has been configured to use a static IP
address rather than dhcp.

> >  test-amd64-amd64-xl-qemuu-ovmf-amd64  3 host-install(3) broken REGR. vs. 25945

This is quite mysterious:

2014-04-22 21:21:43 Z executing ssh ... root@10.80.249.148 cat /proc/partitions
2014-04-22 21:21:43 Z | major minor  #blocks  name
2014-04-22 21:21:43 Z | 
2014-04-22 21:21:43 Z |   11        0    1048575 sr0
2014-04-22 21:21:43 Z |    8        0  244140625 sda
2014-04-22 21:21:43 Z |    8       16  244198584 sdb
2014-04-22 21:21:43 Z executing ssh ... root@10.80.249.148 pvcreate /dev/sdb
  Device /dev/sdb not found (or ignored by filtering).
2014-04-22 21:21:44 Z command nonzero waitstatus 1280: ssh -o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=100 -o ServerAliveInterval=100 -o PasswordAuthentication=no -o ChallengeResponseAuthentication=no -o UserKnownHostsFile=tmp/t.known_hosts_25949.test-amd64-amd64-xl-qemuu-ovmf-amd64 root@10.80.249.148 pvcreate /dev/sdb

Twice in a row.  But it doesn't seem to have been repeated after that.

Ian.

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [xen-unstable test] 25949: regressions - trouble: broken/fail/pass
  2014-04-24 11:15   ` Ian Jackson
@ 2014-04-24 11:36     ` David Vrabel
  0 siblings, 0 replies; 6+ messages in thread
From: David Vrabel @ 2014-04-24 11:36 UTC (permalink / raw)
  To: Ian Jackson; +Cc: xen-devel, Jan Beulich

On 24/04/14 12:15, Ian Jackson wrote:
> Jan Beulich writes ("Re: [Xen-devel] [xen-unstable test] 25949: regressions - trouble: broken/fail/pass"):
>>>>> On 23.04.14 at 03:08, <Ian.Jackson@eu.citrix.com> wrote:
>>> flight 25949 xen-unstable real [real]
>>> http://www.chiark.greenend.org.uk/~xensrcts/logs/25949/ 
>>>
>>> Regressions :-(
>>>
>>> Tests which did not succeed and are blocking,
>>> including tests which could not be run:
>>>  test-amd64-i386-xl-qemuu-win7-amd64  5 xen-boot fail REGR. vs. 25945
>>
>> The log appears to show that Dom0 booted up fine - were there any
>> infrastructure problems (also taking into consideration the broken
>> host-install above)?
> 
> Actually, the serial log shows a problem.
> 
> Firstly, timing.  Unfortunately there is a discrepancy in the
> timestamps because the new osstest vm didn't have ntp installed (now
> fixed).  I'm going to suffix osstest vm timestamps with [o] and serial
> log timestamps with [l].
> 
> 18:10:41[o] is when osstest timed out waiting for
> the system's boot scripts to complete.  It had been waiting since
> 18:08:59[o], which is when the sshd started accepting connections on
> port 22 (all of this is from 5.ts-host-reboot.log).
> 
> At 18:10:42[o] osstest sends the first debug keys, requesting serial
> input to go to xen (this is from 6.ts-logs-capture.log).  You can see
> this in the serial log at 18:10:19[l] (in serial-grain-weevil.log).
> 
> In the serial log you can see that the host has in fact not finished
> booting by this time.  osstest requests a whole bunch of debug keys
> from Xen until 18:11:07[l], at which point it sends a debug key
> request that is responded to by the dom0 kernel.  About 11 seconds
> later, at 18:11:18[l], the dom0 finally prints its login prompt.
> 
> So I think in fact the dom0 kernel was stuck somehow and being prodded
> woke it up.  Either that or it was running unconscionably slowly.
> 
> A network problem isn't a very good explanation for this failure
> because by this point dom0 has been configured to use a static IP
> address rather than dhcp.

Stating ntp appears to be stuck.  Could this is fallout from Tuesday's
lab firewall snafu?

David

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2014-04-24 11:36 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-04-23  1:08 [xen-unstable test] 25949: regressions - trouble: broken/fail/pass xen.org
2014-04-23  9:31 ` Jan Beulich
2014-04-23  9:45   ` Ian Campbell
2014-04-23 10:02     ` Jan Beulich
2014-04-24 11:15   ` Ian Jackson
2014-04-24 11:36     ` David Vrabel

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).