* [xen-unstable test] 102522: tolerable FAIL - PUSHED
@ 2016-11-23 15:54 osstest service owner
2016-11-24 15:14 ` some thoughts about merlot{0|1} issues [was: Re: [xen-unstable test] 102522: tolerable FAIL - PUSHED] Dario Faggioli
0 siblings, 1 reply; 12+ messages in thread
From: osstest service owner @ 2016-11-23 15:54 UTC (permalink / raw)
To: xen-devel, osstest-admin
flight 102522 xen-unstable real [real]
http://logs.test-lab.xenproject.org/osstest/logs/102522/
Failures :-/ but no regressions.
Tests which are failing intermittently (not blocking):
test-xtf-amd64-amd64-3 65 leak-check/check fail in 102500 pass in 102522
test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm 9 debian-hvm-install fail pass in 102500
test-amd64-amd64-rumprun-amd64 16 rumprun-demo-xenstorels/xenstorels.repeat fail pass in 102500
Regressions which are regarded as allowable (not blocking):
test-armhf-armhf-xl-rtds 15 guest-start/debian.repeat fail REGR. vs. 102465
test-armhf-armhf-libvirt-xsm 13 saverestore-support-check fail like 102465
test-amd64-i386-xl-qemut-win7-amd64 16 guest-stop fail like 102465
test-amd64-i386-xl-qemuu-win7-amd64 16 guest-stop fail like 102465
test-armhf-armhf-libvirt-qcow2 12 saverestore-support-check fail like 102465
test-armhf-armhf-libvirt-raw 12 saverestore-support-check fail like 102465
test-amd64-amd64-xl-qemut-win7-amd64 16 guest-stop fail like 102465
test-amd64-amd64-xl-qemuu-win7-amd64 16 guest-stop fail like 102465
test-armhf-armhf-libvirt 13 saverestore-support-check fail like 102465
test-amd64-amd64-xl-rtds 9 debian-install fail like 102465
Tests which did not succeed, but are not blocking:
test-amd64-amd64-libvirt 12 migrate-support-check fail never pass
test-amd64-i386-libvirt 12 migrate-support-check fail never pass
test-amd64-amd64-xl-pvh-intel 11 guest-start fail never pass
test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check fail never pass
test-amd64-amd64-libvirt-xsm 12 migrate-support-check fail never pass
test-amd64-amd64-xl-pvh-amd 11 guest-start fail never pass
test-amd64-i386-libvirt-xsm 12 migrate-support-check fail never pass
test-amd64-amd64-libvirt-vhd 11 migrate-support-check fail never pass
test-armhf-armhf-xl-arndale 12 migrate-support-check fail never pass
test-armhf-armhf-xl-arndale 13 saverestore-support-check fail never pass
test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check fail never pass
test-armhf-armhf-libvirt-xsm 12 migrate-support-check fail never pass
test-armhf-armhf-xl-multivcpu 12 migrate-support-check fail never pass
test-armhf-armhf-xl-multivcpu 13 saverestore-support-check fail never pass
test-armhf-armhf-xl 12 migrate-support-check fail never pass
test-armhf-armhf-xl 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-cubietruck 12 migrate-support-check fail never pass
test-armhf-armhf-xl-cubietruck 13 saverestore-support-check fail never pass
test-amd64-amd64-qemuu-nested-amd 16 debian-hvm-install/l1/l2 fail never pass
test-armhf-armhf-xl-credit2 12 migrate-support-check fail never pass
test-armhf-armhf-xl-credit2 13 saverestore-support-check fail never pass
test-armhf-armhf-libvirt-qcow2 11 migrate-support-check fail never pass
test-armhf-armhf-xl-xsm 12 migrate-support-check fail never pass
test-armhf-armhf-xl-xsm 13 saverestore-support-check fail never pass
test-armhf-armhf-libvirt-raw 11 migrate-support-check fail never pass
test-armhf-armhf-xl-vhd 11 migrate-support-check fail never pass
test-armhf-armhf-xl-vhd 12 saverestore-support-check fail never pass
test-armhf-armhf-libvirt 12 migrate-support-check fail never pass
test-armhf-armhf-xl-rtds 12 migrate-support-check fail never pass
test-armhf-armhf-xl-rtds 13 saverestore-support-check fail never pass
version targeted for testing:
xen f678e2c78110e73431217306bbd33c736802d700
baseline version:
xen 58bd0c7985890e0292212f94a56235228a3445c3
Last test of basis 102465 2016-11-21 01:58:00 Z 2 days
Failing since 102489 2016-11-21 17:49:05 Z 1 days 3 attempts
Testing same since 102500 2016-11-22 03:12:15 Z 1 days 2 attempts
------------------------------------------------------------
People who touched revisions under test:
Andrew Cooper <andrew.cooper3@citrix.com>
jobs:
build-amd64-xsm pass
build-armhf-xsm pass
build-i386-xsm pass
build-amd64-xtf pass
build-amd64 pass
build-armhf pass
build-i386 pass
build-amd64-libvirt pass
build-armhf-libvirt pass
build-i386-libvirt pass
build-amd64-oldkern pass
build-i386-oldkern pass
build-amd64-prev pass
build-i386-prev pass
build-amd64-pvops pass
build-armhf-pvops pass
build-i386-pvops pass
build-amd64-rumprun pass
build-i386-rumprun pass
test-xtf-amd64-amd64-1 pass
test-xtf-amd64-amd64-2 pass
test-xtf-amd64-amd64-3 pass
test-xtf-amd64-amd64-4 pass
test-xtf-amd64-amd64-5 pass
test-amd64-amd64-xl pass
test-armhf-armhf-xl pass
test-amd64-i386-xl pass
test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemut-debianhvm-amd64-xsm pass
test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm pass
test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm pass
test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm fail
test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm pass
test-amd64-amd64-libvirt-xsm pass
test-armhf-armhf-libvirt-xsm pass
test-amd64-i386-libvirt-xsm pass
test-amd64-amd64-xl-xsm pass
test-armhf-armhf-xl-xsm pass
test-amd64-i386-xl-xsm pass
test-amd64-amd64-qemuu-nested-amd fail
test-amd64-amd64-xl-pvh-amd fail
test-amd64-i386-qemut-rhel6hvm-amd pass
test-amd64-i386-qemuu-rhel6hvm-amd pass
test-amd64-amd64-xl-qemut-debianhvm-amd64 pass
test-amd64-i386-xl-qemut-debianhvm-amd64 pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-freebsd10-amd64 pass
test-amd64-amd64-xl-qemuu-ovmf-amd64 pass
test-amd64-i386-xl-qemuu-ovmf-amd64 pass
test-amd64-amd64-rumprun-amd64 fail
test-amd64-amd64-xl-qemut-win7-amd64 fail
test-amd64-i386-xl-qemut-win7-amd64 fail
test-amd64-amd64-xl-qemuu-win7-amd64 fail
test-amd64-i386-xl-qemuu-win7-amd64 fail
test-armhf-armhf-xl-arndale pass
test-amd64-amd64-xl-credit2 pass
test-armhf-armhf-xl-credit2 pass
test-armhf-armhf-xl-cubietruck pass
test-amd64-i386-freebsd10-i386 pass
test-amd64-i386-rumprun-i386 pass
test-amd64-amd64-qemuu-nested-intel pass
test-amd64-amd64-xl-pvh-intel fail
test-amd64-i386-qemut-rhel6hvm-intel pass
test-amd64-i386-qemuu-rhel6hvm-intel pass
test-amd64-amd64-libvirt pass
test-armhf-armhf-libvirt pass
test-amd64-i386-libvirt pass
test-amd64-amd64-migrupgrade pass
test-amd64-i386-migrupgrade pass
test-amd64-amd64-xl-multivcpu pass
test-armhf-armhf-xl-multivcpu pass
test-amd64-amd64-pair pass
test-amd64-i386-pair pass
test-amd64-amd64-libvirt-pair pass
test-amd64-i386-libvirt-pair pass
test-amd64-amd64-amd64-pvgrub pass
test-amd64-amd64-i386-pvgrub pass
test-amd64-amd64-pygrub pass
test-armhf-armhf-libvirt-qcow2 pass
test-amd64-amd64-xl-qcow2 pass
test-armhf-armhf-libvirt-raw pass
test-amd64-i386-xl-raw pass
test-amd64-amd64-xl-rtds fail
test-armhf-armhf-xl-rtds fail
test-amd64-i386-xl-qemut-winxpsp3-vcpus1 pass
test-amd64-i386-xl-qemuu-winxpsp3-vcpus1 pass
test-amd64-amd64-libvirt-vhd pass
test-armhf-armhf-xl-vhd pass
test-amd64-amd64-xl-qemut-winxpsp3 pass
test-amd64-i386-xl-qemut-winxpsp3 pass
test-amd64-amd64-xl-qemuu-winxpsp3 pass
test-amd64-i386-xl-qemuu-winxpsp3 pass
------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images
Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs
Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master
Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary
Pushing revision :
+ branch=xen-unstable
+ revision=f678e2c78110e73431217306bbd33c736802d700
+ . ./cri-lock-repos
++ . ./cri-common
+++ . ./cri-getconfig
+++ umask 002
+++ getrepos
++++ getconfig Repos
++++ perl -e '
use Osstest;
readglobalconfig();
print $c{"Repos"} or die $!;
'
+++ local repos=/home/osstest/repos
+++ '[' -z /home/osstest/repos ']'
+++ '[' '!' -d /home/osstest/repos ']'
+++ echo /home/osstest/repos
++ repos=/home/osstest/repos
++ repos_lock=/home/osstest/repos/lock
++ '[' x '!=' x/home/osstest/repos/lock ']'
++ OSSTEST_REPOS_LOCK_LOCKED=/home/osstest/repos/lock
++ exec with-lock-ex -w /home/osstest/repos/lock ./ap-push xen-unstable f678e2c78110e73431217306bbd33c736802d700
+ branch=xen-unstable
+ revision=f678e2c78110e73431217306bbd33c736802d700
+ . ./cri-lock-repos
++ . ./cri-common
+++ . ./cri-getconfig
+++ umask 002
+++ getrepos
++++ getconfig Repos
++++ perl -e '
use Osstest;
readglobalconfig();
print $c{"Repos"} or die $!;
'
+++ local repos=/home/osstest/repos
+++ '[' -z /home/osstest/repos ']'
+++ '[' '!' -d /home/osstest/repos ']'
+++ echo /home/osstest/repos
++ repos=/home/osstest/repos
++ repos_lock=/home/osstest/repos/lock
++ '[' x/home/osstest/repos/lock '!=' x/home/osstest/repos/lock ']'
+ . ./cri-common
++ . ./cri-getconfig
++ umask 002
+ select_xenbranch
+ case "$branch" in
+ tree=xen
+ xenbranch=xen-unstable
+ '[' xxen = xlinux ']'
+ linuxbranch=
+ '[' x = x ']'
+ qemuubranch=qemu-upstream-unstable
+ select_prevxenbranch
++ ./cri-getprevxenbranch xen-unstable
+ prevxenbranch=xen-4.7-testing
+ '[' xf678e2c78110e73431217306bbd33c736802d700 = x ']'
+ : tested/2.6.39.x
+ . ./ap-common
++ : osstest@xenbits.xen.org
+++ getconfig OsstestUpstream
+++ perl -e '
use Osstest;
readglobalconfig();
print $c{"OsstestUpstream"} or die $!;
'
++ :
++ : git://xenbits.xen.org/xen.git
++ : osstest@xenbits.xen.org:/home/xen/git/xen.git
++ : git://xenbits.xen.org/qemu-xen-traditional.git
++ : git://git.kernel.org
++ : git://git.kernel.org/pub/scm/linux/kernel/git
++ : git
++ : git://xenbits.xen.org/xtf.git
++ : osstest@xenbits.xen.org:/home/xen/git/xtf.git
++ : git://xenbits.xen.org/xtf.git
++ : git://xenbits.xen.org/libvirt.git
++ : osstest@xenbits.xen.org:/home/xen/git/libvirt.git
++ : git://xenbits.xen.org/libvirt.git
++ : git://xenbits.xen.org/osstest/rumprun.git
++ : git
++ : git://xenbits.xen.org/osstest/rumprun.git
++ : osstest@xenbits.xen.org:/home/xen/git/osstest/rumprun.git
++ : git://git.seabios.org/seabios.git
++ : osstest@xenbits.xen.org:/home/xen/git/osstest/seabios.git
++ : git://xenbits.xen.org/osstest/seabios.git
++ : https://github.com/tianocore/edk2.git
++ : osstest@xenbits.xen.org:/home/xen/git/osstest/ovmf.git
++ : git://xenbits.xen.org/osstest/ovmf.git
++ : git://xenbits.xen.org/osstest/linux-firmware.git
++ : osstest@xenbits.xen.org:/home/osstest/ext/linux-firmware.git
++ : git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
++ : osstest@xenbits.xen.org:/home/xen/git/linux-pvops.git
++ : git://xenbits.xen.org/linux-pvops.git
++ : tested/linux-3.14
++ : tested/linux-arm-xen
++ '[' xgit://xenbits.xen.org/linux-pvops.git = x ']'
++ '[' x = x ']'
++ : git://xenbits.xen.org/linux-pvops.git
++ : tested/linux-arm-xen
++ : git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
++ : tested/2.6.39.x
++ : daily-cron.xen-unstable
++ : daily-cron.xen-unstable
++ : daily-cron.xen-unstable
++ : daily-cron.xen-unstable
++ : daily-cron.xen-unstable
++ : daily-cron.xen-unstable
++ : daily-cron.xen-unstable
++ : http://hg.uk.xensource.com/carbon/trunk/linux-2.6.27
++ : git://xenbits.xen.org/qemu-xen.git
++ : osstest@xenbits.xen.org:/home/xen/git/qemu-xen.git
++ : daily-cron.xen-unstable
++ : git://xenbits.xen.org/qemu-xen.git
++ : git://git.qemu.org/qemu.git
+ TREE_LINUX=osstest@xenbits.xen.org:/home/xen/git/linux-pvops.git
+ TREE_QEMU_UPSTREAM=osstest@xenbits.xen.org:/home/xen/git/qemu-xen.git
+ TREE_XEN=osstest@xenbits.xen.org:/home/xen/git/xen.git
+ TREE_LIBVIRT=osstest@xenbits.xen.org:/home/xen/git/libvirt.git
+ TREE_RUMPRUN=osstest@xenbits.xen.org:/home/xen/git/osstest/rumprun.git
+ TREE_SEABIOS=osstest@xenbits.xen.org:/home/xen/git/osstest/seabios.git
+ TREE_OVMF=osstest@xenbits.xen.org:/home/xen/git/osstest/ovmf.git
+ TREE_XTF=osstest@xenbits.xen.org:/home/xen/git/xtf.git
+ info_linux_tree xen-unstable
+ case $1 in
+ return 1
+ case "$branch" in
+ cd /home/osstest/repos/xen
+ git push osstest@xenbits.xen.org:/home/xen/git/xen.git f678e2c78110e73431217306bbd33c736802d700:refs/heads/master
To osstest@xenbits.xen.org:/home/xen/git/xen.git
58bd0c7..f678e2c f678e2c78110e73431217306bbd33c736802d700 -> master
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* some thoughts about merlot{0|1} issues [was: Re: [xen-unstable test] 102522: tolerable FAIL - PUSHED]
2016-11-23 15:54 [xen-unstable test] 102522: tolerable FAIL - PUSHED osstest service owner
@ 2016-11-24 15:14 ` Dario Faggioli
2016-11-24 15:31 ` Jan Beulich
0 siblings, 1 reply; 12+ messages in thread
From: Dario Faggioli @ 2016-11-24 15:14 UTC (permalink / raw)
To: xen-devel; +Cc: Boris Ostrovsky, Ian Jackson, Wei Liu, Jan Beulich
[-- Attachment #1.1: Type: text/plain, Size: 5834 bytes --]
On Wed, 2016-11-23 at 15:54 +0000, osstest service owner wrote:
> flight 102522 xen-unstable real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/102522/
>
> Regressions which are regarded as allowable (not blocking):
> test-amd64-amd64-xl-rtds 9 debian-
> install fail like 102465
>
This is on merlot1, and as far as I can tell, this test is failing
there since quite a while (is that the correct interpretation of this
table?):
http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-rtds/xen-unstable
This is using RTDS as scheduler, but that should not be the problem. In
fact, what's failing is xen-create-image timing out.
Basically, it starts creating the VM filesystem via debootstrap, but
does not manage to finish doing that within the 2530 secs timeout:
http://logs.test-lab.xenproject.org/osstest/logs/102522/test-amd64-amd64-xl-rtds/9.ts-debian-install.log
2016-11-23 13:12:35 Z command timed out [2500]: timeout 2530 ssh -o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=100 -o ServerAliveInterval=100 -o PasswordAuthentication=no -o ChallengeResponseAuthentication=no -o UserKnownHostsFile=tmp/t.known_hosts_102522.test-amd64-amd64-xl-rtds root@172.16.144.21 http_proxy=http://cache:3143/ \
xen-create-image \
In other runs on different hosts, still under RTDS, that takes about
650 seconds:
http://logs.test-lab.xenproject.org/osstest/logs/102532/test-amd64-amd64-xl-rtds/9.ts-debian-install.log
And I've tried myself on my test box, and it took 10m20s.
We know from here, that, this time, it got stuck rather early:
http://logs.test-lab.xenproject.org/osstest/logs/102522/test-amd64-amd64-xl-rtds/merlot1---var-log-xen-tools-debian.guest.osstest.log
I've looked at a handful of other instances, and it seems to be
_always_ like that.
The system appears alive though, or at least right after the timeout,
the ts-log-capture phase --which includes issuing commands on the host
and copying files from there-- succeeds.
Also, not sure it means much, but xen-create-image starts at
12:30:55 and times out at 13:12:35. Looking in:
http://logs.test-lab.xenproject.org/osstest/logs/102522/test-amd64-amd64-xl-rtds/merlot1---var-log-daemon.log
we see:
Nov 23 12:04:55 merlot1 ntpd[3003]: Listening on routing socket on fd #22 for interface updates
[..]
Nov 23 12:27:22 merlot1 init: Id "T0" respawning too fast: disabled for 5 minutes
Nov 23 12:34:04 merlot1 init: Id "T0" respawning too fast: disabled for 5 minutes
Nov 23 12:40:47 merlot1 init: Id "T0" respawning too fast: disabled for 5 minutes
Nov 23 12:47:31 merlot1 init: Id "T0" respawning too fast: disabled for 5 minutes
Nov 23 12:54:14 merlot1 init: Id "T0" respawning too fast: disabled for 5 minutes
Nov 23 13:00:57 merlot1 init: Id "T0" respawning too fast: disabled for 5 minutes
Nov 23 13:07:40 merlot1 init: Id "T0" respawning too fast: disabled for 5 minutes
So, again, at least something is alive on the host, and writing in the
logs, even during the time that debootstrap seems stuck.
There are 2 running vcpus:
Name ID VCPU CPU State Time(s) Affinity (Hard / Soft)
Domain-0 0 1 2 r-- 35.7 all / all
Domain-0 0 17 9 r-- 38.3 all / all
vcpu 17 is running on CPU 9 which is on node 1, which has _0_ memory:
node: memsize memfree distances
0: 9216 7856 10,16,16,16
1: 0 0 16,10,16,16
2: 8175 7779 16,16,10,16
3: 0 0 16,16,16,10
(see here: http://logs.test-lab.xenproject.org/osstest/logs/102522/test-amd64-amd64-xl-rtds/merlot1-output-xl_info_-n )
but that should not mean much, and in other, still failing, runs, this
does not happen (also, this is just what is going on while we collect
logs).
Now, about serial output:
http://logs.test-lab.xenproject.org/osstest/logs/102522/test-amd64-amd64-xl-rtds/serial-merlot1.log
When dumping ACPI C states, here's how things look like for _all_ CPUs:
Nov 23 13:13:00.382134 (XEN) ==cpu3==
Nov 23 13:13:00.382157 (XEN) active state: C-1
Nov 23 13:13:00.390096 (XEN) max_cstate: C7
Nov 23 13:13:00.390125 (XEN) states:
Nov 23 13:13:00.390148 (XEN) C1: type[C1] latency[000] usage[00000000] method[ HALT] duration[0]
Nov 23 13:13:00.398055 (XEN) C0: usage[00000000] duration[4229118701384]
Nov 23 13:13:00.398090 (XEN) PC2[0] PC3[0] PC6[0] PC7[0]
Nov 23 13:13:00.406088 (XEN) CC3[0] CC6[0] CC7[0]
And I checked other runs, and it's the same everywhere.
I remember that Jan suggested trying to pass max_cstate=1 to Xen at
boot. I was about to ask Ian to do that for this host, but it looks
like we're using only C0 and C1 already anyway.
Boot command line looks like this:
xen_commandline : placeholder conswitch=x watchdog com1=115200,8n1 console=com1,vga gdb=com1 dom0_mem=512M,max:512M ucode=scan sched=rtds
which makes the above look a bit weird to me... But I've played much
more with Intel boxes than with AMD ones, I admit.
For now, I'm done. At some point, I'll recall either merlot0 or merlot1
out of OSSTest, take it back for myself, and try to investigate more.
If, it in the meantime, any of this rings a bell for anyone, feel free
to speak up.
Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* some thoughts about merlot{0|1} issues [was: Re: [xen-unstable test] 102522: tolerable FAIL - PUSHED]
2016-11-24 15:14 ` some thoughts about merlot{0|1} issues [was: Re: [xen-unstable test] 102522: tolerable FAIL - PUSHED] Dario Faggioli
@ 2016-11-24 15:31 ` Jan Beulich
2016-11-28 13:48 ` Boris Ostrovsky
0 siblings, 1 reply; 12+ messages in thread
From: Jan Beulich @ 2016-11-24 15:31 UTC (permalink / raw)
To: Dario Faggioli; +Cc: xen-devel, Boris Ostrovsky, Ian Jackson, Wei Liu
>>> On 24.11.16 at 16:14, <dario.faggioli@citrix.com> wrote:
> When dumping ACPI C states, here's how things look like for _all_ CPUs:
> Nov 23 13:13:00.382134 (XEN) ==cpu3==
> Nov 23 13:13:00.382157 (XEN) active state: C-1
> Nov 23 13:13:00.390096 (XEN) max_cstate: C7
> Nov 23 13:13:00.390125 (XEN) states:
> Nov 23 13:13:00.390148 (XEN) C1: type[C1] latency[000] usage[00000000] method[ HALT] duration[0]
> Nov 23 13:13:00.398055 (XEN) C0: usage[00000000] duration[4229118701384]
> Nov 23 13:13:00.398090 (XEN) PC2[0] PC3[0] PC6[0] PC7[0]
> Nov 23 13:13:00.406088 (XEN) CC3[0] CC6[0] CC7[0]
>
> And I checked other runs, and it's the same everywhere.
>
> I remember that Jan suggested trying to pass max_cstate=1 to Xen at
> boot. I was about to ask Ian to do that for this host, but it looks
> like we're using only C0 and C1 already anyway.
This indeed looks surprising for a half way modern system - is the
BIOS perhaps limiting C-states (maybe instructed to via BIOS setup)?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: some thoughts about merlot{0|1} issues [was: Re: [xen-unstable test] 102522: tolerable FAIL - PUSHED]
2016-11-24 15:31 ` Jan Beulich
@ 2016-11-28 13:48 ` Boris Ostrovsky
2016-11-28 15:16 ` Konrad Rzeszutek Wilk
2016-11-28 17:19 ` Dario Faggioli
0 siblings, 2 replies; 12+ messages in thread
From: Boris Ostrovsky @ 2016-11-28 13:48 UTC (permalink / raw)
To: Jan Beulich, Dario Faggioli; +Cc: xen-devel, Ian Jackson, Wei Liu
On 11/24/2016 10:31 AM, Jan Beulich wrote:
>>>> On 24.11.16 at 16:14, <dario.faggioli@citrix.com> wrote:
>> When dumping ACPI C states, here's how things look like for _all_ CPUs:
>> Nov 23 13:13:00.382134 (XEN) ==cpu3==
>> Nov 23 13:13:00.382157 (XEN) active state: C-1
>> Nov 23 13:13:00.390096 (XEN) max_cstate: C7
>> Nov 23 13:13:00.390125 (XEN) states:
>> Nov 23 13:13:00.390148 (XEN) C1: type[C1] latency[000] usage[00000000] method[ HALT] duration[0]
>> Nov 23 13:13:00.398055 (XEN) C0: usage[00000000] duration[4229118701384]
>> Nov 23 13:13:00.398090 (XEN) PC2[0] PC3[0] PC6[0] PC7[0]
>> Nov 23 13:13:00.406088 (XEN) CC3[0] CC6[0] CC7[0]
>>
>> And I checked other runs, and it's the same everywhere.
>>
>> I remember that Jan suggested trying to pass max_cstate=1 to Xen at
>> boot. I was about to ask Ian to do that for this host, but it looks
>> like we're using only C0 and C1 already anyway.
> This indeed looks surprising for a half way modern system - is the
> BIOS perhaps limiting C-states (maybe instructed to via BIOS setup)?
IIRC some BIOSes indeed provided an option to disable C2. Dumping SSDT
would tell us whether C2 is there is BIOS is not accessible.
-boris
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: some thoughts about merlot{0|1} issues [was: Re: [xen-unstable test] 102522: tolerable FAIL - PUSHED]
2016-11-28 13:48 ` Boris Ostrovsky
@ 2016-11-28 15:16 ` Konrad Rzeszutek Wilk
2016-11-28 16:08 ` Andrew Cooper
2016-11-28 16:45 ` Dario Faggioli
2016-11-28 17:19 ` Dario Faggioli
1 sibling, 2 replies; 12+ messages in thread
From: Konrad Rzeszutek Wilk @ 2016-11-28 15:16 UTC (permalink / raw)
To: Boris Ostrovsky
Cc: xen-devel, Dario Faggioli, Ian Jackson, Wei Liu, Jan Beulich
On Mon, Nov 28, 2016 at 08:48:30AM -0500, Boris Ostrovsky wrote:
> On 11/24/2016 10:31 AM, Jan Beulich wrote:
> >>>> On 24.11.16 at 16:14, <dario.faggioli@citrix.com> wrote:
> >> When dumping ACPI C states, here's how things look like for _all_ CPUs:
> >> Nov 23 13:13:00.382134 (XEN) ==cpu3==
> >> Nov 23 13:13:00.382157 (XEN) active state: C-1
> >> Nov 23 13:13:00.390096 (XEN) max_cstate: C7
> >> Nov 23 13:13:00.390125 (XEN) states:
> >> Nov 23 13:13:00.390148 (XEN) C1: type[C1] latency[000] usage[00000000] method[ HALT] duration[0]
> >> Nov 23 13:13:00.398055 (XEN) C0: usage[00000000] duration[4229118701384]
> >> Nov 23 13:13:00.398090 (XEN) PC2[0] PC3[0] PC6[0] PC7[0]
> >> Nov 23 13:13:00.406088 (XEN) CC3[0] CC6[0] CC7[0]
> >>
> >> And I checked other runs, and it's the same everywhere.
> >>
> >> I remember that Jan suggested trying to pass max_cstate=1 to Xen at
> >> boot. I was about to ask Ian to do that for this host, but it looks
> >> like we're using only C0 and C1 already anyway.
> > This indeed looks surprising for a half way modern system - is the
> > BIOS perhaps limiting C-states (maybe instructed to via BIOS setup)?
>
>
> IIRC some BIOSes indeed provided an option to disable C2. Dumping SSDT
> would tell us whether C2 is there is BIOS is not accessible.
Keep in mind that not all C states are exposed if MWAIT is not exposed
to dom0 (see xen_check_mwait in arch/x86/xen/enlighten.c).
I recall some emails from Andrew about the CPU faulting and his CPUID
patches inhibiting this but it may very well be fixed?
Dario, lastly - did you have xen-acpi-processor loaded or built in your kernel?
>
> -boris
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: some thoughts about merlot{0|1} issues [was: Re: [xen-unstable test] 102522: tolerable FAIL - PUSHED]
2016-11-28 15:16 ` Konrad Rzeszutek Wilk
@ 2016-11-28 16:08 ` Andrew Cooper
2016-11-28 16:20 ` Jan Beulich
2016-11-28 16:45 ` Dario Faggioli
1 sibling, 1 reply; 12+ messages in thread
From: Andrew Cooper @ 2016-11-28 16:08 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk, Boris Ostrovsky
Cc: xen-devel, Dario Faggioli, Ian Jackson, Wei Liu, Jan Beulich
On 28/11/16 15:16, Konrad Rzeszutek Wilk wrote:
> On Mon, Nov 28, 2016 at 08:48:30AM -0500, Boris Ostrovsky wrote:
>> On 11/24/2016 10:31 AM, Jan Beulich wrote:
>>>>>> On 24.11.16 at 16:14, <dario.faggioli@citrix.com> wrote:
>>>> When dumping ACPI C states, here's how things look like for _all_ CPUs:
>>>> Nov 23 13:13:00.382134 (XEN) ==cpu3==
>>>> Nov 23 13:13:00.382157 (XEN) active state: C-1
>>>> Nov 23 13:13:00.390096 (XEN) max_cstate: C7
>>>> Nov 23 13:13:00.390125 (XEN) states:
>>>> Nov 23 13:13:00.390148 (XEN) C1: type[C1] latency[000] usage[00000000] method[ HALT] duration[0]
>>>> Nov 23 13:13:00.398055 (XEN) C0: usage[00000000] duration[4229118701384]
>>>> Nov 23 13:13:00.398090 (XEN) PC2[0] PC3[0] PC6[0] PC7[0]
>>>> Nov 23 13:13:00.406088 (XEN) CC3[0] CC6[0] CC7[0]
>>>>
>>>> And I checked other runs, and it's the same everywhere.
>>>>
>>>> I remember that Jan suggested trying to pass max_cstate=1 to Xen at
>>>> boot. I was about to ask Ian to do that for this host, but it looks
>>>> like we're using only C0 and C1 already anyway.
>>> This indeed looks surprising for a half way modern system - is the
>>> BIOS perhaps limiting C-states (maybe instructed to via BIOS setup)?
>>
>> IIRC some BIOSes indeed provided an option to disable C2. Dumping SSDT
>> would tell us whether C2 is there is BIOS is not accessible.
> Keep in mind that not all C states are exposed if MWAIT is not exposed
> to dom0 (see xen_check_mwait in arch/x86/xen/enlighten.c).
>
> I recall some emails from Andrew about the CPU faulting and his CPUID
> patches inhibiting this but it may very well be fixed?
merlot is AMD hardware, so no faulting.
Also, Xen purposefully leaks EIST into the control domain kernel to work
around that particular Linux bug.
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: some thoughts about merlot{0|1} issues [was: Re: [xen-unstable test] 102522: tolerable FAIL - PUSHED]
2016-11-28 16:08 ` Andrew Cooper
@ 2016-11-28 16:20 ` Jan Beulich
0 siblings, 0 replies; 12+ messages in thread
From: Jan Beulich @ 2016-11-28 16:20 UTC (permalink / raw)
To: Andrew Cooper
Cc: Dario Faggioli, Ian Jackson, xen-devel, Boris Ostrovsky, Wei Liu
>>> On 28.11.16 at 17:08, <andrew.cooper3@citrix.com> wrote:
> On 28/11/16 15:16, Konrad Rzeszutek Wilk wrote:
>> On Mon, Nov 28, 2016 at 08:48:30AM -0500, Boris Ostrovsky wrote:
>>> On 11/24/2016 10:31 AM, Jan Beulich wrote:
>>>>>>> On 24.11.16 at 16:14, <dario.faggioli@citrix.com> wrote:
>>>>> When dumping ACPI C states, here's how things look like for _all_ CPUs:
>>>>> Nov 23 13:13:00.382134 (XEN) ==cpu3==
>>>>> Nov 23 13:13:00.382157 (XEN) active state: C-1
>>>>> Nov 23 13:13:00.390096 (XEN) max_cstate: C7
>>>>> Nov 23 13:13:00.390125 (XEN) states:
>>>>> Nov 23 13:13:00.390148 (XEN) C1: type[C1] latency[000] usage[00000000]
> method[ HALT] duration[0]
>>>>> Nov 23 13:13:00.398055 (XEN) C0: usage[00000000] duration[4229118701384]
>>>>> Nov 23 13:13:00.398090 (XEN) PC2[0] PC3[0] PC6[0] PC7[0]
>>>>> Nov 23 13:13:00.406088 (XEN) CC3[0] CC6[0] CC7[0]
>>>>>
>>>>> And I checked other runs, and it's the same everywhere.
>>>>>
>>>>> I remember that Jan suggested trying to pass max_cstate=1 to Xen at
>>>>> boot. I was about to ask Ian to do that for this host, but it looks
>>>>> like we're using only C0 and C1 already anyway.
>>>> This indeed looks surprising for a half way modern system - is the
>>>> BIOS perhaps limiting C-states (maybe instructed to via BIOS setup)?
>>>
>>> IIRC some BIOSes indeed provided an option to disable C2. Dumping SSDT
>>> would tell us whether C2 is there is BIOS is not accessible.
>> Keep in mind that not all C states are exposed if MWAIT is not exposed
>> to dom0 (see xen_check_mwait in arch/x86/xen/enlighten.c).
>>
>> I recall some emails from Andrew about the CPU faulting and his CPUID
>> patches inhibiting this but it may very well be fixed?
>
> merlot is AMD hardware, so no faulting.
>
> Also, Xen purposefully leaks EIST into the control domain kernel to work
> around that particular Linux bug.
Isn't EIST an Intel-only thing?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: some thoughts about merlot{0|1} issues [was: Re: [xen-unstable test] 102522: tolerable FAIL - PUSHED]
2016-11-28 15:16 ` Konrad Rzeszutek Wilk
2016-11-28 16:08 ` Andrew Cooper
@ 2016-11-28 16:45 ` Dario Faggioli
2016-11-28 17:06 ` Dario Faggioli
1 sibling, 1 reply; 12+ messages in thread
From: Dario Faggioli @ 2016-11-28 16:45 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk, Boris Ostrovsky
Cc: xen-devel, Ian Jackson, Wei Liu, Jan Beulich
[-- Attachment #1.1: Type: text/plain, Size: 1123 bytes --]
On Mon, 2016-11-28 at 10:16 -0500, Konrad Rzeszutek Wilk wrote:
> On Mon, Nov 28, 2016 at 08:48:30AM -0500, Boris Ostrovsky wrote:
> > IIRC some BIOSes indeed provided an option to disable C2. Dumping
> > SSDT
> > would tell us whether C2 is there is BIOS is not accessible.
>
> Dario, lastly - did you have xen-acpi-processor loaded or built in
> your kernel?
>
I'm sure it's there.
In fact, this is an OSSTest box in the colo. I've tried have a look
right now, without taking it off from OSSTest, but it does not have Xen
installed yet (I don't know what job it's executing).
Anyway, when the tests run, the kernel in use is the one build for all
the various tests and boxes of a flight, and it does has
XEN_ACPI_PROCESSOR. That's why I'm saying that I am sure it is there.
Anyway, I'll double check that.
Thanks and Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: some thoughts about merlot{0|1} issues [was: Re: [xen-unstable test] 102522: tolerable FAIL - PUSHED]
2016-11-28 16:45 ` Dario Faggioli
@ 2016-11-28 17:06 ` Dario Faggioli
0 siblings, 0 replies; 12+ messages in thread
From: Dario Faggioli @ 2016-11-28 17:06 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk, Boris Ostrovsky
Cc: xen-devel, Ian Jackson, Wei Liu, Jan Beulich
[-- Attachment #1.1: Type: text/plain, Size: 915 bytes --]
On Mon, 2016-11-28 at 17:45 +0100, Dario Faggioli wrote:
> On Mon, 2016-11-28 at 10:16 -0500, Konrad Rzeszutek Wilk wrote:
> >
> > Dario, lastly - did you have xen-acpi-processor loaded or built in
> > your kernel?
> >
> I'm sure it's there.
>
And I was right, it is there:
root@merlot1:~# zcat /proc/config.gz |grep -i xen_acpi
CONFIG_XEN_ACPI_PROCESSOR=m
But it's not loaded:
root@merlot1:~# lsmod |grep xen
xen_gntalloc 3632 0
And if I try to load it, here's what happens:
root@merlot1:~# modprobe xen-acpi-processor
modprobe: ERROR: could not insert 'xen_acpi_processor': No such device
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: some thoughts about merlot{0|1} issues [was: Re: [xen-unstable test] 102522: tolerable FAIL - PUSHED]
2016-11-28 13:48 ` Boris Ostrovsky
2016-11-28 15:16 ` Konrad Rzeszutek Wilk
@ 2016-11-28 17:19 ` Dario Faggioli
2016-11-28 18:27 ` Boris Ostrovsky
1 sibling, 1 reply; 12+ messages in thread
From: Dario Faggioli @ 2016-11-28 17:19 UTC (permalink / raw)
To: Boris Ostrovsky, Jan Beulich; +Cc: xen-devel, Ian Jackson, Wei Liu
[-- Attachment #1.1: Type: text/plain, Size: 949 bytes --]
On Mon, 2016-11-28 at 08:48 -0500, Boris Ostrovsky wrote:
> On 11/24/2016 10:31 AM, Jan Beulich wrote:
> >
> > This indeed looks surprising for a half way modern system - is the
> > BIOS perhaps limiting C-states (maybe instructed to via BIOS
> > setup)?
>
> IIRC some BIOSes indeed provided an option to disable C2. Dumping
> SSDT
> would tell us whether C2 is there is BIOS is not accessible.
>
I will try to extract that.
In any case, what I'm wondering is, could this ACPI / PM thing be
linked to the behavior we are seeing?
I'm not sure I see how... perhaps the system gets too hot and its
throttled down? (Yes, shooting in the dark, I know.)
Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: some thoughts about merlot{0|1} issues [was: Re: [xen-unstable test] 102522: tolerable FAIL - PUSHED]
2016-11-28 17:19 ` Dario Faggioli
@ 2016-11-28 18:27 ` Boris Ostrovsky
2016-11-29 11:06 ` Dario Faggioli
0 siblings, 1 reply; 12+ messages in thread
From: Boris Ostrovsky @ 2016-11-28 18:27 UTC (permalink / raw)
To: Dario Faggioli, Jan Beulich; +Cc: xen-devel, Ian Jackson, Wei Liu
[-- Attachment #1.1.1: Type: text/plain, Size: 1567 bytes --]
On 11/28/2016 12:19 PM, Dario Faggioli wrote:
> On Mon, 2016-11-28 at 08:48 -0500, Boris Ostrovsky wrote:
>> On 11/24/2016 10:31 AM, Jan Beulich wrote:
>>>
>>> This indeed looks surprising for a half way modern system - is the
>>> BIOS perhaps limiting C-states (maybe instructed to via BIOS
>>> setup)?
>> IIRC some BIOSes indeed provided an option to disable C2. Dumping
>> SSDT
>> would tell us whether C2 is there is BIOS is not accessible.
>>
> I will try to extract that.
>
> In any case, what I'm wondering is, could this ACPI / PM thing be
> linked to the behavior we are seeing?
I find it somewhat unlikely.
BTW, I wouldn't worry about C2 specifically --- it's pretty clear that
no C-state stats are collected at all:
Nov 28 08:07:42.146057 (XEN) active state: C-1
Nov 28 08:07:42.153968 (XEN) max_cstate: C7
Nov 28 08:07:42.153998 (XEN) states:
Nov 28 08:07:42.154021 (XEN) C1: type[C1] latency[000] usage[00000000] method[ HALT] duration[0]
Nov 28 08:07:42.161992 (XEN) C0: usage[00000000] duration[3907882618433]
duration is the only reported value and most likely it's a NOW(). And
the reason C1 is still mentioned is because it is a required C-state,
whether or not it's in SSDT.
It is strange but I don't see anything in the serial log that would
indicate a problem.
>
> I'm not sure I see how... perhaps the system gets too hot and its
> throttled down? (Yes, shooting in the dark, I know.)
I'd expect an event to be triggered (SMI?) and a warning of some sort to
be printed.
-boris
[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: some thoughts about merlot{0|1} issues [was: Re: [xen-unstable test] 102522: tolerable FAIL - PUSHED]
2016-11-28 18:27 ` Boris Ostrovsky
@ 2016-11-29 11:06 ` Dario Faggioli
0 siblings, 0 replies; 12+ messages in thread
From: Dario Faggioli @ 2016-11-29 11:06 UTC (permalink / raw)
To: Boris Ostrovsky, Jan Beulich; +Cc: xen-devel, Ian Jackson, Wei Liu
[-- Attachment #1.1: Type: text/plain, Size: 1253 bytes --]
On Mon, 2016-11-28 at 13:27 -0500, Boris Ostrovsky wrote:
> On 11/28/2016 12:19 PM, Dario Faggioli wrote:
> >
> > In any case, what I'm wondering is, could this ACPI / PM thing be
> > linked to the behavior we are seeing?
>
> I find it somewhat unlikely.
>
> BTW, I wouldn't worry about C2 specifically --- it's pretty clear
> that
> no C-state stats are collected at all:
[..]
> duration is the only reported value and most likely it's a NOW(). And
> the reason C1 is still mentioned is because it is a required C-state,
> whether or not it's in SSDT.
>
I see.
> It is strange but I don't see anything in the serial log that would
> indicate a problem.
>
Ok, thanks for looking. :-)
> > I'm not sure I see how... perhaps the system gets too hot and its
> > throttled down? (Yes, shooting in the dark, I know.)
>
> I'd expect an event to be triggered (SMI?) and a warning of some sort
> to
> be printed.
>
Indeed.
Thanks again and Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2016-11-29 11:06 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-11-23 15:54 [xen-unstable test] 102522: tolerable FAIL - PUSHED osstest service owner
2016-11-24 15:14 ` some thoughts about merlot{0|1} issues [was: Re: [xen-unstable test] 102522: tolerable FAIL - PUSHED] Dario Faggioli
2016-11-24 15:31 ` Jan Beulich
2016-11-28 13:48 ` Boris Ostrovsky
2016-11-28 15:16 ` Konrad Rzeszutek Wilk
2016-11-28 16:08 ` Andrew Cooper
2016-11-28 16:20 ` Jan Beulich
2016-11-28 16:45 ` Dario Faggioli
2016-11-28 17:06 ` Dario Faggioli
2016-11-28 17:19 ` Dario Faggioli
2016-11-28 18:27 ` Boris Ostrovsky
2016-11-29 11:06 ` Dario Faggioli
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).