* [xen-unstable test] 118078: regressions - FAIL
@ 2018-01-16 8:43 osstest service owner
2018-01-16 8:57 ` Jan Beulich
0 siblings, 1 reply; 7+ messages in thread
From: osstest service owner @ 2018-01-16 8:43 UTC (permalink / raw)
To: xen-devel, osstest-admin
flight 118078 xen-unstable real [real]
http://logs.test-lab.xenproject.org/osstest/logs/118078/
Regressions :-(
Tests which did not succeed and are blocking,
including tests which could not be run:
build-arm64-pvops 6 kernel-build fail REGR. vs. 118003
test-amd64-amd64-xl-qemut-win7-amd64 10 windows-install fail REGR. vs. 118003
Tests which did not succeed, but are not blocking:
test-arm64-arm64-xl 1 build-check(1) blocked n/a
test-arm64-arm64-xl-credit2 1 build-check(1) blocked n/a
test-arm64-arm64-examine 1 build-check(1) blocked n/a
test-arm64-arm64-xl-xsm 1 build-check(1) blocked n/a
test-arm64-arm64-libvirt-xsm 1 build-check(1) blocked n/a
test-armhf-armhf-libvirt 14 saverestore-support-check fail like 118003
test-armhf-armhf-libvirt-xsm 14 saverestore-support-check fail like 118003
test-amd64-amd64-xl-qemuu-win7-amd64 17 guest-stop fail like 118003
test-amd64-i386-xl-qemuu-win7-amd64 17 guest-stop fail like 118003
test-amd64-amd64-xl-qemuu-ws16-amd64 17 guest-stop fail like 118003
test-amd64-i386-xl-qemut-win7-amd64 17 guest-stop fail like 118003
test-armhf-armhf-libvirt-raw 13 saverestore-support-check fail like 118003
test-amd64-i386-xl-qemuu-ws16-amd64 17 guest-stop fail like 118003
test-amd64-amd64-xl-qemut-ws16-amd64 17 guest-stop fail like 118003
test-amd64-amd64-xl-pvhv2-intel 12 guest-start fail never pass
test-amd64-amd64-xl-pvhv2-amd 12 guest-start fail never pass
test-amd64-amd64-libvirt 13 migrate-support-check fail never pass
test-amd64-amd64-libvirt-xsm 13 migrate-support-check fail never pass
test-amd64-i386-libvirt 13 migrate-support-check fail never pass
test-amd64-i386-libvirt-xsm 13 migrate-support-check fail never pass
test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm 11 migrate-support-check fail never pass
test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 11 migrate-support-check fail never pass
test-amd64-amd64-qemuu-nested-amd 17 debian-hvm-install/l1/l2 fail never pass
test-amd64-amd64-libvirt-vhd 12 migrate-support-check fail never pass
test-armhf-armhf-xl 13 migrate-support-check fail never pass
test-armhf-armhf-xl 14 saverestore-support-check fail never pass
test-armhf-armhf-xl-xsm 13 migrate-support-check fail never pass
test-armhf-armhf-xl-xsm 14 saverestore-support-check fail never pass
test-armhf-armhf-libvirt 13 migrate-support-check fail never pass
test-armhf-armhf-xl-cubietruck 13 migrate-support-check fail never pass
test-armhf-armhf-libvirt-xsm 13 migrate-support-check fail never pass
test-armhf-armhf-xl-cubietruck 14 saverestore-support-check fail never pass
test-armhf-armhf-xl-multivcpu 13 migrate-support-check fail never pass
test-armhf-armhf-xl-multivcpu 14 saverestore-support-check fail never pass
test-armhf-armhf-xl-rtds 13 migrate-support-check fail never pass
test-armhf-armhf-xl-rtds 14 saverestore-support-check fail never pass
test-armhf-armhf-libvirt-raw 12 migrate-support-check fail never pass
test-armhf-armhf-xl-vhd 12 migrate-support-check fail never pass
test-armhf-armhf-xl-vhd 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-credit2 13 migrate-support-check fail never pass
test-armhf-armhf-xl-credit2 14 saverestore-support-check fail never pass
test-armhf-armhf-xl-arndale 13 migrate-support-check fail never pass
test-armhf-armhf-xl-arndale 14 saverestore-support-check fail never pass
test-amd64-i386-xl-qemut-ws16-amd64 17 guest-stop fail never pass
test-amd64-i386-xl-qemuu-win10-i386 10 windows-install fail never pass
test-amd64-amd64-xl-qemuu-win10-i386 10 windows-install fail never pass
test-amd64-amd64-xl-qemut-win10-i386 10 windows-install fail never pass
test-amd64-i386-xl-qemut-win10-i386 10 windows-install fail never pass
version targeted for testing:
xen bd61fe94bee0556bc2f64999a4a8315b93f90f21
baseline version:
xen 2d70b54e055635ff60526b6949156504b6194b7c
Last test of basis 118003 2018-01-15 04:26:28 Z 1 days
Testing same since 118078 2018-01-15 19:14:52 Z 0 days 1 attempts
------------------------------------------------------------
People who touched revisions under test:
Andrew Cooper <andrew.cooper3@citrix.com>
jobs:
build-amd64-xsm pass
build-arm64-xsm pass
build-armhf-xsm pass
build-i386-xsm pass
build-amd64-xtf pass
build-amd64 pass
build-arm64 pass
build-armhf pass
build-i386 pass
build-amd64-libvirt pass
build-arm64-libvirt pass
build-armhf-libvirt pass
build-i386-libvirt pass
build-amd64-prev pass
build-i386-prev pass
build-amd64-pvops pass
build-arm64-pvops fail
build-armhf-pvops pass
build-i386-pvops pass
build-amd64-rumprun pass
build-i386-rumprun pass
test-xtf-amd64-amd64-1 pass
test-xtf-amd64-amd64-2 pass
test-xtf-amd64-amd64-3 pass
test-xtf-amd64-amd64-4 pass
test-xtf-amd64-amd64-5 pass
test-amd64-amd64-xl pass
test-arm64-arm64-xl blocked
test-armhf-armhf-xl pass
test-amd64-i386-xl pass
test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemut-debianhvm-amd64-xsm pass
test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm pass
test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm pass
test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm pass
test-amd64-amd64-libvirt-xsm pass
test-arm64-arm64-libvirt-xsm blocked
test-armhf-armhf-libvirt-xsm pass
test-amd64-i386-libvirt-xsm pass
test-amd64-amd64-xl-xsm pass
test-arm64-arm64-xl-xsm blocked
test-armhf-armhf-xl-xsm pass
test-amd64-i386-xl-xsm pass
test-amd64-amd64-qemuu-nested-amd fail
test-amd64-amd64-xl-pvhv2-amd fail
test-amd64-i386-qemut-rhel6hvm-amd pass
test-amd64-i386-qemuu-rhel6hvm-amd pass
test-amd64-amd64-xl-qemut-debianhvm-amd64 pass
test-amd64-i386-xl-qemut-debianhvm-amd64 pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-freebsd10-amd64 pass
test-amd64-amd64-xl-qemuu-ovmf-amd64 pass
test-amd64-i386-xl-qemuu-ovmf-amd64 pass
test-amd64-amd64-rumprun-amd64 pass
test-amd64-amd64-xl-qemut-win7-amd64 fail
test-amd64-i386-xl-qemut-win7-amd64 fail
test-amd64-amd64-xl-qemuu-win7-amd64 fail
test-amd64-i386-xl-qemuu-win7-amd64 fail
test-amd64-amd64-xl-qemut-ws16-amd64 fail
test-amd64-i386-xl-qemut-ws16-amd64 fail
test-amd64-amd64-xl-qemuu-ws16-amd64 fail
test-amd64-i386-xl-qemuu-ws16-amd64 fail
test-armhf-armhf-xl-arndale pass
test-amd64-amd64-xl-credit2 pass
test-arm64-arm64-xl-credit2 blocked
test-armhf-armhf-xl-credit2 pass
test-armhf-armhf-xl-cubietruck pass
test-amd64-amd64-examine pass
test-arm64-arm64-examine blocked
test-armhf-armhf-examine pass
test-amd64-i386-examine pass
test-amd64-i386-freebsd10-i386 pass
test-amd64-i386-rumprun-i386 pass
test-amd64-amd64-xl-qemut-win10-i386 fail
test-amd64-i386-xl-qemut-win10-i386 fail
test-amd64-amd64-xl-qemuu-win10-i386 fail
test-amd64-i386-xl-qemuu-win10-i386 fail
test-amd64-amd64-qemuu-nested-intel pass
test-amd64-amd64-xl-pvhv2-intel fail
test-amd64-i386-qemut-rhel6hvm-intel pass
test-amd64-i386-qemuu-rhel6hvm-intel pass
test-amd64-amd64-libvirt pass
test-armhf-armhf-libvirt pass
test-amd64-i386-libvirt pass
test-amd64-amd64-livepatch pass
test-amd64-i386-livepatch pass
test-amd64-amd64-migrupgrade pass
test-amd64-i386-migrupgrade pass
test-amd64-amd64-xl-multivcpu pass
test-armhf-armhf-xl-multivcpu pass
test-amd64-amd64-pair pass
test-amd64-i386-pair pass
test-amd64-amd64-libvirt-pair pass
test-amd64-i386-libvirt-pair pass
test-amd64-amd64-amd64-pvgrub pass
test-amd64-amd64-i386-pvgrub pass
test-amd64-amd64-pygrub pass
test-amd64-amd64-xl-qcow2 pass
test-armhf-armhf-libvirt-raw pass
test-amd64-i386-xl-raw pass
test-amd64-amd64-xl-rtds pass
test-armhf-armhf-xl-rtds pass
test-amd64-amd64-libvirt-vhd pass
test-armhf-armhf-xl-vhd pass
------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images
Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs
Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master
Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary
Not pushing.
------------------------------------------------------------
commit bd61fe94bee0556bc2f64999a4a8315b93f90f21
Author: Andrew Cooper <andrew.cooper3@citrix.com>
Date: Fri Sep 1 12:15:39 2017 +0100
x86/mm: Always set _PAGE_ACCESSED on L4e updates
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
(qemu changes not included)
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [xen-unstable test] 118078: regressions - FAIL
2018-01-16 8:43 [xen-unstable test] 118078: regressions - FAIL osstest service owner
@ 2018-01-16 8:57 ` Jan Beulich
2018-01-16 9:26 ` Paul Durrant
0 siblings, 1 reply; 7+ messages in thread
From: Jan Beulich @ 2018-01-16 8:57 UTC (permalink / raw)
To: Paul Durrant; +Cc: xen-devel, osstest-admin
>>> On 16.01.18 at 09:43, <osstest-admin@xenproject.org> wrote:
> flight 118078 xen-unstable real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/118078/
>
> Regressions :-(
>
> Tests which did not succeed and are blocking,
> including tests which could not be run:
> build-arm64-pvops 6 kernel-build fail REGR. vs. 118003
> test-amd64-amd64-xl-qemut-win7-amd64 10 windows-install fail REGR. vs. 118003
Paul,
is this last one something you could look into?
(XEN) d4: VIRIDIAN GUEST_OS_ID: vendor: 1 os: 4 major: 6 minor: 1 sp: 0 build: 1db0
(XEN) d4: VIRIDIAN HYPERCALL: enabled: 1 pfn: 3ffff
(XEN) d4v0: VIRIDIAN VP_ASSIST_PAGE: enabled: 1 pfn: 3fffe
(XEN) domain_crash called from viridian.c:452
(XEN) Domain 4 (vcpu#0) crashed on cpu#1:
(XEN) ----[ Xen-4.11-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 1
(XEN) RIP: 0010:[<fffff8000265d479>]
(XEN) RFLAGS: 0000000000000286 CONTEXT: hvm guest (d4v0)
(XEN) rax: 0000000000000000 rbx: fffff800027f7e80 rcx: 0000000000000001
(XEN) rdx: 0000000000000000 rsi: fffffa800129d040 rdi: fffff80002805c40
(XEN) rbp: 0000000000000080 rsp: fffff880009b0d80 r8: 0000000000000000
(XEN) r9: fffff800027f7e80 r10: fffffa800129d040 r11: fffff800027f7e90
(XEN) r12: fffff800008129a0 r13: fffff800028b9be0 r14: fffffa8001239b30
(XEN) r15: fffff80000b96080 cr0: 0000000080050031 cr4: 00000000000006b8
(XEN) cr3: 0000000000187000 cr2: 0000000000000000
(XEN) fsb: 0000000000000000 gsb: fffff800027f7d00 gss: fffff800027f7d00
(XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
I.e. the domain_crash() in viridian_start_apic_assist().
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [xen-unstable test] 118078: regressions - FAIL
2018-01-16 8:57 ` Jan Beulich
@ 2018-01-16 9:26 ` Paul Durrant
2018-01-16 17:30 ` Paul Durrant
0 siblings, 1 reply; 7+ messages in thread
From: Paul Durrant @ 2018-01-16 9:26 UTC (permalink / raw)
To: 'Jan Beulich'; +Cc: xen-devel, osstest-admin@xenproject.org
> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: 16 January 2018 08:58
> To: Paul Durrant <Paul.Durrant@citrix.com>
> Cc: xen-devel <xen-devel@lists.xenproject.org>; osstest-
> admin@xenproject.org
> Subject: Re: [Xen-devel] [xen-unstable test] 118078: regressions - FAIL
>
> >>> On 16.01.18 at 09:43, <osstest-admin@xenproject.org> wrote:
> > flight 118078 xen-unstable real [real]
> > http://logs.test-lab.xenproject.org/osstest/logs/118078/
> >
> > Regressions :-(
> >
> > Tests which did not succeed and are blocking,
> > including tests which could not be run:
> > build-arm64-pvops 6 kernel-build fail REGR. vs. 118003
> > test-amd64-amd64-xl-qemut-win7-amd64 10 windows-install fail REGR. vs.
> 118003
>
> Paul,
>
> is this last one something you could look into?
>
> (XEN) d4: VIRIDIAN GUEST_OS_ID: vendor: 1 os: 4 major: 6 minor: 1 sp: 0
> build: 1db0
> (XEN) d4: VIRIDIAN HYPERCALL: enabled: 1 pfn: 3ffff
> (XEN) d4v0: VIRIDIAN VP_ASSIST_PAGE: enabled: 1 pfn: 3fffe
> (XEN) domain_crash called from viridian.c:452
> (XEN) Domain 4 (vcpu#0) crashed on cpu#1:
> (XEN) ----[ Xen-4.11-unstable x86_64 debug=y Not tainted ]----
> (XEN) CPU: 1
> (XEN) RIP: 0010:[<fffff8000265d479>]
> (XEN) RFLAGS: 0000000000000286 CONTEXT: hvm guest (d4v0)
> (XEN) rax: 0000000000000000 rbx: fffff800027f7e80 rcx: 0000000000000001
> (XEN) rdx: 0000000000000000 rsi: fffffa800129d040 rdi: fffff80002805c40
> (XEN) rbp: 0000000000000080 rsp: fffff880009b0d80 r8: 0000000000000000
> (XEN) r9: fffff800027f7e80 r10: fffffa800129d040 r11: fffff800027f7e90
> (XEN) r12: fffff800008129a0 r13: fffff800028b9be0 r14: fffffa8001239b30
> (XEN) r15: fffff80000b96080 cr0: 0000000080050031 cr4: 00000000000006b8
> (XEN) cr3: 0000000000187000 cr2: 0000000000000000
> (XEN) fsb: 0000000000000000 gsb: fffff800027f7d00 gss: fffff800027f7d00
> (XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
>
> I.e. the domain_crash() in viridian_start_apic_assist().
>
Yes, I'll have a look at that.
Paul
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [xen-unstable test] 118078: regressions - FAIL
2018-01-16 9:26 ` Paul Durrant
@ 2018-01-16 17:30 ` Paul Durrant
2018-01-17 8:51 ` Jan Beulich
0 siblings, 1 reply; 7+ messages in thread
From: Paul Durrant @ 2018-01-16 17:30 UTC (permalink / raw)
To: Paul Durrant, 'Jan Beulich'
Cc: xen-devel, osstest-admin@xenproject.org
> -----Original Message-----
> From: Xen-devel [mailto:xen-devel-bounces@lists.xenproject.org] On Behalf
> Of Paul Durrant
> Sent: 16 January 2018 09:27
> To: 'Jan Beulich' <JBeulich@suse.com>
> Cc: xen-devel <xen-devel@lists.xenproject.org>; osstest-
> admin@xenproject.org
> Subject: Re: [Xen-devel] [xen-unstable test] 118078: regressions - FAIL
>
> > -----Original Message-----
> > From: Jan Beulich [mailto:JBeulich@suse.com]
> > Sent: 16 January 2018 08:58
> > To: Paul Durrant <Paul.Durrant@citrix.com>
> > Cc: xen-devel <xen-devel@lists.xenproject.org>; osstest-
> > admin@xenproject.org
> > Subject: Re: [Xen-devel] [xen-unstable test] 118078: regressions - FAIL
> >
> > >>> On 16.01.18 at 09:43, <osstest-admin@xenproject.org> wrote:
> > > flight 118078 xen-unstable real [real]
> > > http://logs.test-lab.xenproject.org/osstest/logs/118078/
> > >
> > > Regressions :-(
> > >
> > > Tests which did not succeed and are blocking,
> > > including tests which could not be run:
> > > build-arm64-pvops 6 kernel-build fail REGR. vs. 118003
> > > test-amd64-amd64-xl-qemut-win7-amd64 10 windows-install fail REGR.
> vs.
> > 118003
> >
> > Paul,
> >
> > is this last one something you could look into?
> >
> > (XEN) d4: VIRIDIAN GUEST_OS_ID: vendor: 1 os: 4 major: 6 minor: 1 sp: 0
> > build: 1db0
> > (XEN) d4: VIRIDIAN HYPERCALL: enabled: 1 pfn: 3ffff
> > (XEN) d4v0: VIRIDIAN VP_ASSIST_PAGE: enabled: 1 pfn: 3fffe
> > (XEN) domain_crash called from viridian.c:452
> > (XEN) Domain 4 (vcpu#0) crashed on cpu#1:
> > (XEN) ----[ Xen-4.11-unstable x86_64 debug=y Not tainted ]----
> > (XEN) CPU: 1
> > (XEN) RIP: 0010:[<fffff8000265d479>]
> > (XEN) RFLAGS: 0000000000000286 CONTEXT: hvm guest (d4v0)
> > (XEN) rax: 0000000000000000 rbx: fffff800027f7e80 rcx: 0000000000000001
> > (XEN) rdx: 0000000000000000 rsi: fffffa800129d040 rdi: fffff80002805c40
> > (XEN) rbp: 0000000000000080 rsp: fffff880009b0d80 r8:
> 0000000000000000
> > (XEN) r9: fffff800027f7e80 r10: fffffa800129d040 r11: fffff800027f7e90
> > (XEN) r12: fffff800008129a0 r13: fffff800028b9be0 r14: fffffa8001239b30
> > (XEN) r15: fffff80000b96080 cr0: 0000000080050031 cr4:
> 00000000000006b8
> > (XEN) cr3: 0000000000187000 cr2: 0000000000000000
> > (XEN) fsb: 0000000000000000 gsb: fffff800027f7d00 gss: fffff800027f7d00
> > (XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
> >
> > I.e. the domain_crash() in viridian_start_apic_assist().
> >
>
> Yes, I'll have a look at that.
No real clue about this as yet. It is odd that the guest has only set up one of the APIC assist pages and yet has taken an interrupt...
Jan 16 01:46:05.691223 (XEN) Dumping guest's current state at key_handler...
Jan 16 01:46:05.691265 (XEN) Size of VMCB = 4096, paddr = 000000020f7f7000, vaddr = ffff83020f7f7000
Jan 16 01:46:05.699269 (XEN) cr_intercepts = 0xfef3fef3 dr_intercepts = 0xffffffff exception_intercepts = 0x60082
Jan 16 01:46:05.707128 (XEN) general1_intercepts = 0xbdc4000f general2_intercepts = 0x2e7f
Jan 16 01:46:05.715222 (XEN) iopm_base_pa = 0xdfd71000 msrpm_base_pa = 0x20f7f4000 tsc_offset = 0xfffffc36684278c9
Jan 16 01:46:05.723116 (XEN) tlb_control = 0 vintr = 0x1020001 interrupt_shadow = 0
Jan 16 01:46:05.723153 (XEN) eventinj 000000008000002f, valid? 1, ec? 0, type 0, vector 0x2f
Jan 16 01:46:05.731141 (XEN) exitcode = 0x64 exitintinfo = 0
Jan 16 01:46:05.739123 (XEN) exitinfo1 = 0 exitinfo2 = 0
Jan 16 01:46:05.739157 (XEN) np_enable = 0x1 guest_asid = 0x4b49
Jan 16 01:46:05.739187 (XEN) virtual vmload/vmsave = 0, virt_ext = 0
I'd expect it to have interrupts disabled at this point. Seemingly doesn't repro on Intel h/w (although I was testing with Win7 SP1 rather than RTM) so I'll try to find some AMD h/w and try again.
Paul
> Paul
>
> > Jan
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xenproject.org
> https://lists.xenproject.org/mailman/listinfo/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [xen-unstable test] 118078: regressions - FAIL
2018-01-16 17:30 ` Paul Durrant
@ 2018-01-17 8:51 ` Jan Beulich
2018-01-17 9:37 ` Paul Durrant
0 siblings, 1 reply; 7+ messages in thread
From: Jan Beulich @ 2018-01-17 8:51 UTC (permalink / raw)
To: Paul Durrant; +Cc: xen-devel, osstest-admin@xenproject.org
>>> On 16.01.18 at 18:30, <Paul.Durrant@citrix.com> wrote:
>> From: Xen-devel [mailto:xen-devel-bounces@lists.xenproject.org] On Behalf
>> Of Paul Durrant
>> Sent: 16 January 2018 09:27
>> > From: Jan Beulich [mailto:JBeulich@suse.com]
>> > Sent: 16 January 2018 08:58
>> > >>> On 16.01.18 at 09:43, <osstest-admin@xenproject.org> wrote:
>> > > flight 118078 xen-unstable real [real]
>> > > http://logs.test-lab.xenproject.org/osstest/logs/118078/
>> > >
>> > > Regressions :-(
>> > >
>> > > Tests which did not succeed and are blocking,
>> > > including tests which could not be run:
>> > > build-arm64-pvops 6 kernel-build fail REGR. vs. 118003
>> > > test-amd64-amd64-xl-qemut-win7-amd64 10 windows-install fail REGR.
>> vs.
>> > 118003
>> >
>> > Paul,
>> >
>> > is this last one something you could look into?
>> >
>> > (XEN) d4: VIRIDIAN GUEST_OS_ID: vendor: 1 os: 4 major: 6 minor: 1 sp: 0
>> > build: 1db0
>> > (XEN) d4: VIRIDIAN HYPERCALL: enabled: 1 pfn: 3ffff
>> > (XEN) d4v0: VIRIDIAN VP_ASSIST_PAGE: enabled: 1 pfn: 3fffe
>> > (XEN) domain_crash called from viridian.c:452
>> > (XEN) Domain 4 (vcpu#0) crashed on cpu#1:
>> > (XEN) ----[ Xen-4.11-unstable x86_64 debug=y Not tainted ]----
>> > (XEN) CPU: 1
>> > (XEN) RIP: 0010:[<fffff8000265d479>]
>> > (XEN) RFLAGS: 0000000000000286 CONTEXT: hvm guest (d4v0)
>> > (XEN) rax: 0000000000000000 rbx: fffff800027f7e80 rcx: 0000000000000001
>> > (XEN) rdx: 0000000000000000 rsi: fffffa800129d040 rdi: fffff80002805c40
>> > (XEN) rbp: 0000000000000080 rsp: fffff880009b0d80 r8:
>> 0000000000000000
>> > (XEN) r9: fffff800027f7e80 r10: fffffa800129d040 r11: fffff800027f7e90
>> > (XEN) r12: fffff800008129a0 r13: fffff800028b9be0 r14: fffffa8001239b30
>> > (XEN) r15: fffff80000b96080 cr0: 0000000080050031 cr4:
>> 00000000000006b8
>> > (XEN) cr3: 0000000000187000 cr2: 0000000000000000
>> > (XEN) fsb: 0000000000000000 gsb: fffff800027f7d00 gss: fffff800027f7d00
>> > (XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
>> >
>> > I.e. the domain_crash() in viridian_start_apic_assist().
>> >
>>
>> Yes, I'll have a look at that.
>
> No real clue about this as yet. It is odd that the guest has only set up one
> of the APIC assist pages and yet has taken an interrupt...
>
> Jan 16 01:46:05.691223 (XEN) Dumping guest's current state at key_handler...
> Jan 16 01:46:05.691265 (XEN) Size of VMCB = 4096, paddr = 000000020f7f7000,
> vaddr = ffff83020f7f7000
> Jan 16 01:46:05.699269 (XEN) cr_intercepts = 0xfef3fef3 dr_intercepts =
> 0xffffffff exception_intercepts = 0x60082
> Jan 16 01:46:05.707128 (XEN) general1_intercepts = 0xbdc4000f
> general2_intercepts = 0x2e7f
> Jan 16 01:46:05.715222 (XEN) iopm_base_pa = 0xdfd71000 msrpm_base_pa =
> 0x20f7f4000 tsc_offset = 0xfffffc36684278c9
> Jan 16 01:46:05.723116 (XEN) tlb_control = 0 vintr = 0x1020001
> interrupt_shadow = 0
> Jan 16 01:46:05.723153 (XEN) eventinj 000000008000002f, valid? 1, ec? 0,
> type 0, vector 0x2f
> Jan 16 01:46:05.731141 (XEN) exitcode = 0x64 exitintinfo = 0
> Jan 16 01:46:05.739123 (XEN) exitinfo1 = 0 exitinfo2 = 0
> Jan 16 01:46:05.739157 (XEN) np_enable = 0x1 guest_asid = 0x4b49
> Jan 16 01:46:05.739187 (XEN) virtual vmload/vmsave = 0, virt_ext = 0
>
> I'd expect it to have interrupts disabled at this point. Seemingly doesn't
> repro on Intel h/w (although I was testing with Win7 SP1 rather than RTM) so
> I'll try to find some AMD h/w and try again.
Well, it looks to be a random problem in the first place, or else we
would have known about the issue much earlier I think. I.e. I'm
not sure I see what you take the "AMD only" from here.
As to interrupt state - isn't it quite normal for an OS to bring up
the BSP first, enable interrupts on it, and then bring up APs?
That's how we do it in Xen.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [xen-unstable test] 118078: regressions - FAIL
2018-01-17 8:51 ` Jan Beulich
@ 2018-01-17 9:37 ` Paul Durrant
2018-01-17 10:17 ` Paul Durrant
0 siblings, 1 reply; 7+ messages in thread
From: Paul Durrant @ 2018-01-17 9:37 UTC (permalink / raw)
To: 'Jan Beulich'; +Cc: xen-devel, osstest-admin@xenproject.org
> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: 17 January 2018 08:52
> To: Paul Durrant <Paul.Durrant@citrix.com>
> Cc: xen-devel <xen-devel@lists.xenproject.org>; osstest-
> admin@xenproject.org
> Subject: RE: [Xen-devel] [xen-unstable test] 118078: regressions - FAIL
>
> >>> On 16.01.18 at 18:30, <Paul.Durrant@citrix.com> wrote:
> >> From: Xen-devel [mailto:xen-devel-bounces@lists.xenproject.org] On
> Behalf
> >> Of Paul Durrant
> >> Sent: 16 January 2018 09:27
> >> > From: Jan Beulich [mailto:JBeulich@suse.com]
> >> > Sent: 16 January 2018 08:58
> >> > >>> On 16.01.18 at 09:43, <osstest-admin@xenproject.org> wrote:
> >> > > flight 118078 xen-unstable real [real]
> >> > > http://logs.test-lab.xenproject.org/osstest/logs/118078/
> >> > >
> >> > > Regressions :-(
> >> > >
> >> > > Tests which did not succeed and are blocking,
> >> > > including tests which could not be run:
> >> > > build-arm64-pvops 6 kernel-build fail REGR. vs. 118003
> >> > > test-amd64-amd64-xl-qemut-win7-amd64 10 windows-install fail
> REGR.
> >> vs.
> >> > 118003
> >> >
> >> > Paul,
> >> >
> >> > is this last one something you could look into?
> >> >
> >> > (XEN) d4: VIRIDIAN GUEST_OS_ID: vendor: 1 os: 4 major: 6 minor: 1 sp: 0
> >> > build: 1db0
> >> > (XEN) d4: VIRIDIAN HYPERCALL: enabled: 1 pfn: 3ffff
> >> > (XEN) d4v0: VIRIDIAN VP_ASSIST_PAGE: enabled: 1 pfn: 3fffe
> >> > (XEN) domain_crash called from viridian.c:452
> >> > (XEN) Domain 4 (vcpu#0) crashed on cpu#1:
> >> > (XEN) ----[ Xen-4.11-unstable x86_64 debug=y Not tainted ]----
> >> > (XEN) CPU: 1
> >> > (XEN) RIP: 0010:[<fffff8000265d479>]
> >> > (XEN) RFLAGS: 0000000000000286 CONTEXT: hvm guest (d4v0)
> >> > (XEN) rax: 0000000000000000 rbx: fffff800027f7e80 rcx:
> 0000000000000001
> >> > (XEN) rdx: 0000000000000000 rsi: fffffa800129d040 rdi:
> fffff80002805c40
> >> > (XEN) rbp: 0000000000000080 rsp: fffff880009b0d80 r8:
> >> 0000000000000000
> >> > (XEN) r9: fffff800027f7e80 r10: fffffa800129d040 r11: fffff800027f7e90
> >> > (XEN) r12: fffff800008129a0 r13: fffff800028b9be0 r14:
> fffffa8001239b30
> >> > (XEN) r15: fffff80000b96080 cr0: 0000000080050031 cr4:
> >> 00000000000006b8
> >> > (XEN) cr3: 0000000000187000 cr2: 0000000000000000
> >> > (XEN) fsb: 0000000000000000 gsb: fffff800027f7d00 gss:
> fffff800027f7d00
> >> > (XEN) ds: 002b es: 002b fs: 0053 gs: 002b ss: 0018 cs: 0010
> >> >
> >> > I.e. the domain_crash() in viridian_start_apic_assist().
> >> >
> >>
> >> Yes, I'll have a look at that.
> >
> > No real clue about this as yet. It is odd that the guest has only set up one
> > of the APIC assist pages and yet has taken an interrupt...
> >
> > Jan 16 01:46:05.691223 (XEN) Dumping guest's current state at
> key_handler...
> > Jan 16 01:46:05.691265 (XEN) Size of VMCB = 4096, paddr =
> 000000020f7f7000,
> > vaddr = ffff83020f7f7000
> > Jan 16 01:46:05.699269 (XEN) cr_intercepts = 0xfef3fef3 dr_intercepts =
> > 0xffffffff exception_intercepts = 0x60082
> > Jan 16 01:46:05.707128 (XEN) general1_intercepts = 0xbdc4000f
> > general2_intercepts = 0x2e7f
> > Jan 16 01:46:05.715222 (XEN) iopm_base_pa = 0xdfd71000 msrpm_base_pa
> =
> > 0x20f7f4000 tsc_offset = 0xfffffc36684278c9
> > Jan 16 01:46:05.723116 (XEN) tlb_control = 0 vintr = 0x1020001
> > interrupt_shadow = 0
> > Jan 16 01:46:05.723153 (XEN) eventinj 000000008000002f, valid? 1, ec? 0,
> > type 0, vector 0x2f
> > Jan 16 01:46:05.731141 (XEN) exitcode = 0x64 exitintinfo = 0
> > Jan 16 01:46:05.739123 (XEN) exitinfo1 = 0 exitinfo2 = 0
> > Jan 16 01:46:05.739157 (XEN) np_enable = 0x1 guest_asid = 0x4b49
> > Jan 16 01:46:05.739187 (XEN) virtual vmload/vmsave = 0, virt_ext = 0
> >
> > I'd expect it to have interrupts disabled at this point. Seemingly doesn't
> > repro on Intel h/w (although I was testing with Win7 SP1 rather than RTM)
> so
> > I'll try to find some AMD h/w and try again.
>
> Well, it looks to be a random problem in the first place, or else we
> would have known about the issue much earlier I think. I.e. I'm
> not sure I see what you take the "AMD only" from here.
>
Well, I don't see any repro on Intel and the VMCB (rather than VMCS) dump identifies this case as using AMD h/w. I agree that it is random though, so it could indeed be purely coincidental.
> As to interrupt state - isn't it quite normal for an OS to bring up
> the BSP first, enable interrupts on it, and then bring up APs?
> That's how we do it in Xen.
What I meant was that I'd expect the guest to have interrupts disabled whilst poking the MSR to enable APIC assist on that CPU, since enabling APIC assist is clearly going to modify the way in which interrupts are handled. If that's not the case though then I guess that is probably the cause of the issue; I never really considered protecting interrupt handling against APIC assist being enabled on the same CPU.
Paul
>
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [xen-unstable test] 118078: regressions - FAIL
2018-01-17 9:37 ` Paul Durrant
@ 2018-01-17 10:17 ` Paul Durrant
0 siblings, 0 replies; 7+ messages in thread
From: Paul Durrant @ 2018-01-17 10:17 UTC (permalink / raw)
To: Paul Durrant, 'Jan Beulich'
Cc: xen-devel, osstest-admin@xenproject.org
> -----Original Message-----
[snip]
> What I meant was that I'd expect the guest to have interrupts disabled whilst
> poking the MSR to enable APIC assist on that CPU, since enabling APIC assist
> is clearly going to modify the way in which interrupts are handled. If that's
> not the case though then I guess that is probably the cause of the issue; I
> never really considered protecting interrupt handling against APIC assist
> being enabled on the same CPU.
>
I think I've spotted it...
If the guest is indeed setting up APIC assist but not actually using it then I think we get into this situation...
- On return to guest vlapic_has_pending_irq() finds a bit set in the IRR, then vlapic_ack_pending_irq() calls viridian_start_apic_assist() which latches the vector, sets the bit in the ISR and clears it from the IRR.
- The guest then processes the interrupt but EOIs it normally, therefore clearing the bit in the ISR.
- On next return to guest vlapic_has_pending_irq() calls viridian_complete_apic_assist(), which discovers the assist bit still set in the page and therefore leaves the latched vector in place, but finds another bit set in the IRR.
- vlapic_ack_pending_irq() is then called, the ISR is clear, so another call is made to viridian_start_apic_assist() and this then calls domain_crash() because it finds the previously latched vector.
I think the correct solution to this is to call viridian_abort_apic_assist() in vlapic_EOI_set() when the vector is cleared from the ISR.
Paul
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2018-01-17 10:18 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-01-16 8:43 [xen-unstable test] 118078: regressions - FAIL osstest service owner
2018-01-16 8:57 ` Jan Beulich
2018-01-16 9:26 ` Paul Durrant
2018-01-16 17:30 ` Paul Durrant
2018-01-17 8:51 ` Jan Beulich
2018-01-17 9:37 ` Paul Durrant
2018-01-17 10:17 ` Paul Durrant
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).