[xen-unstable-smoke test] 162597: regressions - FAIL

[xen-unstable-smoke test] 162597: regressions - FAIL

[osstest service owner]

flight 162597 xen-unstable-smoke real [real]
flight 162602 xen-unstable-smoke real-retest [real]
http://logs.test-lab.xenproject.org/osstest/logs/162597/
http://logs.test-lab.xenproject.org/osstest/logs/162602/

Regressions :-(

Tests which did not succeed and are blocking,
including tests which could not be run:
 test-armhf-armhf-xl         18 guest-start/debian.repeat fail REGR. vs. 162574

Tests which did not succeed, but are not blocking:
 test-amd64-amd64-libvirt     15 migrate-support-check        fail   never pass
 test-arm64-arm64-xl-xsm      15 migrate-support-check        fail   never pass
 test-arm64-arm64-xl-xsm      16 saverestore-support-check    fail   never pass
 test-armhf-armhf-xl          15 migrate-support-check        fail   never pass
 test-armhf-armhf-xl          16 saverestore-support-check    fail   never pass

version targeted for testing:
 xen                  dfcffb128be46a3e413eaa941744536fe53c94b6
baseline version:
 xen                  3e09045991cde360432bc7437103f8f8a6699359

Last test of basis   162574  2021-06-09 14:00:34 Z    0 days
Testing same since   162584  2021-06-10 00:00:27 Z    0 days    3 attempts

------------------------------------------------------------
People who touched revisions under test:
  Edgar E. Iglesias <edgar.iglesias@xilinx.com>
  Stefano Stabellini <sstabellini@kernel.org>
  Stefano Stabellini <stefano.stabellini@xilinx.com>

jobs:
 build-arm64-xsm                                              pass    
 build-amd64                                                  pass    
 build-armhf                                                  pass    
 build-amd64-libvirt                                          pass    
 test-armhf-armhf-xl                                          fail    
 test-arm64-arm64-xl-xsm                                      pass    
 test-amd64-amd64-xl-qemuu-debianhvm-amd64                    pass    
 test-amd64-amd64-libvirt                                     pass    


------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images

Logs, config files, etc. are available at
    http://logs.test-lab.xenproject.org/osstest/logs

Explanation of these reports, and of osstest in general, is at
    http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
    http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master

Test harness code can be found at
    http://xenbits.xen.org/gitweb?p=osstest.git;a=summary


Not pushing.

------------------------------------------------------------
commit dfcffb128be46a3e413eaa941744536fe53c94b6
Author: Stefano Stabellini <sstabellini@kernel.org>
Date:   Wed Jun 9 10:37:59 2021 -0700

    xen/arm32: SPSR_hyp/SPSR
    
    SPSR_hyp is not meant to be accessed from Hyp mode (EL2); accesses
    trigger UNPREDICTABLE behaviour. Xen should read/write SPSR instead.
    See: ARM DDI 0487D.b page G8-5993.
    
    This fixes booting Xen/arm32 on QEMU.
    
    Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
    Reviewed-by: Julien Grall <jgrall@amazon.com>
    Reviewed-by: Edgar E. Iglesias <edgar.iglesias@xilinx.com>
    Tested-by: Edgar E. Iglesias <edgar.iglesias@xilinx.com>
(qemu changes not included)

Re: [xen-unstable-smoke test] 162597: regressions - FAIL

[Jan Beulich]

On 10.06.2021 12:50, osstest service owner wrote:
> flight 162597 xen-unstable-smoke real [real]
> flight 162602 xen-unstable-smoke real-retest [real]
> http://logs.test-lab.xenproject.org/osstest/logs/162597/
> http://logs.test-lab.xenproject.org/osstest/logs/162602/
> 
> Regressions :-(
> 
> Tests which did not succeed and are blocking,
> including tests which could not be run:
>  test-armhf-armhf-xl         18 guest-start/debian.repeat fail REGR. vs. 162574

This now being the 3rd failure in a row, I guess there's a fair chance
of there actually being something wrong with ...

> commit dfcffb128be46a3e413eaa941744536fe53c94b6
> Author: Stefano Stabellini <sstabellini@kernel.org>
> Date:   Wed Jun 9 10:37:59 2021 -0700
> 
>     xen/arm32: SPSR_hyp/SPSR
>     
>     SPSR_hyp is not meant to be accessed from Hyp mode (EL2); accesses
>     trigger UNPREDICTABLE behaviour. Xen should read/write SPSR instead.
>     See: ARM DDI 0487D.b page G8-5993.
>     
>     This fixes booting Xen/arm32 on QEMU.
>     
>     Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
>     Reviewed-by: Julien Grall <jgrall@amazon.com>
>     Reviewed-by: Edgar E. Iglesias <edgar.iglesias@xilinx.com>
>     Tested-by: Edgar E. Iglesias <edgar.iglesias@xilinx.com>

... this. My Arm-untrained eye couldn't spot anything in the logs.

Jan

Re: [xen-unstable-smoke test] 162597: regressions - FAIL

[Bertrand Marquis]

Hi Jan,

> On 10 Jun 2021, at 12:32, Jan Beulich <jbeulich@suse.com> wrote:
> 
> On 10.06.2021 12:50, osstest service owner wrote:
>> flight 162597 xen-unstable-smoke real [real]
>> flight 162602 xen-unstable-smoke real-retest [real]
>> http://logs.test-lab.xenproject.org/osstest/logs/162597/
>> http://logs.test-lab.xenproject.org/osstest/logs/162602/
>> 
>> Regressions :-(
>> 
>> Tests which did not succeed and are blocking,
>> including tests which could not be run:
>> test-armhf-armhf-xl         18 guest-start/debian.repeat fail REGR. vs. 162574
> 
> This now being the 3rd failure in a row, I guess there's a fair chance
> of there actually being something wrong with ...
> 
>> commit dfcffb128be46a3e413eaa941744536fe53c94b6
>> Author: Stefano Stabellini <sstabellini@kernel.org>
>> Date:   Wed Jun 9 10:37:59 2021 -0700
>> 
>>    xen/arm32: SPSR_hyp/SPSR
>> 
>>    SPSR_hyp is not meant to be accessed from Hyp mode (EL2); accesses
>>    trigger UNPREDICTABLE behaviour. Xen should read/write SPSR instead.
>>    See: ARM DDI 0487D.b page G8-5993.
>> 
>>    This fixes booting Xen/arm32 on QEMU.
>> 
>>    Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
>>    Reviewed-by: Julien Grall <jgrall@amazon.com>
>>    Reviewed-by: Edgar E. Iglesias <edgar.iglesias@xilinx.com>
>>    Tested-by: Edgar E. Iglesias <edgar.iglesias@xilinx.com>
> 
> ... this. My Arm-untrained eye couldn't spot anything in the logs.

I am not sure to read the log correctly so do I see it right that dom0 started and it failed then to start a guest ?

Regards
Bertrand

> 
> Jan
> 
> 

Re: [xen-unstable-smoke test] 162597: regressions - FAIL

[Stefano Stabellini]

On Thu, 10 Jun 2021, Bertrand Marquis wrote:
> Hi Jan,
> 
> > On 10 Jun 2021, at 12:32, Jan Beulich <jbeulich@suse.com> wrote:
> > 
> > On 10.06.2021 12:50, osstest service owner wrote:
> >> flight 162597 xen-unstable-smoke real [real]
> >> flight 162602 xen-unstable-smoke real-retest [real]
> >> http://logs.test-lab.xenproject.org/osstest/logs/162597/
> >> http://logs.test-lab.xenproject.org/osstest/logs/162602/
> >> 
> >> Regressions :-(
> >> 
> >> Tests which did not succeed and are blocking,
> >> including tests which could not be run:
> >> test-armhf-armhf-xl         18 guest-start/debian.repeat fail REGR. vs. 162574
> > 
> > This now being the 3rd failure in a row, I guess there's a fair chance
> > of there actually being something wrong with ...
> > 
> >> commit dfcffb128be46a3e413eaa941744536fe53c94b6
> >> Author: Stefano Stabellini <sstabellini@kernel.org>
> >> Date:   Wed Jun 9 10:37:59 2021 -0700
> >> 
> >>    xen/arm32: SPSR_hyp/SPSR
> >> 
> >>    SPSR_hyp is not meant to be accessed from Hyp mode (EL2); accesses
> >>    trigger UNPREDICTABLE behaviour. Xen should read/write SPSR instead.
> >>    See: ARM DDI 0487D.b page G8-5993.
> >> 
> >>    This fixes booting Xen/arm32 on QEMU.
> >> 
> >>    Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
> >>    Reviewed-by: Julien Grall <jgrall@amazon.com>
> >>    Reviewed-by: Edgar E. Iglesias <edgar.iglesias@xilinx.com>
> >>    Tested-by: Edgar E. Iglesias <edgar.iglesias@xilinx.com>
> > 
> > ... this. My Arm-untrained eye couldn't spot anything in the logs.
> 
> I am not sure to read the log correctly so do I see it right that dom0 started and it failed then to start a guest ?

Thanks Jan for bringing it to my attention. 

I am not an expert in reading OSSTest logs. From the following:

http://logs.test-lab.xenproject.org/osstest/logs/162597/test-armhf-armhf-xl/info.html

I understand that Xen booted and a DomU was started. However,
"migrate-support-check" and "saverestore-support-check" failed. Is that
correct?

If so, it would be really strange for SPSR_hyp/SPSR to cause the problem
because I would expect Xen to hang at boot before Dom0 is started
instead.


I don't have any ARMv7 hardware to try to repro this issue, and ARMv7 is
most certainly required (ARMv8/aarch32 won't repro.)

Could someone more at ease with OSSTest than me arrange for a run with
this commit reverted to verify that it is the issue?



In any case, I tried to figure it out. I guessed it could be a compiler
error. I followed the white rabbit down the ARM ARM hole. I disassebled
the Xen binary [1] from the failed job. "msr SPSR, r11" is 0x0026a38c.

The encoding should be at B9.3.12 of the ARMv7-A DDI 0406C and F5.1.121
of ARMv8 DDI 0487D.b. Unfortunately it doesn't seem to match either one
of them and I don't understand why.


The "mrs r11, SPSR" is generated as 0x00262ecc. That should be described
at F5.1.117 for ARMv8 and B9.3.9 for ARMv7. Also doesn't seem to match.


I guess I am looking at the wrong encoding but I am not exactly sure why.



[1] http://logs.test-lab.xenproject.org/osstest/logs/162597/build-armhf/build/

Re: [xen-unstable-smoke test] 162597: regressions - FAIL

[Jan Beulich]

On 11.06.2021 03:49, Stefano Stabellini wrote:
> On Thu, 10 Jun 2021, Bertrand Marquis wrote:
>>> On 10 Jun 2021, at 12:32, Jan Beulich <jbeulich@suse.com> wrote:
>>> On 10.06.2021 12:50, osstest service owner wrote:
>>>> flight 162597 xen-unstable-smoke real [real]
>>>> flight 162602 xen-unstable-smoke real-retest [real]
>>>> http://logs.test-lab.xenproject.org/osstest/logs/162597/
>>>> http://logs.test-lab.xenproject.org/osstest/logs/162602/
>>>>
>>>> Regressions :-(
>>>>
>>>> Tests which did not succeed and are blocking,
>>>> including tests which could not be run:
>>>> test-armhf-armhf-xl         18 guest-start/debian.repeat fail REGR. vs. 162574
>>>
>>> This now being the 3rd failure in a row, I guess there's a fair chance
>>> of there actually being something wrong with ...
>>>
>>>> commit dfcffb128be46a3e413eaa941744536fe53c94b6
>>>> Author: Stefano Stabellini <sstabellini@kernel.org>
>>>> Date:   Wed Jun 9 10:37:59 2021 -0700
>>>>
>>>>    xen/arm32: SPSR_hyp/SPSR
>>>>
>>>>    SPSR_hyp is not meant to be accessed from Hyp mode (EL2); accesses
>>>>    trigger UNPREDICTABLE behaviour. Xen should read/write SPSR instead.
>>>>    See: ARM DDI 0487D.b page G8-5993.
>>>>
>>>>    This fixes booting Xen/arm32 on QEMU.
>>>>
>>>>    Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
>>>>    Reviewed-by: Julien Grall <jgrall@amazon.com>
>>>>    Reviewed-by: Edgar E. Iglesias <edgar.iglesias@xilinx.com>
>>>>    Tested-by: Edgar E. Iglesias <edgar.iglesias@xilinx.com>
>>>
>>> ... this. My Arm-untrained eye couldn't spot anything in the logs.
>>
>> I am not sure to read the log correctly so do I see it right that dom0 started and it failed then to start a guest ?

Well, in this particular flight it succeeded to create Dom1 (for
guest-start) and then it managed to also create Dom2, but failed to
get the expected "sign of life". It varies at which of the repeated
attempts the failure occurs (in one of the flights it also occurred
right at guest-start), but failure chances are high enough such that
so far in all of the flights things didn't complete successfully.
And with this high a failure rate, it accidentally succeeding and
thus leading to a push would probably do us more bad than good.

> Thanks Jan for bringing it to my attention. 
> 
> I am not an expert in reading OSSTest logs. From the following:
> 
> http://logs.test-lab.xenproject.org/osstest/logs/162597/test-armhf-armhf-xl/info.html
> 
> I understand that Xen booted and a DomU was started. However,
> "migrate-support-check" and "saverestore-support-check" failed. Is that
> correct?

Yes, but these two steps aren't the problem - afaict they always fail,
and hence wouldn't prevent a push.

It's guest-start/debian.repeat which is the problem in this flight.

> If so, it would be really strange for SPSR_hyp/SPSR to cause the problem
> because I would expect Xen to hang at boot before Dom0 is started
> instead.
> 
> 
> I don't have any ARMv7 hardware to try to repro this issue, and ARMv7 is
> most certainly required (ARMv8/aarch32 won't repro.)
> 
> Could someone more at ease with OSSTest than me arrange for a run with
> this commit reverted to verify that it is the issue?
> 
> 
> 
> In any case, I tried to figure it out. I guessed it could be a compiler
> error. I followed the white rabbit down the ARM ARM hole. I disassebled
> the Xen binary [1] from the failed job. "msr SPSR, r11" is 0x0026a38c.
> 
> The encoding should be at B9.3.12 of the ARMv7-A DDI 0406C and F5.1.121
> of ARMv8 DDI 0487D.b. Unfortunately it doesn't seem to match either one
> of them and I don't understand why.
> 
> 
> The "mrs r11, SPSR" is generated as 0x00262ecc. That should be described
> at F5.1.117 for ARMv8 and B9.3.9 for ARMv7. Also doesn't seem to match.

Indeed I was wondering whether perhaps the tool chain has an issue here.
Otoh I'd expect a tool chain issue to yield consistent failures rather
than ones with just a fair probability. Unless, of course, unspecified
behavior is hit, and the hardware indeed behaves randomly in this case.

Jan

Re: [xen-unstable-smoke test] 162597: regressions - FAIL

[Jan Beulich]

On 11.06.2021 03:49, Stefano Stabellini wrote:
> In any case, I tried to figure it out. I guessed it could be a compiler
> error. I followed the white rabbit down the ARM ARM hole. I disassebled
> the Xen binary [1] from the failed job. "msr SPSR, r11" is 0x0026a38c.
> 
> The encoding should be at B9.3.12 of the ARMv7-A DDI 0406C and F5.1.121
> of ARMv8 DDI 0487D.b. Unfortunately it doesn't seem to match either one
> of them and I don't understand why.
> 
> 
> The "mrs r11, SPSR" is generated as 0x00262ecc. That should be described
> at F5.1.117 for ARMv8 and B9.3.9 for ARMv7. Also doesn't seem to match.

According to my looking at the disassembly, the two numbers you've
quoted are the addresses, not insn encodings. Using my own disassembler
(i.e. there's room for that one being wrong), I do get

        E169F00B        msr     spsr_cf, r11
        E14FB000        mrs     r11, spsr

the former of which doesn't look like an exact equivalent of the input
instruction. I guess it really is "msr spsr_cxsf, r11" which is meant?
In gas sources I find this:

      /* Unadorned APSR is equivalent to APSR_nzcvq/CPSR_f (for writes).  This
	 is deprecated, but allow it anyway.  */
      if (is_apsr && lhs)
	{
	  psr_field |= PSR_f;
	  as_tsktsk (_("writing to APSR without specifying a bitmask is "
		       "deprecated"));
	}
      else if (!m_profile)
	/* These bits are never right for M-profile devices: don't set them
	   (only code paths which read/write APSR reach here).  */
	psr_field |= (PSR_c | PSR_f);

There's clearly a comment missing to talk about the "unadorned" SPSR
case, but the effect is exactly what is observed: Rather than
defaulting to the setting of all 4 bits, only two of them get set
when plain "SPSR" is used. I've not been able to spot a place where
the Arm ARM specifies this, but given its size I'm not surprised at
all. I'd like to note though that the MSR description doesn't even
allow for plain "SPSR" (unlike MRS); only SPSR_<...> is described
there.

Based on this analysis I guess I can make a patch despite not being
able to test it, as I'm pretty certain you really want to restore
all of PSR; not just the low half ...

Jan