Re: PowerPC KVM-PR issue

[Christian Zigotzky <chzigotzky@xenosoft.de>]

On 14 June 2020 at 04:52 pm, Christian Zigotzky wrote:
> On 14 June 2020 at 02:53 pm, Nicholas Piggin wrote:
>> Excerpts from Christian Zigotzky's message of June 12, 2020 11:01 pm:
>>> On 11 June 2020 at 04:47 pm, Christian Zigotzky wrote:
>>>> On 10 June 2020 at 01:23 pm, Christian Zigotzky wrote:
>>>>> On 10 June 2020 at 11:06 am, Christian Zigotzky wrote:
>>>>>> On 10 June 2020 at 00:18 am, Christian Zigotzky wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> KVM-PR doesn't work anymore on my Nemo board [1]. I figured out
>>>>>>> that the Git kernels and the kernel 5.7 are affected.
>>>>>>>
>>>>>>> Error message: Fienix kernel: kvmppc_exit_pr_progint: emulation at
>>>>>>> 700 failed (00000000)
>>>>>>>
>>>>>>> I can boot virtual QEMU PowerPC machines with KVM-PR with the
>>>>>>> kernel 5.6 without any problems on my Nemo board.
>>>>>>>
>>>>>>> I tested it with QEMU 2.5.0 and QEMU 5.0.0 today.
>>>>>>>
>>>>>>> Could you please check KVM-PR on your PowerPC machine?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Christian
>>>>>>>
>>>>>>> [1] https://en.wikipedia.org/wiki/AmigaOne_X1000
>>>>>> I figured out that the PowerPC updates 5.7-1 [1] are responsible for
>>>>>> the KVM-PR issue. Please test KVM-PR on your PowerPC machines and
>>>>>> check the PowerPC updates 5.7-1 [1].
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> [1]
>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d38c07afc356ddebaa3ed8ecb3f553340e05c969 
>>>>>>
>>>>>>
>>>>>>
>>>>> I tested the latest Git kernel with Mac-on-Linux/KVM-PR today.
>>>>> Unfortunately I can't use KVM-PR with MoL anymore because of this
>>>>> issue (see screenshots [1]). Please check the PowerPC updates 5.7-1.
>>>>>
>>>>> Thanks
>>>>>
>>>>> [1]
>>>>> -
>>>>> https://i.pinimg.com/originals/0c/b3/64/0cb364a40241fa2b7f297d4272bbb8b7.png 
>>>>>
>>>>> -
>>>>> https://i.pinimg.com/originals/9a/61/d1/9a61d170b1c9f514f7a78a3014ffd18f.png 
>>>>>
>>>>>
>>>> Hi All,
>>>>
>>>> I bisected today because of the KVM-PR issue.
>>>>
>>>> Result:
>>>>
>>>> 9600f261acaaabd476d7833cec2dd20f2919f1a0 is the first bad commit
>>>> commit 9600f261acaaabd476d7833cec2dd20f2919f1a0
>>>> Author: Nicholas Piggin <npiggin@gmail.com>
>>>> Date:   Wed Feb 26 03:35:21 2020 +1000
>>>>
>>>>      powerpc/64s/exception: Move KVM test to common code
>>>>
>>>>      This allows more code to be moved out of unrelocated regions. The
>>>>      system call KVMTEST is changed to be open-coded and remain in the
>>>>      tramp area to avoid having to move it to entry_64.S. The custom
>>>> nature
>>>>      of the system call entry code means the hcall case can be made 
>>>> more
>>>>      streamlined than regular interrupt handlers.
>>>>
>>>>      mpe: Incorporate fix from Nick:
>>>>
>>>>      Moving KVM test to the common entry code missed the case of 
>>>> HMI and
>>>>      MCE, which do not do __GEN_COMMON_ENTRY (because they don't 
>>>> want to
>>>>      switch to virt mode).
>>>>
>>>>      This means a MCE or HMI exception that is taken while KVM is
>>>> running a
>>>>      guest context will not be switched out of that context, and 
>>>> KVM won't
>>>>      be notified. Found by running sigfuz in guest with patched 
>>>> host on
>>>>      POWER9 DD2.3, which causes some TM related HMI interrupts 
>>>> (which are
>>>>      expected and supposed to be handled by KVM).
>>>>
>>>>      This fix adds a __GEN_REALMODE_COMMON_ENTRY for those handlers 
>>>> to add
>>>>      the KVM test. This makes them look a little more like other 
>>>> handlers
>>>>      that all use __GEN_COMMON_ENTRY.
>>>>
>>>>      Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
>>>>      Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
>>>>      Link:
>>>> https://lore.kernel.org/r/20200225173541.1549955-13-npiggin@gmail.com
>>>>
>>>> :040000 040000 ec21cec22d165f8696d69532734cb2985d532cb0
>>>> 87dd49a9cd7202ec79350e8ca26cea01f1dbd93d M    arch
>>>>
>>>> -----
>>>>
>>>> The following commit is the problem: powerpc/64s/exception: Move KVM
>>>> test to common code [1]
>>>>
>>>> These changes were included in the PowerPC updates 5.7-1. [2]
>>>>
>>>> Another test:
>>>>
>>>> git checkout d38c07afc356ddebaa3ed8ecb3f553340e05c969 (PowerPC updates
>>>> 5.7-1 [2] ) -> KVM-PR doesn't work.
>>>>
>>>> After that: git revert d38c07afc356ddebaa3ed8ecb3f553340e05c969 -m 1
>>>> -> KVM-PR works.
>>>>
>>>> Could you please check the first bad commit? [1]
>>>>
>>>> Thanks,
>>>> Christian
>>>>
>>>>
>>>> [1]
>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9600f261acaaabd476d7833cec2dd20f2919f1a0 
>>>>
>>>> [2]
>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d38c07afc356ddebaa3ed8ecb3f553340e05c969 
>>>>
>>> Hi All,
>>>
>>> I tried to revert the __GEN_REALMODE_COMMON_ENTRY fix for the latest 
>>> Git
>>> kernel and for the stable kernel 5.7.2 but without any success. There
>>> was lot of restructuring work during the kernel 5.7 development time in
>>> the PowerPC area so it isn't possible reactivate the old code. That
>>> means we have lost the whole KVM-PR support. I also reported this issue
>>> to Alexander Graf two days ago. He wrote: "Howdy :). It looks pretty
>>> broken. Have you ever made a bisect to see where the problem comes 
>>> from?"
>>>
>>> Please check the KVM-PR code.
>> Does this patch fix it for you?
>>
>> The CTR register reload in the KVM interrupt path used the wrong save
>> area for SLB (and NMI) interrupts.
>>
>> Fixes: 9600f261acaaa ("powerpc/64s/exception: Move KVM test to common 
>> code")
>> Reported-by: Christian Zigotzky <chzigotzky@xenosoft.de>
>> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
>> ---
>>   arch/powerpc/kernel/exceptions-64s.S | 4 ++--
>>   1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/arch/powerpc/kernel/exceptions-64s.S 
>> b/arch/powerpc/kernel/exceptions-64s.S
>> index e70ebb5c318c..fa080694e581 100644
>> --- a/arch/powerpc/kernel/exceptions-64s.S
>> +++ b/arch/powerpc/kernel/exceptions-64s.S
>> @@ -270,7 +270,7 @@ BEGIN_FTR_SECTION
>>   END_FTR_SECTION_IFSET(CPU_FTR_CFAR)
>>       .endif
>>   -    ld    r10,PACA_EXGEN+EX_CTR(r13)
>> +    ld    r10,IAREA+EX_CTR(r13)
>>       mtctr    r10
>>   BEGIN_FTR_SECTION
>>       ld    r10,IAREA+EX_PPR(r13)
>> @@ -298,7 +298,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
>>         .if IKVM_SKIP
>>   89:    mtocrf    0x80,r9
>> -    ld    r10,PACA_EXGEN+EX_CTR(r13)
>> +    ld    r10,IAREA+EX_CTR(r13)
>>       mtctr    r10
>>       ld    r9,IAREA+EX_R9(r13)
>>       ld    r10,IAREA+EX_R10(r13)
> Many thanks for the fix! I will test it with the RC1 tomorrow.
>
> -- Christian

It works! :-) Thanks a lot! Screenshot: 
https://i.pinimg.com/originals/5d/5f/e5/5d5fe584db474dc88bcc641450b2a7e0.png

-- Christian