Re: PowerPC KVM-PR issue

[Christian Zigotzky <chzigotzky@xenosoft.de>]

On 15 June 2020 at 01:39 pm, Christian Zigotzky wrote:
> On 14 June 2020 at 04:52 pm, Christian Zigotzky wrote:
>> On 14 June 2020 at 02:53 pm, Nicholas Piggin wrote:
>>> Excerpts from Christian Zigotzky's message of June 12, 2020 11:01 pm:
>>>> On 11 June 2020 at 04:47 pm, Christian Zigotzky wrote:
>>>>> On 10 June 2020 at 01:23 pm, Christian Zigotzky wrote:
>>>>>> On 10 June 2020 at 11:06 am, Christian Zigotzky wrote:
>>>>>>> On 10 June 2020 at 00:18 am, Christian Zigotzky wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> KVM-PR doesn't work anymore on my Nemo board [1]. I figured out
>>>>>>>> that the Git kernels and the kernel 5.7 are affected.
>>>>>>>>
>>>>>>>> Error message: Fienix kernel: kvmppc_exit_pr_progint: emulation at
>>>>>>>> 700 failed (00000000)
>>>>>>>>
>>>>>>>> I can boot virtual QEMU PowerPC machines with KVM-PR with the
>>>>>>>> kernel 5.6 without any problems on my Nemo board.
>>>>>>>>
>>>>>>>> I tested it with QEMU 2.5.0 and QEMU 5.0.0 today.
>>>>>>>>
>>>>>>>> Could you please check KVM-PR on your PowerPC machine?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Christian
>>>>>>>>
>>>>>>>> [1] https://en.wikipedia.org/wiki/AmigaOne_X1000
>>>>>>> I figured out that the PowerPC updates 5.7-1 [1] are responsible 
>>>>>>> for
>>>>>>> the KVM-PR issue. Please test KVM-PR on your PowerPC machines and
>>>>>>> check the PowerPC updates 5.7-1 [1].
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> [1]
>>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d38c07afc356ddebaa3ed8ecb3f553340e05c969 
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> I tested the latest Git kernel with Mac-on-Linux/KVM-PR today.
>>>>>> Unfortunately I can't use KVM-PR with MoL anymore because of this
>>>>>> issue (see screenshots [1]). Please check the PowerPC updates 5.7-1.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> [1]
>>>>>> -
>>>>>> https://i.pinimg.com/originals/0c/b3/64/0cb364a40241fa2b7f297d4272bbb8b7.png 
>>>>>>
>>>>>> -
>>>>>> https://i.pinimg.com/originals/9a/61/d1/9a61d170b1c9f514f7a78a3014ffd18f.png 
>>>>>>
>>>>>>
>>>>> Hi All,
>>>>>
>>>>> I bisected today because of the KVM-PR issue.
>>>>>
>>>>> Result:
>>>>>
>>>>> 9600f261acaaabd476d7833cec2dd20f2919f1a0 is the first bad commit
>>>>> commit 9600f261acaaabd476d7833cec2dd20f2919f1a0
>>>>> Author: Nicholas Piggin <npiggin@gmail.com>
>>>>> Date:   Wed Feb 26 03:35:21 2020 +1000
>>>>>
>>>>>      powerpc/64s/exception: Move KVM test to common code
>>>>>
>>>>>      This allows more code to be moved out of unrelocated regions. 
>>>>> The
>>>>>      system call KVMTEST is changed to be open-coded and remain in 
>>>>> the
>>>>>      tramp area to avoid having to move it to entry_64.S. The custom
>>>>> nature
>>>>>      of the system call entry code means the hcall case can be 
>>>>> made more
>>>>>      streamlined than regular interrupt handlers.
>>>>>
>>>>>      mpe: Incorporate fix from Nick:
>>>>>
>>>>>      Moving KVM test to the common entry code missed the case of 
>>>>> HMI and
>>>>>      MCE, which do not do __GEN_COMMON_ENTRY (because they don't 
>>>>> want to
>>>>>      switch to virt mode).
>>>>>
>>>>>      This means a MCE or HMI exception that is taken while KVM is
>>>>> running a
>>>>>      guest context will not be switched out of that context, and 
>>>>> KVM won't
>>>>>      be notified. Found by running sigfuz in guest with patched 
>>>>> host on
>>>>>      POWER9 DD2.3, which causes some TM related HMI interrupts 
>>>>> (which are
>>>>>      expected and supposed to be handled by KVM).
>>>>>
>>>>>      This fix adds a __GEN_REALMODE_COMMON_ENTRY for those 
>>>>> handlers to add
>>>>>      the KVM test. This makes them look a little more like other 
>>>>> handlers
>>>>>      that all use __GEN_COMMON_ENTRY.
>>>>>
>>>>>      Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
>>>>>      Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
>>>>>      Link:
>>>>> https://lore.kernel.org/r/20200225173541.1549955-13-npiggin@gmail.com
>>>>>
>>>>> :040000 040000 ec21cec22d165f8696d69532734cb2985d532cb0
>>>>> 87dd49a9cd7202ec79350e8ca26cea01f1dbd93d M    arch
>>>>>
>>>>> -----
>>>>>
>>>>> The following commit is the problem: powerpc/64s/exception: Move KVM
>>>>> test to common code [1]
>>>>>
>>>>> These changes were included in the PowerPC updates 5.7-1. [2]
>>>>>
>>>>> Another test:
>>>>>
>>>>> git checkout d38c07afc356ddebaa3ed8ecb3f553340e05c969 (PowerPC 
>>>>> updates
>>>>> 5.7-1 [2] ) -> KVM-PR doesn't work.
>>>>>
>>>>> After that: git revert d38c07afc356ddebaa3ed8ecb3f553340e05c969 -m 1
>>>>> -> KVM-PR works.
>>>>>
>>>>> Could you please check the first bad commit? [1]
>>>>>
>>>>> Thanks,
>>>>> Christian
>>>>>
>>>>>
>>>>> [1]
>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9600f261acaaabd476d7833cec2dd20f2919f1a0 
>>>>>
>>>>> [2]
>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d38c07afc356ddebaa3ed8ecb3f553340e05c969 
>>>>>
>>>> Hi All,
>>>>
>>>> I tried to revert the __GEN_REALMODE_COMMON_ENTRY fix for the 
>>>> latest Git
>>>> kernel and for the stable kernel 5.7.2 but without any success. There
>>>> was lot of restructuring work during the kernel 5.7 development 
>>>> time in
>>>> the PowerPC area so it isn't possible reactivate the old code. That
>>>> means we have lost the whole KVM-PR support. I also reported this 
>>>> issue
>>>> to Alexander Graf two days ago. He wrote: "Howdy :). It looks pretty
>>>> broken. Have you ever made a bisect to see where the problem comes 
>>>> from?"
>>>>
>>>> Please check the KVM-PR code.
>>> Does this patch fix it for you?
>>>
>>> The CTR register reload in the KVM interrupt path used the wrong save
>>> area for SLB (and NMI) interrupts.
>>>
>>> Fixes: 9600f261acaaa ("powerpc/64s/exception: Move KVM test to 
>>> common code")
>>> Reported-by: Christian Zigotzky <chzigotzky@xenosoft.de>
>>> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
>>> ---
>>>   arch/powerpc/kernel/exceptions-64s.S | 4 ++--
>>>   1 file changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/arch/powerpc/kernel/exceptions-64s.S 
>>> b/arch/powerpc/kernel/exceptions-64s.S
>>> index e70ebb5c318c..fa080694e581 100644
>>> --- a/arch/powerpc/kernel/exceptions-64s.S
>>> +++ b/arch/powerpc/kernel/exceptions-64s.S
>>> @@ -270,7 +270,7 @@ BEGIN_FTR_SECTION
>>>   END_FTR_SECTION_IFSET(CPU_FTR_CFAR)
>>>       .endif
>>>   -    ld    r10,PACA_EXGEN+EX_CTR(r13)
>>> +    ld    r10,IAREA+EX_CTR(r13)
>>>       mtctr    r10
>>>   BEGIN_FTR_SECTION
>>>       ld    r10,IAREA+EX_PPR(r13)
>>> @@ -298,7 +298,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
>>>         .if IKVM_SKIP
>>>   89:    mtocrf    0x80,r9
>>> -    ld    r10,PACA_EXGEN+EX_CTR(r13)
>>> +    ld    r10,IAREA+EX_CTR(r13)
>>>       mtctr    r10
>>>       ld    r9,IAREA+EX_R9(r13)
>>>       ld    r10,IAREA+EX_R10(r13)
>> Many thanks for the fix! I will test it with the RC1 tomorrow.
>>
>> -- Christian
>
> It works! :-) Thanks a lot! Screenshot: 
> https://i.pinimg.com/originals/5d/5f/e5/5d5fe584db474dc88bcc641450b2a7e0.png
>
> -- Christian

Just for info: I successfully tested KVM-PR with the stable kernel 5.7.6 
and with the RC2 of kernel 5.8 today. Thanks a lot for fixing the issue.

-- Christian