5.1.0-rc1 regression: reset fails with kvm and -cpu host

5.1.0-rc1 regression: reset fails with kvm and -cpu host

[Jan Kiszka]

Hi all,

this locks up the guest:

- qemu-system-x86_64 -enable-kvm -cpu host
- trigger hard reset

Host kernel: 5.7.7.
Host CPU: i7-8850H

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: 5.1.0-rc1 regression: reset fails with kvm and -cpu host

[Eduardo Habkost]

Hi Jan,

What was the last version where it worked for you?  Does using
"-cpu host,-vmx" help?


On Wed, Jul 22, 2020 at 11:15:43AM +0200, Jan Kiszka wrote:
> Hi all,
> 
> this locks up the guest:
> 
> - qemu-system-x86_64 -enable-kvm -cpu host
> - trigger hard reset
> 
> Host kernel: 5.7.7.
> Host CPU: i7-8850H
> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux
> 

-- 
Eduardo

Re: 5.1.0-rc1 regression: reset fails with kvm and -cpu host

[Jan Kiszka]

On 22.07.20 19:35, Eduardo Habkost wrote:
> Hi Jan,
> 
> What was the last version where it worked for you?  Does using
> "-cpu host,-vmx" help?

Yeah, -vmx does indeed help.

I didn't have the time to bisect yet. Just check my reflog, picked 
eb6490f544, and that works.

HTH,
Jan

> 
> 
> On Wed, Jul 22, 2020 at 11:15:43AM +0200, Jan Kiszka wrote:
>> Hi all,
>>
>> this locks up the guest:
>>
>> - qemu-system-x86_64 -enable-kvm -cpu host
>> - trigger hard reset
>>
>> Host kernel: 5.7.7.
>> Host CPU: i7-8850H
>>
>> Jan
>>
>> -- 
>> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
>> Corporate Competence Center Embedded Linux
>>
> 

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: 5.1.0-rc1 regression: reset fails with kvm and -cpu host

[Eduardo Habkost]

On Wed, Jul 22, 2020 at 08:05:01PM +0200, Jan Kiszka wrote:
> On 22.07.20 19:35, Eduardo Habkost wrote:
> > Hi Jan,
> > 
> > What was the last version where it worked for you?  Does using
> > "-cpu host,-vmx" help?
> 
> Yeah, -vmx does indeed help.
> 
> I didn't have the time to bisect yet. Just check my reflog, picked
> eb6490f544, and that works.

Thanks!

I could reproduce it locally[1], I will bisect it.

The good news is that "-cpu host,+vmx" still works, on commit
eb6490f544.

[1] Linux 5.6.19-300.fc32.x86_64, Intel Core i7-8665U CPU.

-- 
Eduardo

Re: 5.1.0-rc1 regression: reset fails with kvm and -cpu host

[Eduardo Habkost]

On Wed, Jul 22, 2020 at 04:47:32PM -0400, Eduardo Habkost wrote:
> On Wed, Jul 22, 2020 at 08:05:01PM +0200, Jan Kiszka wrote:
> > On 22.07.20 19:35, Eduardo Habkost wrote:
> > > Hi Jan,
> > > 
> > > What was the last version where it worked for you?  Does using
> > > "-cpu host,-vmx" help?
> > 
> > Yeah, -vmx does indeed help.
> > 
> > I didn't have the time to bisect yet. Just check my reflog, picked
> > eb6490f544, and that works.
> 
> Thanks!
> 
> I could reproduce it locally[1], I will bisect it.
> 
> The good news is that "-cpu host,+vmx" still works, on commit
> eb6490f544.
> 
> [1] Linux 5.6.19-300.fc32.x86_64, Intel Core i7-8665U CPU.

Bisected to:

commit b16c0e20c74218f2d69710cedad11da7dd4d2190
Author: Paolo Bonzini <pbonzini@redhat.com>
Date:   Wed May 20 10:49:22 2020 -0400

    KVM: add support for AMD nested live migration

    Support for nested guest live migration is part of Linux 5.8, add the
    corresponding code to QEMU.  The migration format consists of a few
    flags, is an opaque 4k blob.

    The blob is in VMCB format (the control area represents the L1 VMCB
    control fields, the save area represents the pre-vmentry state; KVM does
    not use the host save area since the AMD manual allows that) but QEMU
    does not really care about that.  However, the flags need to be
    copied to hflags/hflags2 and back.

    In addition, support for retrieving and setting the AMD nested virtualization
    states allows the L1 guest to be reset while running a nested guest, but
    a small bug in CPU reset needs to be fixed for that to work.

    Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>


-- 
Eduardo

Re: 5.1.0-rc1 regression: reset fails with kvm and -cpu host

[Dr. David Alan Gilbert]

* Eduardo Habkost (ehabkost@redhat.com) wrote:
> On Wed, Jul 22, 2020 at 04:47:32PM -0400, Eduardo Habkost wrote:
> > On Wed, Jul 22, 2020 at 08:05:01PM +0200, Jan Kiszka wrote:
> > > On 22.07.20 19:35, Eduardo Habkost wrote:
> > > > Hi Jan,
> > > > 
> > > > What was the last version where it worked for you?  Does using
> > > > "-cpu host,-vmx" help?
> > > 
> > > Yeah, -vmx does indeed help.
> > > 
> > > I didn't have the time to bisect yet. Just check my reflog, picked
> > > eb6490f544, and that works.
> > 
> > Thanks!
> > 
> > I could reproduce it locally[1], I will bisect it.
> > 
> > The good news is that "-cpu host,+vmx" still works, on commit
> > eb6490f544.
> > 
> > [1] Linux 5.6.19-300.fc32.x86_64, Intel Core i7-8665U CPU.
> 
> Bisected to:
> 
> commit b16c0e20c74218f2d69710cedad11da7dd4d2190
> Author: Paolo Bonzini <pbonzini@redhat.com>
> Date:   Wed May 20 10:49:22 2020 -0400
> 
>     KVM: add support for AMD nested live migration
> 
>     Support for nested guest live migration is part of Linux 5.8, add the
>     corresponding code to QEMU.  The migration format consists of a few
>     flags, is an opaque 4k blob.
> 
>     The blob is in VMCB format (the control area represents the L1 VMCB
>     control fields, the save area represents the pre-vmentry state; KVM does
>     not use the host save area since the AMD manual allows that) but QEMU
>     does not really care about that.  However, the flags need to be
>     copied to hflags/hflags2 and back.
> 
>     In addition, support for retrieving and setting the AMD nested virtualization
>     states allows the L1 guest to be reset while running a nested guest, but
>     a small bug in CPU reset needs to be fixed for that to work.
> 
>     Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Guesswork led me to try reverting the chunk in kvm_put_nested_state;
without it the reset seems to work; I can't explain that code though.

Dave

> 
> -- 
> Eduardo
> 
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: 5.1.0-rc1 regression: reset fails with kvm and -cpu host

[Philippe Mathieu-Daudé]

+Vitaly

On 7/23/20 10:40 AM, Dr. David Alan Gilbert wrote:
> * Eduardo Habkost (ehabkost@redhat.com) wrote:
>> On Wed, Jul 22, 2020 at 04:47:32PM -0400, Eduardo Habkost wrote:
>>> On Wed, Jul 22, 2020 at 08:05:01PM +0200, Jan Kiszka wrote:
>>>> On 22.07.20 19:35, Eduardo Habkost wrote:
>>>>> Hi Jan,
>>>>>
>>>>> What was the last version where it worked for you?  Does using
>>>>> "-cpu host,-vmx" help?
>>>>
>>>> Yeah, -vmx does indeed help.
>>>>
>>>> I didn't have the time to bisect yet. Just check my reflog, picked
>>>> eb6490f544, and that works.
>>>
>>> Thanks!
>>>
>>> I could reproduce it locally[1], I will bisect it.
>>>
>>> The good news is that "-cpu host,+vmx" still works, on commit
>>> eb6490f544.
>>>
>>> [1] Linux 5.6.19-300.fc32.x86_64, Intel Core i7-8665U CPU.
>>
>> Bisected to:
>>
>> commit b16c0e20c74218f2d69710cedad11da7dd4d2190
>> Author: Paolo Bonzini <pbonzini@redhat.com>
>> Date:   Wed May 20 10:49:22 2020 -0400
>>
>>     KVM: add support for AMD nested live migration
>>
>>     Support for nested guest live migration is part of Linux 5.8, add the
>>     corresponding code to QEMU.  The migration format consists of a few
>>     flags, is an opaque 4k blob.
>>
>>     The blob is in VMCB format (the control area represents the L1 VMCB
>>     control fields, the save area represents the pre-vmentry state; KVM does
>>     not use the host save area since the AMD manual allows that) but QEMU
>>     does not really care about that.  However, the flags need to be
>>     copied to hflags/hflags2 and back.
>>
>>     In addition, support for retrieving and setting the AMD nested virtualization
>>     states allows the L1 guest to be reset while running a nested guest, but
>>     a small bug in CPU reset needs to be fixed for that to work.
>>
>>     Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> 
> Guesswork led me to try reverting the chunk in kvm_put_nested_state;
> without it the reset seems to work; I can't explain that code though.
> 
> Dave
> 
>>
>> -- 
>> Eduardo
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 
> 

Re: 5.1.0-rc1 regression: reset fails with kvm and -cpu host

[Vitaly Kuznetsov]

Philippe Mathieu-Daudé <philmd@redhat.com> writes:

> +Vitaly
>
> On 7/23/20 10:40 AM, Dr. David Alan Gilbert wrote:
>> * Eduardo Habkost (ehabkost@redhat.com) wrote:
>>> On Wed, Jul 22, 2020 at 04:47:32PM -0400, Eduardo Habkost wrote:
>>>> On Wed, Jul 22, 2020 at 08:05:01PM +0200, Jan Kiszka wrote:
>>>>> On 22.07.20 19:35, Eduardo Habkost wrote:
>>>>>> Hi Jan,
>>>>>>
>>>>>> What was the last version where it worked for you?  Does using
>>>>>> "-cpu host,-vmx" help?
>>>>>
>>>>> Yeah, -vmx does indeed help.
>>>>>
>>>>> I didn't have the time to bisect yet. Just check my reflog, picked
>>>>> eb6490f544, and that works.
>>>>
>>>> Thanks!
>>>>
>>>> I could reproduce it locally[1], I will bisect it.
>>>>
>>>> The good news is that "-cpu host,+vmx" still works, on commit
>>>> eb6490f544.
>>>>
>>>> [1] Linux 5.6.19-300.fc32.x86_64, Intel Core i7-8665U CPU.
>>>
>>> Bisected to:
>>>
>>> commit b16c0e20c74218f2d69710cedad11da7dd4d2190
>>> Author: Paolo Bonzini <pbonzini@redhat.com>
>>> Date:   Wed May 20 10:49:22 2020 -0400
>>>
>>>     KVM: add support for AMD nested live migration
>>>
>>>     Support for nested guest live migration is part of Linux 5.8, add the
>>>     corresponding code to QEMU.  The migration format consists of a few
>>>     flags, is an opaque 4k blob.
>>>
>>>     The blob is in VMCB format (the control area represents the L1 VMCB
>>>     control fields, the save area represents the pre-vmentry state; KVM does
>>>     not use the host save area since the AMD manual allows that) but QEMU
>>>     does not really care about that.  However, the flags need to be
>>>     copied to hflags/hflags2 and back.
>>>
>>>     In addition, support for retrieving and setting the AMD nested virtualization
>>>     states allows the L1 guest to be reset while running a nested guest, but
>>>     a small bug in CPU reset needs to be fixed for that to work.
>>>
>>>     Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>> 
>> Guesswork led me to try reverting the chunk in kvm_put_nested_state;
>> without it the reset seems to work; I can't explain that code though.
>> 

(sorry, missed the beginning of the discussion)

So one does:

(qemu) system_reset 

on Intel wiht '-cpu host' and the result is:

(qemu) KVM: entry failed, hardware error 0x80000021

If you're running a guest on an Intel machine without unrestricted mode
support, the failure can be most likely due to the guest entering an invalid
state for Intel VT. For example, the guest maybe running in big real mode
which is not supported on less recent Intel processors.

EAX=00000064 EBX=91df5efe ECX=00000000 EDX=000003f8
ESI=00000000 EDI=91ee32c0 EBP=90643260 ESP=00013c68
EIP=906428e6 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??

I can take a look (if no one beats me to it).

-- 
Vitaly

Re: 5.1.0-rc1 regression: reset fails with kvm and -cpu host

[Dr. David Alan Gilbert]

* Vitaly Kuznetsov (vkuznets@redhat.com) wrote:
> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
> 
> > +Vitaly
> >
> > On 7/23/20 10:40 AM, Dr. David Alan Gilbert wrote:
> >> * Eduardo Habkost (ehabkost@redhat.com) wrote:
> >>> On Wed, Jul 22, 2020 at 04:47:32PM -0400, Eduardo Habkost wrote:
> >>>> On Wed, Jul 22, 2020 at 08:05:01PM +0200, Jan Kiszka wrote:
> >>>>> On 22.07.20 19:35, Eduardo Habkost wrote:
> >>>>>> Hi Jan,
> >>>>>>
> >>>>>> What was the last version where it worked for you?  Does using
> >>>>>> "-cpu host,-vmx" help?
> >>>>>
> >>>>> Yeah, -vmx does indeed help.
> >>>>>
> >>>>> I didn't have the time to bisect yet. Just check my reflog, picked
> >>>>> eb6490f544, and that works.
> >>>>
> >>>> Thanks!
> >>>>
> >>>> I could reproduce it locally[1], I will bisect it.
> >>>>
> >>>> The good news is that "-cpu host,+vmx" still works, on commit
> >>>> eb6490f544.
> >>>>
> >>>> [1] Linux 5.6.19-300.fc32.x86_64, Intel Core i7-8665U CPU.
> >>>
> >>> Bisected to:
> >>>
> >>> commit b16c0e20c74218f2d69710cedad11da7dd4d2190
> >>> Author: Paolo Bonzini <pbonzini@redhat.com>
> >>> Date:   Wed May 20 10:49:22 2020 -0400
> >>>
> >>>     KVM: add support for AMD nested live migration
> >>>
> >>>     Support for nested guest live migration is part of Linux 5.8, add the
> >>>     corresponding code to QEMU.  The migration format consists of a few
> >>>     flags, is an opaque 4k blob.
> >>>
> >>>     The blob is in VMCB format (the control area represents the L1 VMCB
> >>>     control fields, the save area represents the pre-vmentry state; KVM does
> >>>     not use the host save area since the AMD manual allows that) but QEMU
> >>>     does not really care about that.  However, the flags need to be
> >>>     copied to hflags/hflags2 and back.
> >>>
> >>>     In addition, support for retrieving and setting the AMD nested virtualization
> >>>     states allows the L1 guest to be reset while running a nested guest, but
> >>>     a small bug in CPU reset needs to be fixed for that to work.
> >>>
> >>>     Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> >> 
> >> Guesswork led me to try reverting the chunk in kvm_put_nested_state;
> >> without it the reset seems to work; I can't explain that code though.
> >> 
> 
> (sorry, missed the beginning of the discussion)
> 
> So one does:
> 
> (qemu) system_reset 
> 
> on Intel wiht '-cpu host' and the result is:
> 
> (qemu) KVM: entry failed, hardware error 0x80000021

Interesting; I hadn't seen that error - I just see a hard hung guest
rather than a reset one.

(i7-8650U laptop 5.7.9 fedora 32)

Dave

> If you're running a guest on an Intel machine without unrestricted mode
> support, the failure can be most likely due to the guest entering an invalid
> state for Intel VT. For example, the guest maybe running in big real mode
> which is not supported on less recent Intel processors.
> 
> EAX=00000064 EBX=91df5efe ECX=00000000 EDX=000003f8
> ESI=00000000 EDI=91ee32c0 EBP=90643260 ESP=00013c68
> EIP=906428e6 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =0000 00000000 0000ffff 00009300
> CS =f000 ffff0000 0000ffff 00009b00
> SS =0000 00000000 0000ffff 00009300
> DS =0000 00000000 0000ffff 00009300
> FS =0000 00000000 0000ffff 00009300
> GS =0000 00000000 0000ffff 00009300
> LDT=0000 00000000 0000ffff 00008200
> TR =0000 00000000 0000ffff 00008b00
> GDT=     00000000 0000ffff
> IDT=     00000000 0000ffff
> CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000000
> Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
> 
> I can take a look (if no one beats me to it).
> 
> -- 
> Vitaly
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: 5.1.0-rc1 regression: reset fails with kvm and -cpu host

[Jan Kiszka]

On 23.07.20 14:52, Dr. David Alan Gilbert wrote:
> * Vitaly Kuznetsov (vkuznets@redhat.com) wrote:
>> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
>>
>>> +Vitaly
>>>
>>> On 7/23/20 10:40 AM, Dr. David Alan Gilbert wrote:
>>>> * Eduardo Habkost (ehabkost@redhat.com) wrote:
>>>>> On Wed, Jul 22, 2020 at 04:47:32PM -0400, Eduardo Habkost wrote:
>>>>>> On Wed, Jul 22, 2020 at 08:05:01PM +0200, Jan Kiszka wrote:
>>>>>>> On 22.07.20 19:35, Eduardo Habkost wrote:
>>>>>>>> Hi Jan,
>>>>>>>>
>>>>>>>> What was the last version where it worked for you?  Does using
>>>>>>>> "-cpu host,-vmx" help?
>>>>>>>
>>>>>>> Yeah, -vmx does indeed help.
>>>>>>>
>>>>>>> I didn't have the time to bisect yet. Just check my reflog, picked
>>>>>>> eb6490f544, and that works.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> I could reproduce it locally[1], I will bisect it.
>>>>>>
>>>>>> The good news is that "-cpu host,+vmx" still works, on commit
>>>>>> eb6490f544.
>>>>>>
>>>>>> [1] Linux 5.6.19-300.fc32.x86_64, Intel Core i7-8665U CPU.
>>>>>
>>>>> Bisected to:
>>>>>
>>>>> commit b16c0e20c74218f2d69710cedad11da7dd4d2190
>>>>> Author: Paolo Bonzini <pbonzini@redhat.com>
>>>>> Date:   Wed May 20 10:49:22 2020 -0400
>>>>>
>>>>>      KVM: add support for AMD nested live migration
>>>>>
>>>>>      Support for nested guest live migration is part of Linux 5.8, add the
>>>>>      corresponding code to QEMU.  The migration format consists of a few
>>>>>      flags, is an opaque 4k blob.
>>>>>
>>>>>      The blob is in VMCB format (the control area represents the L1 VMCB
>>>>>      control fields, the save area represents the pre-vmentry state; KVM does
>>>>>      not use the host save area since the AMD manual allows that) but QEMU
>>>>>      does not really care about that.  However, the flags need to be
>>>>>      copied to hflags/hflags2 and back.
>>>>>
>>>>>      In addition, support for retrieving and setting the AMD nested virtualization
>>>>>      states allows the L1 guest to be reset while running a nested guest, but
>>>>>      a small bug in CPU reset needs to be fixed for that to work.
>>>>>
>>>>>      Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>>>>
>>>> Guesswork led me to try reverting the chunk in kvm_put_nested_state;
>>>> without it the reset seems to work; I can't explain that code though.
>>>>
>>
>> (sorry, missed the beginning of the discussion)
>>
>> So one does:
>>
>> (qemu) system_reset
>>
>> on Intel wiht '-cpu host' and the result is:
>>
>> (qemu) KVM: entry failed, hardware error 0x80000021
> 
> Interesting; I hadn't seen that error - I just see a hard hung guest
> rather than a reset one.

I've seen it once or twice, maybe that was also with a more complex 
command line. The point is that an invalid state is very likely loaded 
on reset. Not all invalid states cause KVM to complain, though. Some 
just lock up the guest.

Jan

> 
> (i7-8650U laptop 5.7.9 fedora 32)
> 
> Dave
> 
>> If you're running a guest on an Intel machine without unrestricted mode
>> support, the failure can be most likely due to the guest entering an invalid
>> state for Intel VT. For example, the guest maybe running in big real mode
>> which is not supported on less recent Intel processors.
>>
>> EAX=00000064 EBX=91df5efe ECX=00000000 EDX=000003f8
>> ESI=00000000 EDI=91ee32c0 EBP=90643260 ESP=00013c68
>> EIP=906428e6 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>> ES =0000 00000000 0000ffff 00009300
>> CS =f000 ffff0000 0000ffff 00009b00
>> SS =0000 00000000 0000ffff 00009300
>> DS =0000 00000000 0000ffff 00009300
>> FS =0000 00000000 0000ffff 00009300
>> GS =0000 00000000 0000ffff 00009300
>> LDT=0000 00000000 0000ffff 00008200
>> TR =0000 00000000 0000ffff 00008b00
>> GDT=     00000000 0000ffff
>> IDT=     00000000 0000ffff
>> CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
>> DR6=00000000ffff0ff0 DR7=0000000000000400
>> EFER=0000000000000000
>> Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
>>
>> I can take a look (if no one beats me to it).
>>
>> -- 
>> Vitaly
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: 5.1.0-rc1 regression: reset fails with kvm and -cpu host

[Vitaly Kuznetsov]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> On 23.07.20 14:52, Dr. David Alan Gilbert wrote:
>> * Vitaly Kuznetsov (vkuznets@redhat.com) wrote:
>>> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
>>>
>>>> +Vitaly
>>>>
>>>> On 7/23/20 10:40 AM, Dr. David Alan Gilbert wrote:
>>>>> * Eduardo Habkost (ehabkost@redhat.com) wrote:
>>>>>> On Wed, Jul 22, 2020 at 04:47:32PM -0400, Eduardo Habkost wrote:
>>>>>>> On Wed, Jul 22, 2020 at 08:05:01PM +0200, Jan Kiszka wrote:
>>>>>>>> On 22.07.20 19:35, Eduardo Habkost wrote:
>>>>>>>>> Hi Jan,
>>>>>>>>>
>>>>>>>>> What was the last version where it worked for you?  Does using
>>>>>>>>> "-cpu host,-vmx" help?
>>>>>>>>
>>>>>>>> Yeah, -vmx does indeed help.
>>>>>>>>
>>>>>>>> I didn't have the time to bisect yet. Just check my reflog, picked
>>>>>>>> eb6490f544, and that works.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> I could reproduce it locally[1], I will bisect it.
>>>>>>>
>>>>>>> The good news is that "-cpu host,+vmx" still works, on commit
>>>>>>> eb6490f544.
>>>>>>>
>>>>>>> [1] Linux 5.6.19-300.fc32.x86_64, Intel Core i7-8665U CPU.
>>>>>>
>>>>>> Bisected to:
>>>>>>
>>>>>> commit b16c0e20c74218f2d69710cedad11da7dd4d2190
>>>>>> Author: Paolo Bonzini <pbonzini@redhat.com>
>>>>>> Date:   Wed May 20 10:49:22 2020 -0400
>>>>>>
>>>>>>      KVM: add support for AMD nested live migration
>>>>>>
>>>>>>      Support for nested guest live migration is part of Linux 5.8, add the
>>>>>>      corresponding code to QEMU.  The migration format consists of a few
>>>>>>      flags, is an opaque 4k blob.
>>>>>>
>>>>>>      The blob is in VMCB format (the control area represents the L1 VMCB
>>>>>>      control fields, the save area represents the pre-vmentry state; KVM does
>>>>>>      not use the host save area since the AMD manual allows that) but QEMU
>>>>>>      does not really care about that.  However, the flags need to be
>>>>>>      copied to hflags/hflags2 and back.
>>>>>>
>>>>>>      In addition, support for retrieving and setting the AMD nested virtualization
>>>>>>      states allows the L1 guest to be reset while running a nested guest, but
>>>>>>      a small bug in CPU reset needs to be fixed for that to work.
>>>>>>
>>>>>>      Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>>>>>
>>>>> Guesswork led me to try reverting the chunk in kvm_put_nested_state;
>>>>> without it the reset seems to work; I can't explain that code though.
>>>>>
>>>
>>> (sorry, missed the beginning of the discussion)
>>>
>>> So one does:
>>>
>>> (qemu) system_reset
>>>
>>> on Intel wiht '-cpu host' and the result is:
>>>
>>> (qemu) KVM: entry failed, hardware error 0x80000021
>> 
>> Interesting; I hadn't seen that error - I just see a hard hung guest
>> rather than a reset one.
>
> I've seen it once or twice, maybe that was also with a more complex 
> command line. The point is that an invalid state is very likely loaded 
> on reset. Not all invalid states cause KVM to complain, though. Some 
> just lock up the guest.

This depends on whether the guest has performed VMXON or not I believe.

Anyways, I *think* the fix will be:

diff --git a/target/i386/kvm.c b/target/i386/kvm.c
index 2b6b744..75c2e68 100644
--- a/target/i386/kvm.c
+++ b/target/i386/kvm.c
@@ -3883,7 +3883,7 @@ static int kvm_put_nested_state(X86CPU *cpu)
     } else {
         env->nested_state->flags &= ~KVM_STATE_NESTED_GUEST_MODE;
     }
-    if (env->hflags2 & HF2_GIF_MASK) {
+    if (cpu_has_svm(env) && (env->hflags2 & HF2_GIF_MASK)) {
         env->nested_state->flags |= KVM_STATE_NESTED_GIF_SET;
     } else {
         env->nested_state->flags &= ~KVM_STATE_NESTED_GIF_SET;

As "KVM_STATE_NESTED_GIF_SET" is not relevant to nVMX, this works for me
but let me explore kernel side of this a bit more.

-- 
Vitaly

Re: 5.1.0-rc1 regression: reset fails with kvm and -cpu host

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 931 bytes --]

Yes, that seems correct.

Paolo


Il gio 23 lug 2020, 15:26 Vitaly Kuznetsov <vkuznets@redhat.com> ha scritto:

> This depends on whether the guest has performed VMXON or not I believe.
>
> Anyways, I *think* the fix will be:
>
> diff --git a/target/i386/kvm.c b/target/i386/kvm.c
> index 2b6b744..75c2e68 100644
> --- a/target/i386/kvm.c
> +++ b/target/i386/kvm.c
> @@ -3883,7 +3883,7 @@ static int kvm_put_nested_state(X86CPU *cpu)
>      } else {
>          env->nested_state->flags &= ~KVM_STATE_NESTED_GUEST_MODE;
>      }
> -    if (env->hflags2 & HF2_GIF_MASK) {
> +    if (cpu_has_svm(env) && (env->hflags2 & HF2_GIF_MASK)) {
>          env->nested_state->flags |= KVM_STATE_NESTED_GIF_SET;
>      } else {
>          env->nested_state->flags &= ~KVM_STATE_NESTED_GIF_SET;
>
> As "KVM_STATE_NESTED_GIF_SET" is not relevant to nVMX, this works for me
> but let me explore kernel side of this a bit more.
>
> --
> Vitaly
>
>

[-- Attachment #2: Type: text/html, Size: 1414 bytes --]

Re: 5.1.0-rc1 regression: reset fails with kvm and -cpu host

[Vitaly Kuznetsov]

Paolo Bonzini <pbonzini@redhat.com> writes:

> Yes, that seems correct.
>

Thank you!

It seems we also need to not reset HF2_GIF_MASK in
kvm_get_nested_state() when !svm and KVM_STATE_NESTED_GIF_SET is missing
(as nVMX doesn't set it). I'll be sending the patch shortly.

-- 
Vitaly