How bad is KVM_EXIT_INTERNAL_ERROR?

How bad is KVM_EXIT_INTERNAL_ERROR?

[Dmitry Vyukov]

Hello,

I am getting some KVM_EXIT_INTERNAL_ERROR/KVM_INTERNAL_ERROR_EMULATION
while fuzzing KVM. Does it indicate a bug in kvm code?
To make it clear, the context is that guest does all kind of weird
stuff and if it kills itself it is fine. I am just trying to catch bad
effects on the host.
As far as I can tell, if qemu gets KVM_EXIT_INTERNAL_ERROR it will
fail. So for user it will look like qemu misbehaves. E.g. it will not
auto restart the VM.
If KVM_EXIT_INTERNAL_ERROR/KVM_INTERNAL_ERROR_EMULATION is OK, are
other KVM_EXIT_INTERNAL_ERROR error codes worse? Are there any other
indications that host is affected in a bad way? I mean besides BUGs,
WARNINGs and crashes of the host kernel.

Thanks

Re: How bad is KVM_EXIT_INTERNAL_ERROR?

[Paolo Bonzini]

On 21/12/2016 10:41, Dmitry Vyukov wrote:
> Hello,
> 
> I am getting some KVM_EXIT_INTERNAL_ERROR/KVM_INTERNAL_ERROR_EMULATION
> while fuzzing KVM. Does it indicate a bug in kvm code?

No, it just indicates something that isn't implemented.  Other
KVM_EXIT_INTERNAL_ERROR cases indicate a bug in the userspace code.

If you're running nested virtualization and the nested guest manages to
kill the host, that's bad.  Otherwise, they're harmless.

Paolo

> To make it clear, the context is that guest does all kind of weird
> stuff and if it kills itself it is fine. I am just trying to catch bad
> effects on the host.
> As far as I can tell, if qemu gets KVM_EXIT_INTERNAL_ERROR it will
> fail. So for user it will look like qemu misbehaves. E.g. it will not
> auto restart the VM.
> If KVM_EXIT_INTERNAL_ERROR/KVM_INTERNAL_ERROR_EMULATION is OK, are
> other KVM_EXIT_INTERNAL_ERROR error codes worse? Are there any other
> indications that host is affected in a bad way? I mean besides BUGs,
> WARNINGs and crashes of the host kernel.
> 
> Thanks
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: How bad is KVM_EXIT_INTERNAL_ERROR?

[Dmitry Vyukov]

On Wed, Dec 21, 2016 at 12:00 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>
> On 21/12/2016 10:41, Dmitry Vyukov wrote:
>> Hello,
>>
>> I am getting some KVM_EXIT_INTERNAL_ERROR/KVM_INTERNAL_ERROR_EMULATION
>> while fuzzing KVM. Does it indicate a bug in kvm code?
>
> No, it just indicates something that isn't implemented.  Other
> KVM_EXIT_INTERNAL_ERROR cases indicate a bug in the userspace code.

Something is not implemented where? You mean host userspace (VMM)?
But what about guest container killing whole VM? It looks equivalent
to non-root killing machine.


> If you're running nested virtualization and the nested guest manages to
> kill the host, that's bad.  Otherwise, they're harmless.
>
> Paolo
>
>> To make it clear, the context is that guest does all kind of weird
>> stuff and if it kills itself it is fine. I am just trying to catch bad
>> effects on the host.
>> As far as I can tell, if qemu gets KVM_EXIT_INTERNAL_ERROR it will
>> fail. So for user it will look like qemu misbehaves. E.g. it will not
>> auto restart the VM.
>> If KVM_EXIT_INTERNAL_ERROR/KVM_INTERNAL_ERROR_EMULATION is OK, are
>> other KVM_EXIT_INTERNAL_ERROR error codes worse? Are there any other
>> indications that host is affected in a bad way? I mean besides BUGs,
>> WARNINGs and crashes of the host kernel.
>>
>> Thanks
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>

Re: How bad is KVM_EXIT_INTERNAL_ERROR?

[Paolo Bonzini]

On 21/12/2016 12:06, Dmitry Vyukov wrote:
> On Wed, Dec 21, 2016 at 12:00 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>
>>
>> On 21/12/2016 10:41, Dmitry Vyukov wrote:
>>> Hello,
>>>
>>> I am getting some KVM_EXIT_INTERNAL_ERROR/KVM_INTERNAL_ERROR_EMULATION
>>> while fuzzing KVM. Does it indicate a bug in kvm code?
>>
>> No, it just indicates something that isn't implemented.  Other
>> KVM_EXIT_INTERNAL_ERROR cases indicate a bug in the userspace code.
> 
> Something is not implemented where? You mean host userspace (VMM)?

No, in KVM's instruction emulator.

> But what about guest container killing whole VM? It looks equivalent
> to non-root killing machine.

KVM_INTERNAL_ERROR_EMULATION only kills the whole VM at CPL=0.  At CPL=3
or in a nested VM it would be translated to an undefined opcode
exception, exactly for this reason.

Feel free to send one or two examples, they should be easy to analyze.
But I suspect it's just a combination of executing garbage instructions
with garbage processor state.  If so, an unprivileged guest that can
trigger KVM_INTERNAL_ERROR_EMULATION in a kernel most likely has gotten
arbitrary code execution already.

Paolo

> 
>> If you're running nested virtualization and the nested guest manages to
>> kill the host, that's bad.  Otherwise, they're harmless.
>>
>> Paolo
>>
>>> To make it clear, the context is that guest does all kind of weird
>>> stuff and if it kills itself it is fine. I am just trying to catch bad
>>> effects on the host.
>>> As far as I can tell, if qemu gets KVM_EXIT_INTERNAL_ERROR it will
>>> fail. So for user it will look like qemu misbehaves. E.g. it will not
>>> auto restart the VM.
>>> If KVM_EXIT_INTERNAL_ERROR/KVM_INTERNAL_ERROR_EMULATION is OK, are
>>> other KVM_EXIT_INTERNAL_ERROR error codes worse? Are there any other
>>> indications that host is affected in a bad way? I mean besides BUGs,
>>> WARNINGs and crashes of the host kernel.
>>>
>>> Thanks
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: How bad is KVM_EXIT_INTERNAL_ERROR?

[Dmitry Vyukov]

On Wed, Dec 21, 2016 at 2:26 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>
> On 21/12/2016 12:06, Dmitry Vyukov wrote:
>> On Wed, Dec 21, 2016 at 12:00 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>>
>>>
>>> On 21/12/2016 10:41, Dmitry Vyukov wrote:
>>>> Hello,
>>>>
>>>> I am getting some KVM_EXIT_INTERNAL_ERROR/KVM_INTERNAL_ERROR_EMULATION
>>>> while fuzzing KVM. Does it indicate a bug in kvm code?
>>>
>>> No, it just indicates something that isn't implemented.  Other
>>> KVM_EXIT_INTERNAL_ERROR cases indicate a bug in the userspace code.
>>
>> Something is not implemented where? You mean host userspace (VMM)?
>
> No, in KVM's instruction emulator.

You mean that KVM_INTERNAL_ERROR_EMULATION means that something is not
implemented in KVM's instruction emulator, right?

What about other error codes? You said that they indicate a bug in the
userspace code. What userspace code?
I am not getting KVM_INTERNAL_ERROR_DELIVERY_EV as well.


>> But what about guest container killing whole VM? It looks equivalent
>> to non-root killing machine.
>
> KVM_INTERNAL_ERROR_EMULATION only kills the whole VM at CPL=0.  At CPL=3
> or in a nested VM it would be translated to an undefined opcode
> exception, exactly for this reason.
>
> Feel free to send one or two examples, they should be easy to analyze.
> But I suspect it's just a combination of executing garbage instructions
> with garbage processor state.  If so, an unprivileged guest that can
> trigger KVM_INTERNAL_ERROR_EMULATION in a kernel most likely has gotten
> arbitrary code execution already.

What's the simplest way to verify that the error was triggered by
CPL=0 code? Do KVM_GET_SREGS and check value code segment?



>>> If you're running nested virtualization and the nested guest manages to
>>> kill the host, that's bad.  Otherwise, they're harmless.
>>>
>>> Paolo
>>>
>>>> To make it clear, the context is that guest does all kind of weird
>>>> stuff and if it kills itself it is fine. I am just trying to catch bad
>>>> effects on the host.
>>>> As far as I can tell, if qemu gets KVM_EXIT_INTERNAL_ERROR it will
>>>> fail. So for user it will look like qemu misbehaves. E.g. it will not
>>>> auto restart the VM.
>>>> If KVM_EXIT_INTERNAL_ERROR/KVM_INTERNAL_ERROR_EMULATION is OK, are
>>>> other KVM_EXIT_INTERNAL_ERROR error codes worse? Are there any other
>>>> indications that host is affected in a bad way? I mean besides BUGs,
>>>> WARNINGs and crashes of the host kernel.
>>>>
>>>> Thanks
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>

Re: How bad is KVM_EXIT_INTERNAL_ERROR?

[Paolo Bonzini]

----- Original Message -----
> From: "Dmitry Vyukov" <dvyukov@google.com>
> To: "Paolo Bonzini" <pbonzini@redhat.com>
> Cc: "Radim Krčmář" <rkrcmar@redhat.com>, "KVM list" <kvm@vger.kernel.org>, "Steve Rutherford"
> <srutherford@google.com>
> Sent: Wednesday, December 21, 2016 8:05:15 PM
> Subject: Re: How bad is KVM_EXIT_INTERNAL_ERROR?
> 
> On Wed, Dec 21, 2016 at 2:26 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> >
> >
> > On 21/12/2016 12:06, Dmitry Vyukov wrote:
> >> On Wed, Dec 21, 2016 at 12:00 PM, Paolo Bonzini <pbonzini@redhat.com>
> >> wrote:
> >>>
> >>>
> >>> On 21/12/2016 10:41, Dmitry Vyukov wrote:
> >>>> Hello,
> >>>>
> >>>> I am getting some KVM_EXIT_INTERNAL_ERROR/KVM_INTERNAL_ERROR_EMULATION
> >>>> while fuzzing KVM. Does it indicate a bug in kvm code?
> >>>
> >>> No, it just indicates something that isn't implemented.  Other
> >>> KVM_EXIT_INTERNAL_ERROR cases indicate a bug in the userspace code.
> >>
> >> Something is not implemented where? You mean host userspace (VMM)?
> >
> > No, in KVM's instruction emulator.
> 
> You mean that KVM_INTERNAL_ERROR_EMULATION means that something is not
> implemented in KVM's instruction emulator, right?
> 
> What about other error codes? You said that they indicate a bug in the
> userspace code. What userspace code?

For example setting processor state (KVM_SET_SREGS) to something that
makes no sense.

> > KVM_INTERNAL_ERROR_EMULATION only kills the whole VM at CPL=0.  At CPL=3
> > or in a nested VM it would be translated to an undefined opcode
> > exception, exactly for this reason.
> >
> > Feel free to send one or two examples, they should be easy to analyze.
> > But I suspect it's just a combination of executing garbage instructions
> > with garbage processor state.  If so, an unprivileged guest that can
> > trigger KVM_INTERNAL_ERROR_EMULATION in a kernel most likely has gotten
> > arbitrary code execution already.
> 
> What's the simplest way to verify that the error was triggered by
> CPL=0 code? Do KVM_GET_SREGS and check value code segment?

CPL is actually SS.DPL, so that's what you would have to check.  But
really the only case of KVM_INTERNAL_ERROR_EMULATION that fuzzing can
produce is probably this one:

        if (!is_guest_mode(vcpu) && kvm_x86_ops->get_cpl(vcpu) == 0) {
                vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
                vcpu->run->internal.suberror = KVM_INTERNAL_ERROR_EMULATION;
                vcpu->run->internal.ndata = 0;
                r = EMULATE_FAIL;
        }
        kvm_queue_exception(vcpu, UD_VECTOR);

There are other cases of KVM_INTERNAL_ERROR_EMULATION corresponding to
hardware task switching.  Neither Linux nor Windows provide userspace with
means to trigger those, but nested virtualization probably could on AMD.

Paolo

Re: How bad is KVM_EXIT_INTERNAL_ERROR?

[Dmitry Vyukov]

On Thu, Dec 22, 2016 at 10:05 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> On Wed, Dec 21, 2016 at 2:26 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> >
>> >
>> > On 21/12/2016 12:06, Dmitry Vyukov wrote:
>> >> On Wed, Dec 21, 2016 at 12:00 PM, Paolo Bonzini <pbonzini@redhat.com>
>> >> wrote:
>> >>>
>> >>>
>> >>> On 21/12/2016 10:41, Dmitry Vyukov wrote:
>> >>>> Hello,
>> >>>>
>> >>>> I am getting some KVM_EXIT_INTERNAL_ERROR/KVM_INTERNAL_ERROR_EMULATION
>> >>>> while fuzzing KVM. Does it indicate a bug in kvm code?
>> >>>
>> >>> No, it just indicates something that isn't implemented.  Other
>> >>> KVM_EXIT_INTERNAL_ERROR cases indicate a bug in the userspace code.
>> >>
>> >> Something is not implemented where? You mean host userspace (VMM)?
>> >
>> > No, in KVM's instruction emulator.
>>
>> You mean that KVM_INTERNAL_ERROR_EMULATION means that something is not
>> implemented in KVM's instruction emulator, right?
>>
>> What about other error codes? You said that they indicate a bug in the
>> userspace code. What userspace code?
>
> For example setting processor state (KVM_SET_SREGS) to something that
> makes no sense.

For now I always set sregs to simple "canonical" values that put the
processor into 64-bit long mode. But I guess that guest can do the
setup that makes no sense later.

>> > KVM_INTERNAL_ERROR_EMULATION only kills the whole VM at CPL=0.  At CPL=3
>> > or in a nested VM it would be translated to an undefined opcode
>> > exception, exactly for this reason.
>> >
>> > Feel free to send one or two examples, they should be easy to analyze.
>> > But I suspect it's just a combination of executing garbage instructions
>> > with garbage processor state.  If so, an unprivileged guest that can
>> > trigger KVM_INTERNAL_ERROR_EMULATION in a kernel most likely has gotten
>> > arbitrary code execution already.
>>
>> What's the simplest way to verify that the error was triggered by
>> CPL=0 code? Do KVM_GET_SREGS and check value code segment?
>
> CPL is actually SS.DPL, so that's what you would have to check.  But
> really the only case of KVM_INTERNAL_ERROR_EMULATION that fuzzing can
> produce is probably this one:
>
>         if (!is_guest_mode(vcpu) && kvm_x86_ops->get_cpl(vcpu) == 0) {
>                 vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
>                 vcpu->run->internal.suberror = KVM_INTERNAL_ERROR_EMULATION;
>                 vcpu->run->internal.ndata = 0;
>                 r = EMULATE_FAIL;
>         }
>         kvm_queue_exception(vcpu, UD_VECTOR);
>
> There are other cases of KVM_INTERNAL_ERROR_EMULATION corresponding to
> hardware task switching.  Neither Linux nor Windows provide userspace with
> means to trigger those, but nested virtualization probably could on AMD.

Yes, I am getting only that one (added WARN to the other one and it
does not fire).


I understand the picture now. I will ignore all KVM_EXIT_INTERNAL_ERROR.

Thanks!