[bisected] KVM in 3.19-rc1 is completely broken

[bisected] KVM in 3.19-rc1 is completely broken

[Andy Lutomirski]

I can't get it to work at all.  It fails with:

KVM internal error. Suberror: 1
emulation failure
EAX=000dee58 EBX=00000000 ECX=00000000 EDX=00000cfd
ESI=00000059 EDI=00000000 EBP=00000000 ESP=00006fc4
EIP=000f17f4 EFL=00010012 [----A--] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
SS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
FS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
GS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
GDT=     000f6c58 00000037
IDT=     000f6c96 00000000
CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=e8 75 fc ff ff 89 f2 a8 10 89 d8 75 0a b9 74 17 ff ff ff d1 <5b>
5e c3 5b 5e e9 76 ff ff ff 57 56 53 8b 35 38 65 0f 00 85 f6 0f 88 be
00 00 00 0f b7 f6

The problem started with:

commit 0e60b0799fedc495a5c57dbd669de3c10d72edd2
Author: Igor Mammedov <imammedo@redhat.com>
Date:   Mon Dec 1 17:29:26 2014 +0000

    kvm: change memslot sorting rule from size to GFN

    it will allow to use binary search for GFN -> memslot
    lookups, reducing lookup cost with large slots amount.

    Signed-off-by: Igor Mammedov <imammedo@redhat.com>
    Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>


--Andy

Re: [bisected] KVM in 3.19-rc1 is completely broken

[Chen, Tiejun]

On 2014/12/23 9:38, Andy Lutomirski wrote:
> I can't get it to work at all.  It fails with:
>

How can we reproduce this? I just try the following command,

qemu-system-x86_64 -machine pc,accel=kvm -m 2048 -smp 2 -hda ubuntu.img

but at least I can boot that successfully and already live for a while. 
Here I installed ubuntu14.04 as a guest, and

Linux commit: aa39477b5692611b91ac9455ae588738852b3f60
Qemu commit: 7e58e2ac7778cca3234c33387e49577bb7732714

So any specific Kconfigs need to be enabled?

Tiejun

> KVM internal error. Suberror: 1
> emulation failure
> EAX=000dee58 EBX=00000000 ECX=00000000 EDX=00000cfd
> ESI=00000059 EDI=00000000 EBP=00000000 ESP=00006fc4
> EIP=000f17f4 EFL=00010012 [----A--] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
> CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
> SS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
> DS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
> FS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
> GS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
> LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
> TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
> GDT=     000f6c58 00000037
> IDT=     000f6c96 00000000
> CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
> DR3=0000000000000000
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000000
> Code=e8 75 fc ff ff 89 f2 a8 10 89 d8 75 0a b9 74 17 ff ff ff d1 <5b>
> 5e c3 5b 5e e9 76 ff ff ff 57 56 53 8b 35 38 65 0f 00 85 f6 0f 88 be
> 00 00 00 0f b7 f6
>
> The problem started with:
>
> commit 0e60b0799fedc495a5c57dbd669de3c10d72edd2
> Author: Igor Mammedov <imammedo@redhat.com>
> Date:   Mon Dec 1 17:29:26 2014 +0000
>
>      kvm: change memslot sorting rule from size to GFN
>
>      it will allow to use binary search for GFN -> memslot
>      lookups, reducing lookup cost with large slots amount.
>
>      Signed-off-by: Igor Mammedov <imammedo@redhat.com>
>      Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>
>
> --Andy
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: [bisected] KVM in 3.19-rc1 is completely broken

[Paolo Bonzini]

On 23/12/2014 09:19, Chen, Tiejun wrote:
> 
> How can we reproduce this? I just try the following command,
> 
> qemu-system-x86_64 -machine pc,accel=kvm -m 2048 -smp 2 -hda ubuntu.img
> 
> but at least I can boot that successfully and already live for a while.
> Here I installed ubuntu14.04 as a guest, and
> 
> Linux commit: aa39477b5692611b91ac9455ae588738852b3f60
> Qemu commit: 7e58e2ac7778cca3234c33387e49577bb7732714
> 
> So any specific Kconfigs need to be enabled?

I also cannot reproduce it (obviously).

Paolo

Re: [bisected] KVM in 3.19-rc1 is completely broken

[Andy Lutomirski]

On Tue, Dec 23, 2014 at 12:19 AM, Chen, Tiejun <tiejun.chen@intel.com> wrote:
> On 2014/12/23 9:38, Andy Lutomirski wrote:
>>
>> I can't get it to work at all.  It fails with:
>>
>
> How can we reproduce this? I just try the following command,
>
> qemu-system-x86_64 -machine pc,accel=kvm -m 2048 -smp 2 -hda ubuntu.img
>
> but at least I can boot that successfully and already live for a while. Here
> I installed ubuntu14.04 as a guest, and
>
> Linux commit: aa39477b5692611b91ac9455ae588738852b3f60
> Qemu commit: 7e58e2ac7778cca3234c33387e49577bb7732714
>
> So any specific Kconfigs need to be enabled?

I can reproduce it on Fedora 21 with:

rmmod kvm-intel
modprobe kvm-intel nested=1
virtme-configkernel --defconfig
make -j12 bzImage
virtme-run --pwd --kimg arch/x86/boot/bzImage --qemu-opts -m 1024 -smp 3

[wait for L1 to boot]

virtme-run --kimg arch/x86/boot/bzImage


This is with virtme from here:

https://git.kernel.org/cgit/utils/kernel/virtme/virtme.git/

I can try to be more helpful in the morning.


--Andy



>
> Tiejun
>
>> KVM internal error. Suberror: 1
>> emulation failure
>> EAX=000dee58 EBX=00000000 ECX=00000000 EDX=00000cfd
>> ESI=00000059 EDI=00000000 EBP=00000000 ESP=00006fc4
>> EIP=000f17f4 EFL=00010012 [----A--] CPL=0 II=0 A20=1 SMM=0 HLT=0
>> ES =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
>> CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
>> SS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
>> DS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
>> FS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
>> GS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
>> LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
>> TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
>> GDT=     000f6c58 00000037
>> IDT=     000f6c96 00000000
>> CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>> DR3=0000000000000000
>> DR6=00000000ffff0ff0 DR7=0000000000000400
>> EFER=0000000000000000
>> Code=e8 75 fc ff ff 89 f2 a8 10 89 d8 75 0a b9 74 17 ff ff ff d1 <5b>
>> 5e c3 5b 5e e9 76 ff ff ff 57 56 53 8b 35 38 65 0f 00 85 f6 0f 88 be
>> 00 00 00 0f b7 f6
>>
>> The problem started with:
>>
>> commit 0e60b0799fedc495a5c57dbd669de3c10d72edd2
>> Author: Igor Mammedov <imammedo@redhat.com>
>> Date:   Mon Dec 1 17:29:26 2014 +0000
>>
>>      kvm: change memslot sorting rule from size to GFN
>>
>>      it will allow to use binary search for GFN -> memslot
>>      lookups, reducing lookup cost with large slots amount.
>>
>>      Signed-off-by: Igor Mammedov <imammedo@redhat.com>
>>      Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>>
>>
>> --Andy
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>



-- 
Andy Lutomirski
AMA Capital Management, LLC

Re: [bisected] KVM in 3.19-rc1 is completely broken

[Paolo Bonzini]

> I can reproduce it using the same steps on a Sandy Bridge laptop, with
> whatever QEMU is packaged in Fedora 21.  I attached the config.
> 
> I also submitted a virtme update for Fedora Rawhide and 21 (20 is
> still building) in case it helps.  The build is here:
> 
> http://koji.fedoraproject.org/koji/buildinfo?buildID=600732

The other reporter bisected it to
0e60b0799fedc495a5c57dbd669de3c10d72edd2.  Can you try its parent?

Also, does it break with 3.18 host and 3.19-rc1 guest, or with
3.19-rc1 host and 3.18 guest?  (Sorry I should do this myself
but I'm a bit swamped due to vacation until Jan 6th).

Paolo

Re: [bisected] KVM in 3.19-rc1 is completely broken

[Andy Lutomirski]

On Tue, Dec 23, 2014 at 1:13 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>> I can reproduce it using the same steps on a Sandy Bridge laptop, with
>> whatever QEMU is packaged in Fedora 21.  I attached the config.
>>
>> I also submitted a virtme update for Fedora Rawhide and 21 (20 is
>> still building) in case it helps.  The build is here:
>>
>> http://koji.fedoraproject.org/koji/buildinfo?buildID=600732
>
> The other reporter bisected it to
> 0e60b0799fedc495a5c57dbd669de3c10d72edd2.  Can you try its parent?

That's what I bisected it to.  The parent works.

>
> Also, does it break with 3.18 host and 3.19-rc1 guest, or with
> 3.19-rc1 host and 3.18 guest?  (Sorry I should do this myself
> but I'm a bit swamped due to vacation until Jan 6th).
>

The breakage is with 3.17.7-something L0 and the same test kernel as
L1 and L2.  I think it breaks the same way with 3.19-rc1 as host and
guest without any nesting, but that's awkward to test right now.

--Andy

> Paolo



-- 
Andy Lutomirski
AMA Capital Management, LLC

Re: [bisected] KVM in 3.19-rc1 is completely broken

[Chen, Tiejun]

On 2014/12/24 5:29, Andy Lutomirski wrote:
> On Tue, Dec 23, 2014 at 1:13 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>
>>> I can reproduce it using the same steps on a Sandy Bridge laptop, with
>>> whatever QEMU is packaged in Fedora 21.  I attached the config.
>>>
>>> I also submitted a virtme update for Fedora Rawhide and 21 (20 is
>>> still building) in case it helps.  The build is here:
>>>
>>> http://koji.fedoraproject.org/koji/buildinfo?buildID=600732
>>
>> The other reporter bisected it to
>> 0e60b0799fedc495a5c57dbd669de3c10d72edd2.  Can you try its parent?
>
> That's what I bisected it to.  The parent works.
>
>>
>> Also, does it break with 3.18 host and 3.19-rc1 guest, or with
>> 3.19-rc1 host and 3.18 guest?  (Sorry I should do this myself
>> but I'm a bit swamped due to vacation until Jan 6th).
>>
>
> The breakage is with 3.17.7-something L0 and the same test kernel as
> L1 and L2.  I think it breaks the same way with 3.19-rc1 as host and
> guest without any nesting, but that's awkward to test right now.
>


Andy,

Could you try this?

Signed-off-by: Tiejun Chen <tiejun.chen@intel.com>
---
  virt/kvm/kvm_main.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index f528343..a2d928c 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -672,6 +672,7 @@ static void update_memslots(struct kvm_memslots *slots,
         WARN_ON(mslots[i].id != id);
         if (!new->npages) {
                 new->base_gfn = 0;
+               new->flags = 0;
                 if (mslots[i].npages)
                         slots->used_slots--;
         } else {
@@ -688,7 +689,7 @@ static void update_memslots(struct kvm_memslots *slots,
                 i++;
         }
         while (i > 0 &&
-              new->base_gfn > mslots[i - 1].base_gfn) {
+              new->base_gfn >= mslots[i - 1].base_gfn) {
                 mslots[i] = mslots[i - 1];
                 slots->id_to_index[mslots[i].id] = i;
                 i--;
-- 
1.9.1

Tiejun

Re: [bisected] KVM in 3.19-rc1 is completely broken

[Andy Lutomirski]

On Wed, Dec 24, 2014 at 12:23 AM, Chen, Tiejun <tiejun.chen@intel.com> wrote:
> On 2014/12/24 5:29, Andy Lutomirski wrote:
>>
>> On Tue, Dec 23, 2014 at 1:13 PM, Paolo Bonzini <pbonzini@redhat.com>
>> wrote:
>>>
>>>
>>>> I can reproduce it using the same steps on a Sandy Bridge laptop, with
>>>> whatever QEMU is packaged in Fedora 21.  I attached the config.
>>>>
>>>> I also submitted a virtme update for Fedora Rawhide and 21 (20 is
>>>> still building) in case it helps.  The build is here:
>>>>
>>>> http://koji.fedoraproject.org/koji/buildinfo?buildID=600732
>>>
>>>
>>> The other reporter bisected it to
>>> 0e60b0799fedc495a5c57dbd669de3c10d72edd2.  Can you try its parent?
>>
>>
>> That's what I bisected it to.  The parent works.
>>
>>>
>>> Also, does it break with 3.18 host and 3.19-rc1 guest, or with
>>> 3.19-rc1 host and 3.18 guest?  (Sorry I should do this myself
>>> but I'm a bit swamped due to vacation until Jan 6th).
>>>
>>
>> The breakage is with 3.17.7-something L0 and the same test kernel as
>> L1 and L2.  I think it breaks the same way with 3.19-rc1 as host and
>> guest without any nesting, but that's awkward to test right now.
>>
>
>
> Andy,
>
> Could you try this?
>
> Signed-off-by: Tiejun Chen <tiejun.chen@intel.com>

I applied it by hand, and it survives extremely light testing.

Tested-by: Andy Lutomirski <luto@amacapital.net>

Thanks,
Andy

> ---
>  virt/kvm/kvm_main.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index f528343..a2d928c 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -672,6 +672,7 @@ static void update_memslots(struct kvm_memslots *slots,
>         WARN_ON(mslots[i].id != id);
>         if (!new->npages) {
>                 new->base_gfn = 0;
> +               new->flags = 0;
>                 if (mslots[i].npages)
>                         slots->used_slots--;
>         } else {
> @@ -688,7 +689,7 @@ static void update_memslots(struct kvm_memslots *slots,
>                 i++;
>         }
>         while (i > 0 &&
> -              new->base_gfn > mslots[i - 1].base_gfn) {
> +              new->base_gfn >= mslots[i - 1].base_gfn) {
>                 mslots[i] = mslots[i - 1];
>                 slots->id_to_index[mslots[i].id] = i;
>                 i--;
> --
> 1.9.1
>
> Tiejun



-- 
Andy Lutomirski
AMA Capital Management, LLC

Re: [bisected] KVM in 3.19-rc1 is completely broken

[Chen, Tiejun]

On 2014/12/25 1:11, Andy Lutomirski wrote:
> On Wed, Dec 24, 2014 at 12:23 AM, Chen, Tiejun <tiejun.chen@intel.com> wrote:
>> On 2014/12/24 5:29, Andy Lutomirski wrote:
>>>
>>> On Tue, Dec 23, 2014 at 1:13 PM, Paolo Bonzini <pbonzini@redhat.com>
>>> wrote:
>>>>
>>>>
>>>>> I can reproduce it using the same steps on a Sandy Bridge laptop, with
>>>>> whatever QEMU is packaged in Fedora 21.  I attached the config.
>>>>>
>>>>> I also submitted a virtme update for Fedora Rawhide and 21 (20 is
>>>>> still building) in case it helps.  The build is here:
>>>>>
>>>>> http://koji.fedoraproject.org/koji/buildinfo?buildID=600732
>>>>
>>>>
>>>> The other reporter bisected it to
>>>> 0e60b0799fedc495a5c57dbd669de3c10d72edd2.  Can you try its parent?
>>>
>>>
>>> That's what I bisected it to.  The parent works.
>>>
>>>>
>>>> Also, does it break with 3.18 host and 3.19-rc1 guest, or with
>>>> 3.19-rc1 host and 3.18 guest?  (Sorry I should do this myself
>>>> but I'm a bit swamped due to vacation until Jan 6th).
>>>>
>>>
>>> The breakage is with 3.17.7-something L0 and the same test kernel as
>>> L1 and L2.  I think it breaks the same way with 3.19-rc1 as host and
>>> guest without any nesting, but that's awkward to test right now.
>>>
>>
>>
>> Andy,
>>
>> Could you try this?
>>
>> Signed-off-by: Tiejun Chen <tiejun.chen@intel.com>
>
> I applied it by hand, and it survives extremely light testing.
>
> Tested-by: Andy Lutomirski <luto@amacapital.net>
>

Looks good so thanks for your validation.

Tiejun

Re: [bisected] KVM in 3.19-rc1 is completely broken

[Chen, Tiejun]

On 2014/12/25 8:55, Chen, Tiejun wrote:
>
>
> On 2014/12/25 1:11, Andy Lutomirski wrote:
>> On Wed, Dec 24, 2014 at 12:23 AM, Chen, Tiejun <tiejun.chen@intel.com>
>> wrote:
>>> On 2014/12/24 5:29, Andy Lutomirski wrote:
>>>>
>>>> On Tue, Dec 23, 2014 at 1:13 PM, Paolo Bonzini <pbonzini@redhat.com>
>>>> wrote:
>>>>>
>>>>>
>>>>>> I can reproduce it using the same steps on a Sandy Bridge laptop,
>>>>>> with
>>>>>> whatever QEMU is packaged in Fedora 21.  I attached the config.
>>>>>>
>>>>>> I also submitted a virtme update for Fedora Rawhide and 21 (20 is
>>>>>> still building) in case it helps.  The build is here:
>>>>>>
>>>>>> http://koji.fedoraproject.org/koji/buildinfo?buildID=600732
>>>>>
>>>>>
>>>>> The other reporter bisected it to
>>>>> 0e60b0799fedc495a5c57dbd669de3c10d72edd2.  Can you try its parent?
>>>>
>>>>
>>>> That's what I bisected it to.  The parent works.
>>>>
>>>>>
>>>>> Also, does it break with 3.18 host and 3.19-rc1 guest, or with
>>>>> 3.19-rc1 host and 3.18 guest?  (Sorry I should do this myself
>>>>> but I'm a bit swamped due to vacation until Jan 6th).
>>>>>
>>>>
>>>> The breakage is with 3.17.7-something L0 and the same test kernel as
>>>> L1 and L2.  I think it breaks the same way with 3.19-rc1 as host and
>>>> guest without any nesting, but that's awkward to test right now.
>>>>
>>>
>>>
>>> Andy,
>>>
>>> Could you try this?
>>>
>>> Signed-off-by: Tiejun Chen <tiejun.chen@intel.com>
>>
>> I applied it by hand, and it survives extremely light testing.
>>
>> Tested-by: Andy Lutomirski <luto@amacapital.net>
>>
>
> Looks good so thanks for your validation.
>

I refine that I posted that fix in another thread since looks that will 
broken !next case. And I myself already run those test instructions you 
showed previously, now looks good.

Tiejun