Help debugging a regression in KVM Module

Help debugging a regression in KVM Module

[Peter Lieven]

Hi,

some time a go I stumbled across a regression in the KVM Module that has been introduced somewhere
between 3.17 and 3.19.

I have a rather old openSUSE guest with an XFS filesystem which realiably crashes after some live migrations.
I originally believed that the issue might be related to my setup with a 3.12 host kernel and kvm-kmod 3.19,
but I now found that it is also still present with a 3.19 host kernel with included 3.19 kvm module.

My idea was to continue testing on a 3.12 host kernel and then bisect all commits to the kvm related parts.

Now my question is how to best bisect only kvm related changes (those that go into kvm-kmod)?

Thanks,
Peter

[Qemu-devel] Help debugging a regression in KVM Module

[Peter Lieven]

Hi,

some time a go I stumbled across a regression in the KVM Module that has been introduced somewhere
between 3.17 and 3.19.

I have a rather old openSUSE guest with an XFS filesystem which realiably crashes after some live migrations.
I originally believed that the issue might be related to my setup with a 3.12 host kernel and kvm-kmod 3.19,
but I now found that it is also still present with a 3.19 host kernel with included 3.19 kvm module.

My idea was to continue testing on a 3.12 host kernel and then bisect all commits to the kvm related parts.

Now my question is how to best bisect only kvm related changes (those that go into kvm-kmod)?

Thanks,
Peter

Re: Help debugging a regression in KVM Module

[Paolo Bonzini]

----- Original Message -----
> From: "Peter Lieven" <pl@kamp.de>
> To: qemu-devel@nongnu.org, kvm@vger.kernel.org
> Cc: "Paolo Bonzini" <pbonzini@redhat.com>
> Sent: Friday, August 14, 2015 1:11:34 PM
> Subject: Help debugging a regression in KVM Module
> 
> Hi,
> 
> some time a go I stumbled across a regression in the KVM Module that has been
> introduced somewhere
> between 3.17 and 3.19.
> 
> I have a rather old openSUSE guest with an XFS filesystem which realiably
> crashes after some live migrations.
> I originally believed that the issue might be related to my setup with a 3.12
> host kernel and kvm-kmod 3.19,
> but I now found that it is also still present with a 3.19 host kernel with
> included 3.19 kvm module.
> 
> My idea was to continue testing on a 3.12 host kernel and then bisect all
> commits to the kvm related parts.
> 
> Now my question is how to best bisect only kvm related changes (those that go
> into kvm-kmod)?

I haven't forgotten this.  Sorry. :(

Unfortunately I'll be away for three weeks, but I'll make it a priority
when I'm back.

Paolo

Re: [Qemu-devel] Help debugging a regression in KVM Module

[Paolo Bonzini]

----- Original Message -----
> From: "Peter Lieven" <pl@kamp.de>
> To: qemu-devel@nongnu.org, kvm@vger.kernel.org
> Cc: "Paolo Bonzini" <pbonzini@redhat.com>
> Sent: Friday, August 14, 2015 1:11:34 PM
> Subject: Help debugging a regression in KVM Module
> 
> Hi,
> 
> some time a go I stumbled across a regression in the KVM Module that has been
> introduced somewhere
> between 3.17 and 3.19.
> 
> I have a rather old openSUSE guest with an XFS filesystem which realiably
> crashes after some live migrations.
> I originally believed that the issue might be related to my setup with a 3.12
> host kernel and kvm-kmod 3.19,
> but I now found that it is also still present with a 3.19 host kernel with
> included 3.19 kvm module.
> 
> My idea was to continue testing on a 3.12 host kernel and then bisect all
> commits to the kvm related parts.
> 
> Now my question is how to best bisect only kvm related changes (those that go
> into kvm-kmod)?

I haven't forgotten this.  Sorry. :(

Unfortunately I'll be away for three weeks, but I'll make it a priority
when I'm back.

Paolo

Re: Help debugging a regression in KVM Module

[Peter Lieven]

Am 14.08.2015 um 15:01 schrieb Paolo Bonzini:
>
> ----- Original Message -----
>> From: "Peter Lieven" <pl@kamp.de>
>> To: qemu-devel@nongnu.org, kvm@vger.kernel.org
>> Cc: "Paolo Bonzini" <pbonzini@redhat.com>
>> Sent: Friday, August 14, 2015 1:11:34 PM
>> Subject: Help debugging a regression in KVM Module
>>
>> Hi,
>>
>> some time a go I stumbled across a regression in the KVM Module that has been
>> introduced somewhere
>> between 3.17 and 3.19.
>>
>> I have a rather old openSUSE guest with an XFS filesystem which realiably
>> crashes after some live migrations.
>> I originally believed that the issue might be related to my setup with a 3.12
>> host kernel and kvm-kmod 3.19,
>> but I now found that it is also still present with a 3.19 host kernel with
>> included 3.19 kvm module.
>>
>> My idea was to continue testing on a 3.12 host kernel and then bisect all
>> commits to the kvm related parts.
>>
>> Now my question is how to best bisect only kvm related changes (those that go
>> into kvm-kmod)?
> I haven't forgotten this.  Sorry. :(
>
> Unfortunately I'll be away for three weeks, but I'll make it a priority
> when I'm back.

Its not time critical, but I think its worth investigating as it might affect
other systems as well - and maybe XFS is only very sensitive.

I suppose you are going on vacation. Enjoy!

Peter

Re: [Qemu-devel] Help debugging a regression in KVM Module

[Peter Lieven]

Am 14.08.2015 um 15:01 schrieb Paolo Bonzini:
>
> ----- Original Message -----
>> From: "Peter Lieven" <pl@kamp.de>
>> To: qemu-devel@nongnu.org, kvm@vger.kernel.org
>> Cc: "Paolo Bonzini" <pbonzini@redhat.com>
>> Sent: Friday, August 14, 2015 1:11:34 PM
>> Subject: Help debugging a regression in KVM Module
>>
>> Hi,
>>
>> some time a go I stumbled across a regression in the KVM Module that has been
>> introduced somewhere
>> between 3.17 and 3.19.
>>
>> I have a rather old openSUSE guest with an XFS filesystem which realiably
>> crashes after some live migrations.
>> I originally believed that the issue might be related to my setup with a 3.12
>> host kernel and kvm-kmod 3.19,
>> but I now found that it is also still present with a 3.19 host kernel with
>> included 3.19 kvm module.
>>
>> My idea was to continue testing on a 3.12 host kernel and then bisect all
>> commits to the kvm related parts.
>>
>> Now my question is how to best bisect only kvm related changes (those that go
>> into kvm-kmod)?
> I haven't forgotten this.  Sorry. :(
>
> Unfortunately I'll be away for three weeks, but I'll make it a priority
> when I'm back.

Its not time critical, but I think its worth investigating as it might affect
other systems as well - and maybe XFS is only very sensitive.

I suppose you are going on vacation. Enjoy!

Peter

Re: [Qemu-devel] Help debugging a regression in KVM Module

[Alex Bennée]

Peter Lieven <pl@kamp.de> writes:

> Hi,
>
> some time a go I stumbled across a regression in the KVM Module that has been introduced somewhere
> between 3.17 and 3.19.
>
> I have a rather old openSUSE guest with an XFS filesystem which realiably crashes after some live migrations.
> I originally believed that the issue might be related to my setup with a 3.12 host kernel and kvm-kmod 3.19,
> but I now found that it is also still present with a 3.19 host kernel with included 3.19 kvm module.
>
> My idea was to continue testing on a 3.12 host kernel and then bisect all commits to the kvm related parts.
>
> Now my question is how to best bisect only kvm related changes (those
> that go into kvm-kmod)?

In general I don't bother. As it is a bisection you eliminate half the
commits at a time you get their fairly quickly anyway. However you can
tell bisect which parts of the tree you car about:

  git bisect start -- arch/arm64/kvm include/linux/kvm* include/uapi/linux/kvm* virt/kvm/


>
> Thanks,
> Peter

-- 
Alex Bennée

Re: [Qemu-devel] Help debugging a regression in KVM Module

[Alex Bennée]

Peter Lieven <pl@kamp.de> writes:

> Hi,
>
> some time a go I stumbled across a regression in the KVM Module that has been introduced somewhere
> between 3.17 and 3.19.
>
> I have a rather old openSUSE guest with an XFS filesystem which realiably crashes after some live migrations.
> I originally believed that the issue might be related to my setup with a 3.12 host kernel and kvm-kmod 3.19,
> but I now found that it is also still present with a 3.19 host kernel with included 3.19 kvm module.
>
> My idea was to continue testing on a 3.12 host kernel and then bisect all commits to the kvm related parts.
>
> Now my question is how to best bisect only kvm related changes (those
> that go into kvm-kmod)?

In general I don't bother. As it is a bisection you eliminate half the
commits at a time you get their fairly quickly anyway. However you can
tell bisect which parts of the tree you car about:

  git bisect start -- arch/arm64/kvm include/linux/kvm* include/uapi/linux/kvm* virt/kvm/


>
> Thanks,
> Peter

-- 
Alex Bennée

Re: [Qemu-devel] Help debugging a regression in KVM Module

[Peter Lieven]

Am 14.08.2015 um 22:01 schrieb Alex Bennée:
> Peter Lieven <pl@kamp.de> writes:
>
>> Hi,
>>
>> some time a go I stumbled across a regression in the KVM Module that has been introduced somewhere
>> between 3.17 and 3.19.
>>
>> I have a rather old openSUSE guest with an XFS filesystem which realiably crashes after some live migrations.
>> I originally believed that the issue might be related to my setup with a 3.12 host kernel and kvm-kmod 3.19,
>> but I now found that it is also still present with a 3.19 host kernel with included 3.19 kvm module.
>>
>> My idea was to continue testing on a 3.12 host kernel and then bisect all commits to the kvm related parts.
>>
>> Now my question is how to best bisect only kvm related changes (those
>> that go into kvm-kmod)?
> In general I don't bother. As it is a bisection you eliminate half the
> commits at a time you get their fairly quickly anyway. However you can
> tell bisect which parts of the tree you car about:
>
>    git bisect start -- arch/arm64/kvm include/linux/kvm* include/uapi/linux/kvm* virt/kvm/

Yes, I just have to find out how exactly that works out if I want to bisect the linux submodule
of the kvm-kmod repository. But thanks for the pointer on how to limit the directories.

Thanks,
Peter

Re: [Qemu-devel] Help debugging a regression in KVM Module

[Peter Lieven]

Am 14.08.2015 um 22:01 schrieb Alex Bennée:
> Peter Lieven <pl@kamp.de> writes:
>
>> Hi,
>>
>> some time a go I stumbled across a regression in the KVM Module that has been introduced somewhere
>> between 3.17 and 3.19.
>>
>> I have a rather old openSUSE guest with an XFS filesystem which realiably crashes after some live migrations.
>> I originally believed that the issue might be related to my setup with a 3.12 host kernel and kvm-kmod 3.19,
>> but I now found that it is also still present with a 3.19 host kernel with included 3.19 kvm module.
>>
>> My idea was to continue testing on a 3.12 host kernel and then bisect all commits to the kvm related parts.
>>
>> Now my question is how to best bisect only kvm related changes (those
>> that go into kvm-kmod)?
> In general I don't bother. As it is a bisection you eliminate half the
> commits at a time you get their fairly quickly anyway. However you can
> tell bisect which parts of the tree you car about:
>
>    git bisect start -- arch/arm64/kvm include/linux/kvm* include/uapi/linux/kvm* virt/kvm/

Yes, I just have to find out how exactly that works out if I want to bisect the linux submodule
of the kvm-kmod repository. But thanks for the pointer on how to limit the directories.

Thanks,
Peter

Re: [Qemu-devel] Help debugging a regression in KVM Module

[Peter Lieven]

Am 14.08.2015 um 22:01 schrieb Alex Bennée:
> Peter Lieven <pl@kamp.de> writes:
>
>> Hi,
>>
>> some time a go I stumbled across a regression in the KVM Module that has been introduced somewhere
>> between 3.17 and 3.19.
>>
>> I have a rather old openSUSE guest with an XFS filesystem which realiably crashes after some live migrations.
>> I originally believed that the issue might be related to my setup with a 3.12 host kernel and kvm-kmod 3.19,
>> but I now found that it is also still present with a 3.19 host kernel with included 3.19 kvm module.
>>
>> My idea was to continue testing on a 3.12 host kernel and then bisect all commits to the kvm related parts.
>>
>> Now my question is how to best bisect only kvm related changes (those
>> that go into kvm-kmod)?
> In general I don't bother. As it is a bisection you eliminate half the
> commits at a time you get their fairly quickly anyway. However you can
> tell bisect which parts of the tree you car about:
>
>    git bisect start -- arch/arm64/kvm include/linux/kvm* include/uapi/linux/kvm* virt/kvm/

After some experiments I was able to find out the bad commit that introduced the regression:

commit f30ebc312ca9def25650b4e1d01cdb425c310dca
Author: Radim Krčmář <rkrcmar@redhat.com>
Date:   Thu Oct 30 15:06:47 2014 +0100

It seems that this optimisation is not working reliabliy after live migration. I can't reproduce if
I take a 3.19 kernel and revert this single commit.

Peter

Re: [Qemu-devel] Help debugging a regression in KVM Module

[Peter Lieven]

Am 14.08.2015 um 22:01 schrieb Alex Bennée:
> Peter Lieven <pl@kamp.de> writes:
>
>> Hi,
>>
>> some time a go I stumbled across a regression in the KVM Module that has been introduced somewhere
>> between 3.17 and 3.19.
>>
>> I have a rather old openSUSE guest with an XFS filesystem which realiably crashes after some live migrations.
>> I originally believed that the issue might be related to my setup with a 3.12 host kernel and kvm-kmod 3.19,
>> but I now found that it is also still present with a 3.19 host kernel with included 3.19 kvm module.
>>
>> My idea was to continue testing on a 3.12 host kernel and then bisect all commits to the kvm related parts.
>>
>> Now my question is how to best bisect only kvm related changes (those
>> that go into kvm-kmod)?
> In general I don't bother. As it is a bisection you eliminate half the
> commits at a time you get their fairly quickly anyway. However you can
> tell bisect which parts of the tree you car about:
>
>    git bisect start -- arch/arm64/kvm include/linux/kvm* include/uapi/linux/kvm* virt/kvm/

After some experiments I was able to find out the bad commit that introduced the regression:

commit f30ebc312ca9def25650b4e1d01cdb425c310dca
Author: Radim Krčmář <rkrcmar@redhat.com>
Date:   Thu Oct 30 15:06:47 2014 +0100

It seems that this optimisation is not working reliabliy after live migration. I can't reproduce if
I take a 3.19 kernel and revert this single commit.

Peter

Re: [Qemu-devel] Help debugging a regression in KVM Module

[Radim Krčmář]

2015-08-18 16:54+0200, Peter Lieven:
> After some experiments I was able to find out the bad commit that introduced the regression:
> 
> commit f30ebc312ca9def25650b4e1d01cdb425c310dca
> Author: Radim Krčmář <rkrcmar@redhat.com>
> Date:   Thu Oct 30 15:06:47 2014 +0100
> 
> It seems that this optimisation is not working reliabliy after live migration. I can't reproduce if
> I take a 3.19 kernel and revert this single commit.

Hello, this bug has gone unnoticed for a long time so it is fixed only
since v4.1 (and v3.19.stable was dead at that point).

commit b6ac069532218027f2991cba01d7a72a200688b0
Author: Radim Krčmář <rkrcmar@redhat.com>
Date:   Fri Jun 5 20:57:41 2015 +0200

    KVM: x86: fix lapic.timer_mode on restore
    
    lapic.timer_mode was not properly initialized after migration, which
    broke few useful things, like login, by making every sleep eternal.
    
    Fix this by calling apic_update_lvtt in kvm_apic_post_state_restore.
    
    There are other slowpaths that update lvtt, so this patch makes sure
    something similar doesn't happen again by calling apic_update_lvtt
    after every modification.
    
    Cc: stable@vger.kernel.org
    Fixes: f30ebc312ca9 ("KVM: x86: optimize some accesses to LVTT and SPIV")
    Signed-off-by: Radim Krčmář <rkrcmar@redhat.com>
    Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

Re: [Qemu-devel] Help debugging a regression in KVM Module

[Radim Krčmář]

2015-08-18 16:54+0200, Peter Lieven:
> After some experiments I was able to find out the bad commit that introduced the regression:
> 
> commit f30ebc312ca9def25650b4e1d01cdb425c310dca
> Author: Radim Krčmář <rkrcmar@redhat.com>
> Date:   Thu Oct 30 15:06:47 2014 +0100
> 
> It seems that this optimisation is not working reliabliy after live migration. I can't reproduce if
> I take a 3.19 kernel and revert this single commit.

Hello, this bug has gone unnoticed for a long time so it is fixed only
since v4.1 (and v3.19.stable was dead at that point).

commit b6ac069532218027f2991cba01d7a72a200688b0
Author: Radim Krčmář <rkrcmar@redhat.com>
Date:   Fri Jun 5 20:57:41 2015 +0200

    KVM: x86: fix lapic.timer_mode on restore
    
    lapic.timer_mode was not properly initialized after migration, which
    broke few useful things, like login, by making every sleep eternal.
    
    Fix this by calling apic_update_lvtt in kvm_apic_post_state_restore.
    
    There are other slowpaths that update lvtt, so this patch makes sure
    something similar doesn't happen again by calling apic_update_lvtt
    after every modification.
    
    Cc: stable@vger.kernel.org
    Fixes: f30ebc312ca9 ("KVM: x86: optimize some accesses to LVTT and SPIV")
    Signed-off-by: Radim Krčmář <rkrcmar@redhat.com>
    Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

Re: [Qemu-devel] Help debugging a regression in KVM Module

[Peter Lieven]

> Am 18.08.2015 um 17:25 schrieb Radim Krčmář <rkrcmar@redhat.com>:
> 
> 2015-08-18 16:54+0200, Peter Lieven:
>> After some experiments I was able to find out the bad commit that introduced the regression:
>> 
>> commit f30ebc312ca9def25650b4e1d01cdb425c310dca
>> Author: Radim Krčmář <rkrcmar@redhat.com>
>> Date:   Thu Oct 30 15:06:47 2014 +0100
>> 
>> It seems that this optimisation is not working reliabliy after live migration. I can't reproduce if
>> I take a 3.19 kernel and revert this single commit.
> 
> Hello, this bug has gone unnoticed for a long time so it is fixed only
> since v4.1 (and v3.19.stable was dead at that point).

thanks for the pointer. i noticed the regression some time ago, but never found the time to debug. some distros rely on 3.19 e.g. Ubuntu LTS 14.04.2. I will try to ping the maintainer.

Peter

> 
> commit b6ac069532218027f2991cba01d7a72a200688b0
> Author: Radim Krčmář <rkrcmar@redhat.com>
> Date:   Fri Jun 5 20:57:41 2015 +0200
> 
>    KVM: x86: fix lapic.timer_mode on restore
> 
>    lapic.timer_mode was not properly initialized after migration, which
>    broke few useful things, like login, by making every sleep eternal.
> 
>    Fix this by calling apic_update_lvtt in kvm_apic_post_state_restore.
> 
>    There are other slowpaths that update lvtt, so this patch makes sure
>    something similar doesn't happen again by calling apic_update_lvtt
>    after every modification.
> 
>    Cc: stable@vger.kernel.org
>    Fixes: f30ebc312ca9 ("KVM: x86: optimize some accesses to LVTT and SPIV")
>    Signed-off-by: Radim Krčmář <rkrcmar@redhat.com>
>    Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

Re: [Qemu-devel] Help debugging a regression in KVM Module

[Peter Lieven]

> Am 18.08.2015 um 17:25 schrieb Radim Krčmář <rkrcmar@redhat.com>:
> 
> 2015-08-18 16:54+0200, Peter Lieven:
>> After some experiments I was able to find out the bad commit that introduced the regression:
>> 
>> commit f30ebc312ca9def25650b4e1d01cdb425c310dca
>> Author: Radim Krčmář <rkrcmar@redhat.com>
>> Date:   Thu Oct 30 15:06:47 2014 +0100
>> 
>> It seems that this optimisation is not working reliabliy after live migration. I can't reproduce if
>> I take a 3.19 kernel and revert this single commit.
> 
> Hello, this bug has gone unnoticed for a long time so it is fixed only
> since v4.1 (and v3.19.stable was dead at that point).

thanks for the pointer. i noticed the regression some time ago, but never found the time to debug. some distros rely on 3.19 e.g. Ubuntu LTS 14.04.2. I will try to ping the maintainer.

Peter

> 
> commit b6ac069532218027f2991cba01d7a72a200688b0
> Author: Radim Krčmář <rkrcmar@redhat.com>
> Date:   Fri Jun 5 20:57:41 2015 +0200
> 
>    KVM: x86: fix lapic.timer_mode on restore
> 
>    lapic.timer_mode was not properly initialized after migration, which
>    broke few useful things, like login, by making every sleep eternal.
> 
>    Fix this by calling apic_update_lvtt in kvm_apic_post_state_restore.
> 
>    There are other slowpaths that update lvtt, so this patch makes sure
>    something similar doesn't happen again by calling apic_update_lvtt
>    after every modification.
> 
>    Cc: stable@vger.kernel.org
>    Fixes: f30ebc312ca9 ("KVM: x86: optimize some accesses to LVTT and SPIV")
>    Signed-off-by: Radim Krčmář <rkrcmar@redhat.com>
>    Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>