Re: [Qemu-devel] race between kvm-kmod-3.0 and kvm-kmod-3.3 // was: race condition in qemu-kvm-1.0.1

Re: [Qemu-devel] race between kvm-kmod-3.0 and kvm-kmod-3.3 // was: race condition in qemu-kvm-1.0.1

[Avi Kivity]

On 06/28/2012 12:38 PM, Peter Lieven wrote:
> does anyone know whats that here in handle_mmio?
> 
>     /* hack: Red Hat 7.1 generates these weird accesses. */
>     if ((addr > 0xa0000-4 && addr <= 0xa0000) && kvm_run->mmio.len == 3)
>         return 0;
> 

Just what it says.  There is a 4-byte access to address 0x9ffff.  The
first byte lies in RAM, the next three bytes are in mmio.  qemu is
geared to power-of-two accesses even though x86 can generate accesses to
any number of bytes between 1 and 8.

It appears that this has happened with your guest.  It's not impossible
that it's genuine.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] race between kvm-kmod-3.0 and kvm-kmod-3.3 // was: race condition in qemu-kvm-1.0.1

[Peter Lieven]

On 02.07.2012 17:05, Avi Kivity wrote:
> On 06/28/2012 12:38 PM, Peter Lieven wrote:
>> does anyone know whats that here in handle_mmio?
>>
>>      /* hack: Red Hat 7.1 generates these weird accesses. */
>>      if ((addr>  0xa0000-4&&  addr<= 0xa0000)&&  kvm_run->mmio.len == 3)
>>          return 0;
>>
> Just what it says.  There is a 4-byte access to address 0x9ffff.  The
> first byte lies in RAM, the next three bytes are in mmio.  qemu is
> geared to power-of-two accesses even though x86 can generate accesses to
> any number of bytes between 1 and 8.
I just stumbled across the word "hack" in the comment. When the race
occurs the CPU is basically reading from 0xa0000 in an endless loop.
> It appears that this has happened with your guest.  It's not impossible
> that it's genuine.
>
I had a lot to do the last days, but I update our build environment to
Ubuntu LTS 12.04 64-bit Server which is based on Linux 3.2.0. I still
see the issue. If I use the kvm Module provided with the kernel it is
working correctly. If I use kvm-kmod-3.4 with qemu-kvm-1.0.1 (both
from sourceforge) I can reproduce the race condition.

I will keep you posted when I have more evidence.

Thanks,
Peter

Re: [Qemu-devel] race between kvm-kmod-3.0 and kvm-kmod-3.3 // was: race condition in qemu-kvm-1.0.1

[Peter Lieven]

Further output from my testing.

Working:
Linux 2.6.38 with included kvm module
Linux 3.0.0 with included kvm module

Not-Working:
Linux 3.2.0 with included kvm module
Linux 2.6.28 with kvm-kmod 3.4
Linux 3.0.0 with kvm-kmod 3.4
Linux 3.2.0 with kvm-kmod 3.4

I can trigger the race with any of qemu-kvm 0.12.5, 1.0 or 1.0.1.
It might be that the code was introduced somewhere between 3.0.0
and 3.2.0 in the kvm kernel module and that the flaw is not
in qemu-kvm.

Any hints?

Thanks,
Peter


On 02.07.2012 17:05, Avi Kivity wrote:
> On 06/28/2012 12:38 PM, Peter Lieven wrote:
>> does anyone know whats that here in handle_mmio?
>>
>>      /* hack: Red Hat 7.1 generates these weird accesses. */
>>      if ((addr>  0xa0000-4&&  addr<= 0xa0000)&&  kvm_run->mmio.len == 3)
>>          return 0;
>>
> Just what it says.  There is a 4-byte access to address 0x9ffff.  The
> first byte lies in RAM, the next three bytes are in mmio.  qemu is
> geared to power-of-two accesses even though x86 can generate accesses to
> any number of bytes between 1 and 8.
>
> It appears that this has happened with your guest.  It's not impossible
> that it's genuine.
>

Re: [Qemu-devel] race between kvm-kmod-3.0 and kvm-kmod-3.3 // was: race condition in qemu-kvm-1.0.1

[Avi Kivity]

On 07/03/2012 04:01 PM, Peter Lieven wrote:
> Further output from my testing.
> 
> Working:
> Linux 2.6.38 with included kvm module
> Linux 3.0.0 with included kvm module
> 
> Not-Working:
> Linux 3.2.0 with included kvm module
> Linux 2.6.28 with kvm-kmod 3.4
> Linux 3.0.0 with kvm-kmod 3.4
> Linux 3.2.0 with kvm-kmod 3.4
> 
> I can trigger the race with any of qemu-kvm 0.12.5, 1.0 or 1.0.1.
> It might be that the code was introduced somewhere between 3.0.0
> and 3.2.0 in the kvm kernel module and that the flaw is not
> in qemu-kvm.
> 
> Any hints?
> 

A bisect could tell us where the problem is.

To avoid bisecting all of linux, try

   git bisect v3.2 v3.0 virt/kvm arch/x86/kvm



-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] race between kvm-kmod-3.0 and kvm-kmod-3.3 // was: race condition in qemu-kvm-1.0.1

[Peter Lieven]

On 03.07.2012 15:13, Avi Kivity wrote:
> On 07/03/2012 04:01 PM, Peter Lieven wrote:
>> Further output from my testing.
>>
>> Working:
>> Linux 2.6.38 with included kvm module
>> Linux 3.0.0 with included kvm module
>>
>> Not-Working:
>> Linux 3.2.0 with included kvm module
>> Linux 2.6.28 with kvm-kmod 3.4
>> Linux 3.0.0 with kvm-kmod 3.4
>> Linux 3.2.0 with kvm-kmod 3.4
>>
>> I can trigger the race with any of qemu-kvm 0.12.5, 1.0 or 1.0.1.
>> It might be that the code was introduced somewhere between 3.0.0
>> and 3.2.0 in the kvm kernel module and that the flaw is not
>> in qemu-kvm.
>>
>> Any hints?
>>
> A bisect could tell us where the problem is.
>
> To avoid bisecting all of linux, try
>
>     git bisect v3.2 v3.0 virt/kvm arch/x86/kvm
>
>
would it also be ok to bisect kvm-kmod?

thanks,
peter

Re: [Qemu-devel] race between kvm-kmod-3.0 and kvm-kmod-3.3 // was: race condition in qemu-kvm-1.0.1

[Avi Kivity]

On 07/03/2012 04:15 PM, Peter Lieven wrote:
> On 03.07.2012 15:13, Avi Kivity wrote:
>> On 07/03/2012 04:01 PM, Peter Lieven wrote:
>>> Further output from my testing.
>>>
>>> Working:
>>> Linux 2.6.38 with included kvm module
>>> Linux 3.0.0 with included kvm module
>>>
>>> Not-Working:
>>> Linux 3.2.0 with included kvm module
>>> Linux 2.6.28 with kvm-kmod 3.4
>>> Linux 3.0.0 with kvm-kmod 3.4
>>> Linux 3.2.0 with kvm-kmod 3.4
>>>
>>> I can trigger the race with any of qemu-kvm 0.12.5, 1.0 or 1.0.1.
>>> It might be that the code was introduced somewhere between 3.0.0
>>> and 3.2.0 in the kvm kernel module and that the flaw is not
>>> in qemu-kvm.
>>>
>>> Any hints?
>>>
>> A bisect could tell us where the problem is.
>>
>> To avoid bisecting all of linux, try
>>
>>     git bisect v3.2 v3.0 virt/kvm arch/x86/kvm
>>
>>
> would it also be ok to bisect kvm-kmod?

Yes, but note that kvm-kmod is spread across two repositories which are
not often tested out of sync, so you may get build failures.


-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] race between kvm-kmod-3.0 and kvm-kmod-3.3 // was: race condition in qemu-kvm-1.0.1

[Peter Lieven]

On 07/03/12 15:25, Avi Kivity wrote:
> On 07/03/2012 04:15 PM, Peter Lieven wrote:
>> On 03.07.2012 15:13, Avi Kivity wrote:
>>> On 07/03/2012 04:01 PM, Peter Lieven wrote:
>>>> Further output from my testing.
>>>>
>>>> Working:
>>>> Linux 2.6.38 with included kvm module
>>>> Linux 3.0.0 with included kvm module
>>>>
>>>> Not-Working:
>>>> Linux 3.2.0 with included kvm module
>>>> Linux 2.6.28 with kvm-kmod 3.4
>>>> Linux 3.0.0 with kvm-kmod 3.4
>>>> Linux 3.2.0 with kvm-kmod 3.4
>>>>
>>>> I can trigger the race with any of qemu-kvm 0.12.5, 1.0 or 1.0.1.
>>>> It might be that the code was introduced somewhere between 3.0.0
>>>> and 3.2.0 in the kvm kernel module and that the flaw is not
>>>> in qemu-kvm.
>>>>
>>>> Any hints?
>>>>
>>> A bisect could tell us where the problem is.
>>>
>>> To avoid bisecting all of linux, try
>>>
>>>      git bisect v3.2 v3.0 virt/kvm arch/x86/kvm
>>>
>>>
>> would it also be ok to bisect kvm-kmod?
> Yes, but note that kvm-kmod is spread across two repositories which are
> not often tested out of sync, so you may get build failures.
>
ok, i just started with this with a 3.0 (good) and 3.2 (bad) vanilla 
kernel. i can confirm
the bug and i am no starting to bisect. it will take while with my 
equipment if anyone
has a powerful testbed to run this i would greatly appreciate help.

if anyone wants to reproduce:

a) v3.2 from git.kernel.org
b) qemu-kvm 1.0.1 from sourceforge
c) ubuntu 64-bit 12.04 server cd
d) empty (e.g. all zero) hard disk image

cmdline:
./qemu-system-x86_64 -m 512 -cdrom 
/home/lieven/Downloads/ubuntu-12.04-server-amd64.iso -hda 
/dev/hd1/vmtest -vnc :1 -monitor stdio -boot dc

then choose boot from first harddisk and try to quit the qemu monitor 
with 'quit'. -> hypervisor hangs.

peter

Re: [Qemu-devel] race between kvm-kmod-3.0 and kvm-kmod-3.3 // was: race condition in qemu-kvm-1.0.1

[Peter Lieven]

On 07/03/12 15:13, Avi Kivity wrote:
> On 07/03/2012 04:01 PM, Peter Lieven wrote:
>> Further output from my testing.
>>
>> Working:
>> Linux 2.6.38 with included kvm module
>> Linux 3.0.0 with included kvm module
>>
>> Not-Working:
>> Linux 3.2.0 with included kvm module
>> Linux 2.6.28 with kvm-kmod 3.4
>> Linux 3.0.0 with kvm-kmod 3.4
>> Linux 3.2.0 with kvm-kmod 3.4
>>
>> I can trigger the race with any of qemu-kvm 0.12.5, 1.0 or 1.0.1.
>> It might be that the code was introduced somewhere between 3.0.0
>> and 3.2.0 in the kvm kernel module and that the flaw is not
>> in qemu-kvm.
>>
>> Any hints?
>>
> A bisect could tell us where the problem is.
>
> To avoid bisecting all of linux, try
>
>     git bisect v3.2 v3.0 virt/kvm arch/x86/kvm
here we go:

commit ca7d58f375c650cf36900cb1da1ca2cc99b13393
Author: Xiao Guangrong <xiaoguangrong@cn.fujitsu.com>
Date:   Wed Jul 13 14:31:08 2011 +0800

     KVM: x86: fix broken read emulation spans a page boundary

Re: [Qemu-devel] race between kvm-kmod-3.0 and kvm-kmod-3.3 // was: race condition in qemu-kvm-1.0.1

[Xiao Guangrong]

On 07/05/2012 07:12 AM, Peter Lieven wrote:
> On 07/03/12 15:13, Avi Kivity wrote:
>> On 07/03/2012 04:01 PM, Peter Lieven wrote:
>>> Further output from my testing.
>>>
>>> Working:
>>> Linux 2.6.38 with included kvm module
>>> Linux 3.0.0 with included kvm module
>>>
>>> Not-Working:
>>> Linux 3.2.0 with included kvm module
>>> Linux 2.6.28 with kvm-kmod 3.4
>>> Linux 3.0.0 with kvm-kmod 3.4
>>> Linux 3.2.0 with kvm-kmod 3.4
>>>
>>> I can trigger the race with any of qemu-kvm 0.12.5, 1.0 or 1.0.1.
>>> It might be that the code was introduced somewhere between 3.0.0
>>> and 3.2.0 in the kvm kernel module and that the flaw is not
>>> in qemu-kvm.
>>>
>>> Any hints?
>>>
>> A bisect could tell us where the problem is.
>>
>> To avoid bisecting all of linux, try
>>
>>     git bisect v3.2 v3.0 virt/kvm arch/x86/kvm
> here we go:
> 
> commit ca7d58f375c650cf36900cb1da1ca2cc99b13393
> Author: Xiao Guangrong <xiaoguangrong@cn.fujitsu.com>
> Date:   Wed Jul 13 14:31:08 2011 +0800
> 
>     KVM: x86: fix broken read emulation spans a page boundary

Ah, i will try to reproduce it and fix it.

Thanks for your work.

Re: [Qemu-devel] race between kvm-kmod-3.0 and kvm-kmod-3.3 // was: race condition in qemu-kvm-1.0.1

[Xiao Guangrong]

On 06/28/2012 05:11 PM, Peter Lieven wrote:

> that here is bascially whats going on:
> 
>   qemu-kvm-1.0-2506  [010] 60996.908000: kvm_mmio:             mmio read len 3 gpa 0xa0000 val 0x10ff
>     qemu-kvm-1.0-2506  [010] 60996.908000: vcpu_match_mmio:      gva 0xa0000 gpa 0xa0000 Read GPA
>     qemu-kvm-1.0-2506  [010] 60996.908000: kvm_mmio:             mmio unsatisfied-read len 1 gpa 0xa0000 val 0x0
>     qemu-kvm-1.0-2506  [010] 60996.908000: kvm_userspace_exit:   reason KVM_EXIT_MMIO (6)
>     qemu-kvm-1.0-2506  [010] 60996.908000: kvm_mmio:             mmio read len 3 gpa 0xa0000 val 0x10ff
>     qemu-kvm-1.0-2506  [010] 60996.908000: vcpu_match_mmio:      gva 0xa0000 gpa 0xa0000 Read GPA
>     qemu-kvm-1.0-2506  [010] 60996.908000: kvm_mmio:             mmio unsatisfied-read len 1 gpa 0xa0000 val 0x0
>     qemu-kvm-1.0-2506  [010] 60996.908000: kvm_userspace_exit:   reason KVM_EXIT_MMIO (6)
>     qemu-kvm-1.0-2506  [010] 60996.908000: kvm_mmio:             mmio read len 3 gpa 0xa0000 val 0x10ff
>     qemu-kvm-1.0-2506  [010] 60996.908000: vcpu_match_mmio:      gva 0xa0000 gpa 0xa0000 Read GPA
>     qemu-kvm-1.0-2506  [010] 60996.908000: kvm_mmio:             mmio unsatisfied-read len 1 gpa 0xa0000 val 0x0
>     qemu-kvm-1.0-2506  [010] 60996.908000: kvm_userspace_exit:   reason KVM_EXIT_MMIO (6)
>     qemu-kvm-1.0-2506  [010] 60996.908000: kvm_mmio:             mmio read len 3 gpa 0xa0000 val 0x10ff
>     qemu-kvm-1.0-2506  [010] 60996.908000: vcpu_match_mmio:      gva 0xa0000 gpa 0xa0000 Read GPA
>     qemu-kvm-1.0-2506  [010] 60996.908000: kvm_mmio:             mmio unsatisfied-read len 1 gpa 0xa0000 val 0x0
>     qemu-kvm-1.0-2506  [010] 60996.908000: kvm_userspace_exit:   reason KVM_EXIT_MMIO (6)
> 

There are two mmio emulation after user-space-exit, it is caused by mmio
read access which spans two pages. But it should be fixed by:

commit f78146b0f9230765c6315b2e14f56112513389ad
Author: Avi Kivity <avi@redhat.com>
Date:   Wed Apr 18 19:22:47 2012 +0300

    KVM: Fix page-crossing MMIO

    MMIO that are split across a page boundary are currently broken - the
    code does not expect to be aborted by the exit to userspace for the
    first MMIO fragment.

    This patch fixes the problem by generalizing the current code for handling
    16-byte MMIOs to handle a number of "fragments", and changes the MMIO
    code to create those fragments.

    Signed-off-by: Avi Kivity <avi@redhat.com>
    Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

Could you please pull the code from:
https://git.kernel.org/pub/scm/virt/kvm/kvm.git
and trace it again?

Re: [Qemu-devel] race between kvm-kmod-3.0 and kvm-kmod-3.3 // was: race condition in qemu-kvm-1.0.1

[Peter Lieven]

On 05.07.2012 10:51, Xiao Guangrong wrote:
> On 06/28/2012 05:11 PM, Peter Lieven wrote:
>
>> that here is bascially whats going on:
>>
>>    qemu-kvm-1.0-2506  [010] 60996.908000: kvm_mmio:             mmio read len 3 gpa 0xa0000 val 0x10ff
>>      qemu-kvm-1.0-2506  [010] 60996.908000: vcpu_match_mmio:      gva 0xa0000 gpa 0xa0000 Read GPA
>>      qemu-kvm-1.0-2506  [010] 60996.908000: kvm_mmio:             mmio unsatisfied-read len 1 gpa 0xa0000 val 0x0
>>      qemu-kvm-1.0-2506  [010] 60996.908000: kvm_userspace_exit:   reason KVM_EXIT_MMIO (6)
>>      qemu-kvm-1.0-2506  [010] 60996.908000: kvm_mmio:             mmio read len 3 gpa 0xa0000 val 0x10ff
>>      qemu-kvm-1.0-2506  [010] 60996.908000: vcpu_match_mmio:      gva 0xa0000 gpa 0xa0000 Read GPA
>>      qemu-kvm-1.0-2506  [010] 60996.908000: kvm_mmio:             mmio unsatisfied-read len 1 gpa 0xa0000 val 0x0
>>      qemu-kvm-1.0-2506  [010] 60996.908000: kvm_userspace_exit:   reason KVM_EXIT_MMIO (6)
>>      qemu-kvm-1.0-2506  [010] 60996.908000: kvm_mmio:             mmio read len 3 gpa 0xa0000 val 0x10ff
>>      qemu-kvm-1.0-2506  [010] 60996.908000: vcpu_match_mmio:      gva 0xa0000 gpa 0xa0000 Read GPA
>>      qemu-kvm-1.0-2506  [010] 60996.908000: kvm_mmio:             mmio unsatisfied-read len 1 gpa 0xa0000 val 0x0
>>      qemu-kvm-1.0-2506  [010] 60996.908000: kvm_userspace_exit:   reason KVM_EXIT_MMIO (6)
>>      qemu-kvm-1.0-2506  [010] 60996.908000: kvm_mmio:             mmio read len 3 gpa 0xa0000 val 0x10ff
>>      qemu-kvm-1.0-2506  [010] 60996.908000: vcpu_match_mmio:      gva 0xa0000 gpa 0xa0000 Read GPA
>>      qemu-kvm-1.0-2506  [010] 60996.908000: kvm_mmio:             mmio unsatisfied-read len 1 gpa 0xa0000 val 0x0
>>      qemu-kvm-1.0-2506  [010] 60996.908000: kvm_userspace_exit:   reason KVM_EXIT_MMIO (6)
>>
> There are two mmio emulation after user-space-exit, it is caused by mmio
> read access which spans two pages. But it should be fixed by:
>
> commit f78146b0f9230765c6315b2e14f56112513389ad
> Author: Avi Kivity<avi@redhat.com>
> Date:   Wed Apr 18 19:22:47 2012 +0300
>
>      KVM: Fix page-crossing MMIO
>
>      MMIO that are split across a page boundary are currently broken - the
>      code does not expect to be aborted by the exit to userspace for the
>      first MMIO fragment.
>
>      This patch fixes the problem by generalizing the current code for handling
>      16-byte MMIOs to handle a number of "fragments", and changes the MMIO
>      code to create those fragments.
>
>      Signed-off-by: Avi Kivity<avi@redhat.com>
>      Signed-off-by: Marcelo Tosatti<mtosatti@redhat.com>
>
> Could you please pull the code from:
> https://git.kernel.org/pub/scm/virt/kvm/kvm.git
> and trace it again?
Thank you very much, this fixes the issue I have seen.

Thanks,
Peter