Re: KVM 12 and BUG with preempt kernel

Re: KVM 12 and BUG with preempt kernel

[roucaries bastien]

teThis time cc kvm-devel
Problem summary:
My host kernel oops:
Unable to handle kernel paging request at 00000000b7f8143d RIP:
> > [<ffffffff88376d73>] :kvm:paging32_gva_to_gpa+0x43/0x120

On 1/24/07, Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:
> roucaries bastien wrote:
> > Hi,
> >
> > First thank for your nice piece of software kvm.
> > However I have to report some bugs and oops on the host side
> > that seems to be related to preempt:
>
> Please post bugs to kvm-devel or to the sourceforge bug tracker.
>
>
> > -----------------------------------------------------------------------------------------------------------------
> >
> > Unable to handle kernel paging request at 00000000b7f8143d RIP:
> > [<ffffffff88376d73>] :kvm:paging32_gva_to_gpa+0x43/0x120
>
> Does the attached patch fix the bug?

It seems yes (10 boots). Nethertheless I think it is time to report
another bug (present without this patch therefore unrelated).
Unfortunatly I have no trace because it crash my machine and I do not
manage to find it.

If I booot my debian guest with framebuffer activated after 20s to 15
minutes I see on my host screen a cloud of colorfull pixel. These
pixel flash and seems related to guest frame buffer (same colors
scheme but because it not the same resolution I can't read it). About
3s latter my box crash and is dead (cap lock does not even toggles the
keyboard led). Therefore it seems that the guest achieve to write the
host memory :-(, the good point is that he does not poke to my hard
drive registers.

This behavior is 100% reproductible. If I run the host and I switch to
the text console I can't even get a trace because the screen is
corrupted. Unfortunatly I have no serial port on this box (laptop).

>Index: paging_tmpl.h
>===================================================================
>--- paging_tmpl.h       (revision 4320)
>+++ paging_tmpl.h       (working copy)
>@@ -443,31 +443,17 @@
>static gpa_t FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, gva_t vaddr)
> {
        struct guest_walker walker;
-       pt_element_t guest_pte;
-       gpa_t gpa;
+       gpa_t gpa = UNMAPPED_GVA;
+       int r;

-       FNAME(walk_addr)(&walker, vcpu, vaddr, 0, 0, 0);
-       guest_pte = *walker.ptep;
-       FNAME(release_walker)(&walker);
+       r = FNAME(walk_addr)(&walker, vcpu, vaddr, 0, 0, 0);

-       if (!is_present_pte(guest_pte))
-               return UNMAPPED_GVA;
-
-       if (walker.level == PT_DIRECTORY_LEVEL) {
-               ASSERT((guest_pte & PT_PAGE_SIZE_MASK));
-               ASSERT(PTTYPE == 64 || is_pse(vcpu));
-
-               gpa = (guest_pte & PT_DIR_BASE_ADDR_MASK) | (vaddr &
-                       (PT_LEVEL_MASK(PT_PAGE_TABLE_LEVEL) | ~PAGE_MASK));
-
-               if (PTTYPE == 32 && is_cpuid_PSE36())
-                       gpa |= (guest_pte & PT32_DIR_PSE36_MASK) <<
-                                       (32 - PT32_DIR_PSE36_SHIFT);
-       } else {
-               gpa = (guest_pte & PT_BASE_ADDR_MASK);
-               gpa |= (vaddr & ~PAGE_MASK);
+       if (r) {
+               gpa = (gpa_t)walker.gfn << PAGE_SHIFT;
+               gpa |= vaddr & ~PAGE_MASK;
        }

+       FNAME(release_walker)(&walker);
        return gpa;
 }

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: KVM 12 and BUG with preempt kernel

[Avi Kivity]

roucaries bastien wrote:
>
> It seems yes (10 boots). 

Good.  I'll commit the patch.

> Nethertheless I think it is time to report
> another bug (present without this patch therefore unrelated).
> Unfortunatly I have no trace because it crash my machine and I do not
> manage to find it.
>
> If I booot my debian guest with framebuffer activated after 20s to 15
> minutes I see on my host screen a cloud of colorfull pixel. These
> pixel flash and seems related to guest frame buffer (same colors
> scheme but because it not the same resolution I can't read it). About
> 3s latter my box crash and is dead (cap lock does not even toggles the
> keyboard led). Therefore it seems that the guest achieve to write the
> host memory :-(, the good point is that he does not poke to my hard
> drive registers.
>
> This behavior is 100% reproductible. If I run the host and I switch to
> the text console I can't even get a trace because the screen is
> corrupted.


You can start the guest in vnc mode (-vnc 1), so the host can remain in 
text mode, or use netconsole.


Alternatively, please post exact instructions for reproducing.



>> Index: paging_tmpl.h
>> ===================================================================
>> --- paging_tmpl.h       (revision 4320)
>> +++ paging_tmpl.h       (working copy)
>> @@ -443,31 +443,17 @@
>> static gpa_t FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, gva_t vaddr)
>> {
>        struct guest_walker walker;
> -       pt_element_t guest_pte;
> -       gpa_t gpa;
> +       gpa_t gpa = UNMAPPED_GVA;
> +       int r;
>
> -       FNAME(walk_addr)(&walker, vcpu, vaddr, 0, 0, 0);
> -       guest_pte = *walker.ptep;
> -       FNAME(release_walker)(&walker);
> +       r = FNAME(walk_addr)(&walker, vcpu, vaddr, 0, 0, 0);
>
> -       if (!is_present_pte(guest_pte))
> -               return UNMAPPED_GVA;
> -
> -       if (walker.level == PT_DIRECTORY_LEVEL) {
> -               ASSERT((guest_pte & PT_PAGE_SIZE_MASK));
> -               ASSERT(PTTYPE == 64 || is_pse(vcpu));
> -
> -               gpa = (guest_pte & PT_DIR_BASE_ADDR_MASK) | (vaddr &
> -                       (PT_LEVEL_MASK(PT_PAGE_TABLE_LEVEL) | 
> ~PAGE_MASK));
> -
> -               if (PTTYPE == 32 && is_cpuid_PSE36())
> -                       gpa |= (guest_pte & PT32_DIR_PSE36_MASK) <<
> -                                       (32 - PT32_DIR_PSE36_SHIFT);
> -       } else {
> -               gpa = (guest_pte & PT_BASE_ADDR_MASK);
> -               gpa |= (vaddr & ~PAGE_MASK);
> +       if (r) {
> +               gpa = (gpa_t)walker.gfn << PAGE_SHIFT;
> +               gpa |= vaddr & ~PAGE_MASK;
>        }
>
> +       FNAME(release_walker)(&walker);
>        return gpa;
> }


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: KVM 12 and BUG with preempt kernel

[roucaries bastien]

On 1/24/07, Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:
> roucaries bastien wrote:
> >
> > It seems yes (10 boots).
>
> Good.  I'll commit the patch.
>
> > Nethertheless I think it is time to report
> > another bug (present without this patch therefore unrelated).
> > Unfortunatly I have no trace because it crash my machine and I do not
> > manage to find it.
> >
> > If I booot my debian guest with framebuffer activated after 20s to 15
> > minutes I see on my host screen a cloud of colorfull pixel. These
> > pixel flash and seems related to guest frame buffer (same colors
> > scheme but because it not the same resolution I can't read it). About
> > 3s latter my box crash and is dead (cap lock does not even toggles the
> > keyboard led). Therefore it seems that the guest achieve to write the
> > host memory :-(, the good point is that he does not poke to my hard
> > drive registers.
> >
> > This behavior is 100% reproductible. If I run the host and I switch to
> > the text console I can't even get a trace because the screen is
> > corrupted.
>
>
> You can start the guest in vnc mode (-vnc 1), so the host can remain in
> text mode, or use netconsole.
The problem is not switching between x and console but read the oops
because kvm write to video mem :-(. And I do not think netconsole
could help because my computer does not answer to irq, I suspect
something nasty. The only think that could help is nmi watchdog but
unfortunatly the bios declare timer interupt as XT-pic and not io-apic
(bios bug forwaded to asus), therefore no nmi.

Moreover I try vnc switch and kvm is stable, stange isn't.

To reproduce:
- use an preempt host kernel on amd64, kvm compiled as x86_64
- install a testing debian on a guest using sdl display, use
framebuffer install (default)
- wait....
- when you see colorfull pixel on the top of your sceen, kill kvm
(kill only accelerate the crash)
- wait 1s
- kernel will crash

I suspect I am seeing this colorfull pixel because I use a vesa driver
on X and the guest write to it. But I do not understand why because
lspci of guest say:
00:02.0 VGA compatible controller: Cirrus Logic GD 5446 (prog-if 00 [VGA])
        Flags: bus master, fast devsel, latency 0
        Memory at e0000000 (32-bit, prefetchable) [size=32M]
        Memory at e2000000 (32-bit, non-prefetchable) [size=4K]
and host:
02:00.0 VGA compatible controller: nVidia Corporation G70 [GeForce Go
7600] (rev a1) (prog-if 00 [VGA])
        Subsystem: ASUSTeK Computer Inc. Unknown device 1322
        Flags: bus master, fast devsel, latency 0, IRQ 11
        Memory at de000000 (32-bit, non-prefetchable) [size=16M]
        Memory at c0000000 (64-bit, prefetchable) [size=256M]
        Memory at dd000000 (64-bit, non-prefetchable) [size=16M]

Memory resources do not overlap and other guest device are pio driven.
I do not understand this kind of crash. Any idea?

Thank you for your time

bastien

>
> Alternatively, please post exact instructions for reproducing.
>
>
>
> >> Index: paging_tmpl.h
> >> ===================================================================
> >> --- paging_tmpl.h       (revision 4320)
> >> +++ paging_tmpl.h       (working copy)
> >> @@ -443,31 +443,17 @@
> >> static gpa_t FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, gva_t vaddr)
> >> {
> >        struct guest_walker walker;
> > -       pt_element_t guest_pte;
> > -       gpa_t gpa;
> > +       gpa_t gpa = UNMAPPED_GVA;
> > +       int r;
> >
> > -       FNAME(walk_addr)(&walker, vcpu, vaddr, 0, 0, 0);
> > -       guest_pte = *walker.ptep;
> > -       FNAME(release_walker)(&walker);
> > +       r = FNAME(walk_addr)(&walker, vcpu, vaddr, 0, 0, 0);
> >
> > -       if (!is_present_pte(guest_pte))
> > -               return UNMAPPED_GVA;
> > -
> > -       if (walker.level == PT_DIRECTORY_LEVEL) {
> > -               ASSERT((guest_pte & PT_PAGE_SIZE_MASK));
> > -               ASSERT(PTTYPE == 64 || is_pse(vcpu));
> > -
> > -               gpa = (guest_pte & PT_DIR_BASE_ADDR_MASK) | (vaddr &
> > -                       (PT_LEVEL_MASK(PT_PAGE_TABLE_LEVEL) |
> > ~PAGE_MASK));
> > -
> > -               if (PTTYPE == 32 && is_cpuid_PSE36())
> > -                       gpa |= (guest_pte & PT32_DIR_PSE36_MASK) <<
> > -                                       (32 - PT32_DIR_PSE36_SHIFT);
> > -       } else {
> > -               gpa = (guest_pte & PT_BASE_ADDR_MASK);
> > -               gpa |= (vaddr & ~PAGE_MASK);
> > +       if (r) {
> > +               gpa = (gpa_t)walker.gfn << PAGE_SHIFT;
> > +               gpa |= vaddr & ~PAGE_MASK;
> >        }
> >
> > +       FNAME(release_walker)(&walker);
> >        return gpa;
> > }
>
>
> --
> error compiling committee.c: too many arguments to function
>
>

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: KVM 12 and BUG with preempt kernel

[Avi Kivity]

roucaries bastien wrote:
>>
>> > Nethertheless I think it is time to report
>> > another bug (present without this patch therefore unrelated).
>> > Unfortunatly I have no trace because it crash my machine and I do not
>> > manage to find it.
>> >
>> > If I booot my debian guest with framebuffer activated after 20s to 15
>> > minutes I see on my host screen a cloud of colorfull pixel. These
>> > pixel flash and seems related to guest frame buffer (same colors
>> > scheme but because it not the same resolution I can't read it). About
>> > 3s latter my box crash and is dead (cap lock does not even toggles the
>> > keyboard led). Therefore it seems that the guest achieve to write the
>> > host memory :-(, the good point is that he does not poke to my hard
>> > drive registers.
>> >
>> > This behavior is 100% reproductible. If I run the host and I switch to
>> > the text console I can't even get a trace because the screen is
>> > corrupted.
>>
>>
>> You can start the guest in vnc mode (-vnc 1), so the host can remain in
>> text mode, or use netconsole.
> The problem is not switching between x and console but read the oops
> because kvm write to video mem :-(. And I do not think netconsole
> could help because my computer does not answer to irq, I suspect
> something nasty. The only think that could help is nmi watchdog but
> unfortunatly the bios declare timer interupt as XT-pic and not io-apic
> (bios bug forwaded to asus), therefore no nmi.
>
> Moreover I try vnc switch and kvm is stable, stange isn't.
>
> To reproduce:
> - use an preempt host kernel on amd64, kvm compiled as x86_64

Which host kernel version are you using?

> - install a testing debian on a guest using sdl display, use
> framebuffer install (default)

Please provide a link to the installer so I can try it too.

> - wait....
> - when you see colorfull pixel on the top of your sceen, kill kvm
> (kill only accelerate the crash)
> - wait 1s
> - kernel will crash
>
> I suspect I am seeing this colorfull pixel because I use a vesa driver
> on X and the guest write to it.

No, the guest cannot write directly to host memory.  It writes to a 
virtualized framebuffer.



-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: KVM 12 and BUG with preempt kernel

[roucaries bastien]

On 1/25/07, Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:
> roucaries bastien wrote:
> > On 1/24/07, roucaries bastien <roucaries.bastien-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >> On 1/24/07, Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:
> >> > roucaries bastien wrote:
> >> > >>
> >
> >>
> >> PS2: -vnc flag seems to be a good workarround, 4 hours without any
> >> crash.
> >>
> > After 4 hours and a half it crach with -vnc only a little bit harder
> > to get.
> >
>
> I tried installing debian testing on an amd machine.  So far everything
> works.
>
> You'll need to give more details about your machine or about how to
> reproduce this if you want progress.

Ok using all the debugs option on a new kernel I achieve to get this
during the problem. I think it is a non sense but...
BUG: soft lockup detected on CPU#1!

Call Trace:
 <IRQ>  [<ffffffff802c28a9>] softlockup_tick+0xf9/0x140
 [<ffffffff8029d167>] update_process_times+0x57/0x90
 [<ffffffff8027dda4>] smp_local_timer_interrupt+0x34/0x60
 [<ffffffff8027e56f>] smp_apic_timer_interrupt+0x5f/0x80
 [<ffffffff802712f5>] default_idle+0x35/0x60
 [<ffffffff802712c0>] default_idle+0x0/0x60
 [<ffffffff80265d8b>] apic_timer_interrupt+0x6b/0x70
 <EOI>  [<ffffffff802712f5>] default_idle+0x35/0x60
 [<ffffffff802712f7>] default_idle+0x37/0x60
 [<ffffffff802712f5>] default_idle+0x35/0x60
 [<ffffffff8024f952>] cpu_idle+0xa2/0xe0
 [<ffffffff8027db73>] start_secondary+0x4d3/0x4f0

And I achieve to recover my system killing X. Perhaps an interruption
problem. Using full debug the hang disapear strange isn't (smell like
a race condition).

Regards bastien
> --
> error compiling committee.c: too many arguments to function
>
>

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: KVM 12 and BUG with preempt kernel

[roucaries bastien]

On 1/26/07, roucaries bastien <roucaries.bastien-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On 1/25/07, Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:
> > roucaries bastien wrote:
> > > On 1/24/07, roucaries bastien <roucaries.bastien-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > >> On 1/24/07, Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote:
> > >> > roucaries bastien wrote:
> > >> > >>
> > >
> > >>
> > >> PS2: -vnc flag seems to be a good workarround, 4 hours without any
> > >> crash.
> > >>
> > > After 4 hours and a half it crach with -vnc only a little bit harder
> > > to get.
> > >
> >
> > I tried installing debian testing on an amd machine.  So far everything
> > works.
> >
> > You'll need to give more details about your machine or about how to
> > reproduce this if you want progress.
>
> Ok using all the debugs option on a new kernel I achieve to get this
> during the problem. I think it is a non sense but...
> BUG: soft lockup detected on CPU#1!
>
> Call Trace:
>  <IRQ>  [<ffffffff802c28a9>] softlockup_tick+0xf9/0x140
>  [<ffffffff8029d167>] update_process_times+0x57/0x90
>  [<ffffffff8027dda4>] smp_local_timer_interrupt+0x34/0x60
>  [<ffffffff8027e56f>] smp_apic_timer_interrupt+0x5f/0x80
>  [<ffffffff802712f5>] default_idle+0x35/0x60
>  [<ffffffff802712c0>] default_idle+0x0/0x60
>  [<ffffffff80265d8b>] apic_timer_interrupt+0x6b/0x70
>  <EOI>  [<ffffffff802712f5>] default_idle+0x35/0x60
>  [<ffffffff802712f7>] default_idle+0x37/0x60
>  [<ffffffff802712f5>] default_idle+0x35/0x60
>  [<ffffffff8024f952>] cpu_idle+0xa2/0xe0
>  [<ffffffff8027db73>] start_secondary+0x4d3/0x4f0
>
> And I achieve to recover my system killing X. Perhaps an interruption
> problem. Using full debug the hang disapear strange isn't (smell like
> a race condition).

Sorry to answer myself this time with full debug only the geust freeze
during harwware access any idea in order to debug?


> Regards bastien
> > --
> > error compiling committee.c: too many arguments to function
> >
> >
>

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV