[Bug 65561] New: KVM:Entry failed on Single stepping sti instruction

[Bug 65561] New: KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

            Bug ID: 65561
           Summary: KVM:Entry failed on Single stepping sti instruction
           Product: Virtualization
           Version: unspecified
    Kernel Version: 3.8.8
          Hardware: x86-64
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: high
          Priority: P1
         Component: kvm
          Assignee: virtualization_kvm@kernel-bugs.osdl.org
          Reporter: jatin.iitdelhi@gmail.com
        Regression: No

Hello, I am running a code in which I need to single step some section of code
running in a qemu(+kvm) emulated virtual machine with. The vm hangs everytime I
try to single step the sti instruction. Exact error message is:

KVM: entry failed, hardware error 0x80000021

If you're running a guest on an Intel machine without unrestricted mode
support, the failure can be most likely due to the guest entering an invalid
state for Intel VT. For example, the guest maybe running in big real mode
which is not supported on less recent Intel processors.

EAX=00000001 EBX=00000001 ECX=00000000 EDX=00000050
ESI=00000000 EDI=0000038f EBP=c000edc8 ESP=c000ed90
EIP=c0022f85 EFL=00000302 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
SS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
FS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
GS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0028 c0107000 00000067 00408b00 DPL=0 TSS32-busy
GDT=     c004ae20 0000002f
IDT=     c00495e0 000007ff
CR0=80010015 CR2=00000000 CR3=00101000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000 
DR6=00000000ffff4ff0 DR7=0000000000000400
EFER=0000000000000000
Code=44 24 04 65 00 00 00 c7 04 24 78 c9 03 c0 e8 78 fc 00 00 fb <8b> 45 f4 c9
c3 55 89 e5 83 ec 10 e8 6d ff ff ff 89 45 fc fa 8b 45 fc c9 c3 55 89 e5 83 ec

The problem is reproducible and I am hitting it everytime I try to run it.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

Jidong Xiao <jidong.xiao@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jidong.xiao@gmail.com

--- Comment #1 from Jidong Xiao <jidong.xiao@gmail.com> ---
Hi, I think this has been fixed since kernel 3.11, can you try any newer
version, from 3.11 to 3.14, and see if you can still reproduce the problem? If
not, then we can close this ticket.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #2 from Jatin Kumar <jatin.iitdelhi@gmail.com> ---
Hello Jidong, thanks for the info. I will try and let you know.

While you are at this, can you please help me another single stepping issue and
the issue is:
1. While single stepping, the instruction immediately next to `out` (EE)
instruction is skipped. I am not getting debug trap after the execution of
`out` instruction completes.

2. Also is there anything special about `out` instruction that causes
interrupts to come at instruction immediately next to it, when not running in
single stepping mode.

That would be a really great help in my project.

Thanks in advance.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

Paolo Bonzini <bonzini@gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bonzini@gnu.org

--- Comment #3 from Paolo Bonzini <bonzini@gnu.org> ---
> 2. Also is there anything special about `out` instruction that causes 
> interrupts to come at instruction immediately next to it, when not running in 
> single stepping mode.

The "out" instruction exits to QEMU and has a much higher latency than an
bare-metal.  This, plus the usage of mutexes in QEMU, may make it more likely
that interrupts occur right after an "out" instruction.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #4 from Jidong Xiao <jidong.xiao@gmail.com> ---
Jatin, are you using gdb to do the single step?(In reply to Jatin Kumar from
comment #2)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #5 from Jatin Kumar <jatin.iitdelhi@gmail.com> ---
@Paolo: Thanks for the info. That makes sense to me.

(In reply to Jidong Xiao from comment #4)
> Jatin, are you using gdb to do the single step?(In reply to Jatin Kumar from
> comment #2)

No. I have been doing single stepping from inside the OS. It single steps its
own code using debug trap untill some condition is met. During this whenever I
hit an out instruction, the very next instruction is not skipped while single
stepping. It may have been executing it because overall the code seems to be
working correctly.
I couldn't find anything online in this regard so I am curious if this is
something QEMU specific.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #6 from Jatin Kumar <jatin.iitdelhi@gmail.com> ---
(In reply to Jatin Kumar from comment #5)

> whenever I hit an out instruction, the very next instruction is not skipped

Sorry I meant 'is skipped'.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #7 from Paolo Bonzini <bonzini@gnu.org> ---
Hi Jidong, no this is not fixed yet.  Basically OUT instructions are emulated
by KVM, and support for single-stepping and breakpoints in the emulator is
quite minimal.

3.12 added some support but OUTs and writes to memory are still broken:

commit 0912c9771e9902f752e890e93af495cc06a786ac
Author: Paolo Bonzini <pbonzini@redhat.com>
Date:   Tue Aug 27 15:41:43 2013 +0200

    KVM: x86: add comments where MMIO does not return to the emulator

    Support for single-step in the emulator (new in 3.12) does not work for
    MMIO or PIO writes, because they are completed without returning to
    the emulator.  This is not worse than what we had in 3.11; still, add
    comments so that the issue is not forgotten.

    Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
    Signed-off-by: Gleb Natapov <gleb@redhat.com>

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #8 from Jidong Xiao <jidong.xiao@gmail.com> ---
Hi, Jatin,

"from inside the OS" means from within the Guest OS right?
(In reply to Jatin Kumar from comment #5)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #9 from Jidong Xiao <jidong.xiao@gmail.com> ---
Hi, Paolo,

It seems that Gleb's patch 

commit 03617c188f41eeeb4223c919ee7e66e5a114f2c6
"KVM: VMX: mark unusable segment as nonpresent"

fixed a similar problem like this, look at this:
https://bugzilla.redhat.com/show_bug.cgi?id=854983,

Even though they are triggered by different reasons, they both cause the
"hardware error 0x80000021" issue. So I feel that patch could also fix this
single step issue.
(In reply to Paolo Bonzini from comment #7)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #10 from Paolo Bonzini <bonzini@gnu.org> ---
No, hardware error 0x80000021 includes pretty much everything that could go
wrong in vmx.c. :)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #11 from Jidong Xiao <jidong.xiao@gmail.com> ---
Hi, Paolo, thanks for your explanation. I am interested in fixing this. So
what's the technical challenge here?

When you say "support for single-stepping and breakpoints in the emulator is
quite minimal", do you mean it's a problem in the Qemu side, rather than
something wrong in the kvm kernel modules? If so, and if we want to fix this,
we need to make some changes in the Qemu code, right?
(In reply to Paolo Bonzini from comment #10)
> No, hardware error 0x80000021 includes pretty much everything that could go
> wrong in vmx.c. :)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #12 from Paolo Bonzini <bonzini@gnu.org> ---
> Hi, Paolo, thanks for your explanation. I am interested in fixing this. So
> what's the technical challenge here?

The first step is to reproduce the failure.  To do this you can make a 
patch to kvm-unit-tests 
(git://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git).

0x80000021 means invalid guest state in the VMCS (the VM information 
that is passed to the processor).  So you could try adding some printk 
to understand what is the invalid guest state.  To do this you need to 
download the Intel manuals (known as "Intel SDM", google is your 
friend).  You can post the testcase patch on kvm@vger.kernel.org once 
you have a reproducer.

> When you say "support for single-stepping and breakpoints in the emulator is
> quite minimal", do you mean it's a problem in the Qemu side, rather than
> something wrong in the kvm kernel modules? If so, and if we want to fix this,
> we need to make some changes in the Qemu code, right?

No, the emulator is part of KVM, see arch/x86/kvm/emulate.c.  I have 
some pending patches for it, destined to 3.17.  There is a small chance 
that they fix the bug.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #13 from Jidong Xiao <jidong.xiao@gmail.com> ---
Great, I will try. Yes I have the inter manual and I have studied it for a
while, so basically I know the data structure of VMCS. To reproduce the
failure, so, if I use gdb in a guest OS, like to debug a program inside the
Guest OS, and run the single step command in gdb, that should trigger this bug
right? Assuming the program includes a sti instruction. (In reply to Paolo
Bonzini from comment #12)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #14 from Jatin Kumar <jatin.iitdelhi@gmail.com> ---
(In reply to Jidong Xiao from comment #8)
> Hi, Jatin,
> 
> "from inside the OS" means from within the Guest OS right?
> (In reply to Jatin Kumar from comment #5)

Hello Jidong,
Yes it means from within guest OS. Sample code block is like this:

  intr_register_int(1, 3, INTR_OFF, intr_debug_handler_out, "#DB Debug
Exception");
  asm volatile(
    "pushfl \n\t"
    "orl $0x100, (%%esp) \n\t"
    "popfl \n\t"
    "nop \n\t"
    "nop \n\t"
    "outb %b0, %w1 \n\t"
    "nop \n\t"     <------- Not getting Debug Trap before executing this
    "nop \n\t"
    "pushfl \n\t"
    "xorl $0x100, (%%esp) \n\t"
    "popfl \n\t"
    :: "a"(2), "Nd" (80)
  );

Debug handler simply prints the EIP from interrupt frame and I don't see the
EIP of marked instruction.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #15 from Paolo Bonzini <bonzini@gnu.org> ---
> if I use gdb in a guest OS, like to debug a program inside the Guest OS, and 
> run the single step command in gdb, that should trigger this bug right

Yes.  For kvm-unit-tests you would modify x86/debug.c, which already tests
singlestep.  Jatin, can you confirm that _any_ single-stepping over sti
triggers the bug?  Does it matter if IF=0 or IF=1 before the sti?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #16 from Jidong Xiao <jidong.xiao@gmail.com> ---
Alright, thank you Paolo, I will try and let you know once I am done.(In reply
to Paolo Bonzini from comment #15)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #17 from Jidong Xiao <jidong.xiao@gmail.com> ---
Hi,Paolo,

I am not familiar with kvm-unit-tests, and I cannot find any documents describe
it. So I use gdb to debug.

And yes I can reproduce the problem. I just use Jatin's sample code to
construct a c program, like this:

linux:~/code/cvedr # cat ss.c
#include <stdio.h>

main(){
asm __volatile__(
    "pushfl \n\t"
    "orl $0x100, (%%esp) \n\t"
    "popfl \n\t"
    "nop \n\t"
    "nop \n\t"
    "outb %b0, %w1 \n\t"
    "nop \n\t"
    "nop \n\t"
    "pushfl \n\t"
    "xorl $0x100, (%%esp) \n\t"
    "popfl \n\t"
    :: "a"(2), "Nd" (80)
  );
}

And I ran the c program with gdb.
(gdb) disas main
Dump of assembler code for function main:
   0x080483e4 <+0>:     push   %ebp
   0x080483e5 <+1>:     mov    %esp,%ebp
   0x080483e7 <+3>:     mov    $0x2,%eax
   0x080483ec <+8>:     pushf
   0x080483ed <+9>:     orl    $0x100,(%esp)
   0x080483f4 <+16>:    popf
   0x080483f5 <+17>:    nop
   0x080483f6 <+18>:    nop
=> 0x080483f7 <+19>:    out    %al,$0x50
   0x080483f9 <+21>:    nop
   0x080483fa <+22>:    nop
   0x080483fb <+23>:    pushf
   0x080483fc <+24>:    xorl   $0x100,(%esp)
   0x08048403 <+31>:    popf
   0x08048404 <+32>:    pop    %ebp
   0x08048405 <+33>:    ret
End of assembler dump.
(gdb) nexti

Program received signal SIGSEGV, Segmentation fault.
0x080483f7 in main ()
(gdb)

You can see when the program counter points to the out instruction, and I used
nexti command to do single step execution, and it ends up a segmentation fault.
According to the kvm code, it seems that x86_emulate_insn() will be called,
indeed I don't see any code in that function takes care of the out instruction.
But why this only affect single step execution? I actually have some other
program that includes some out instructions, and the program runs okay.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #18 from Jidong Xiao <jidong.xiao@gmail.com> ---
Hi, Paolo,

I am still reading the source code so as to understand the root cause of this
problem. In particular, I added some printk statements in x86_emulate_insn().
For example, I added the following printk statement.

...
        switch (c->b) {
        case 0x00 ... 0x05:
                printk(KERN_ERR "DEBUG: Passed %s %d
\n",__FUNCTION__,__LINE__);
              add:              /* add */
                emulate_2op_SrcV("add", c->src, c->dst, ctxt->eflags);
                break;
...

And then in the guest OS, I run a c program like this:

linux:~/code/testc # cat test.c
#include <stdio.h>

int main(void)
{
        int foo = 10, bar = 15;
        __asm__ __volatile__("add  %%ebx,%%eax"
                             :"=a"(foo)
                             :"a"(foo), "b"(bar)
                             );
        printf("foo+bar=%d\n", foo);
        return 0;
}

I thought that because I am using the "add" instruction, the printk statement
in the host os kernel should be invoked, however, my experimental result prove
it's untrue. So I am totally confused with this kvm emulator.

If the kvm trying to emulate these instructions, why that, when I run these
instructions in the guest, the corresponding emulate code are not invoked? So
how to trigger this printk statement? Thanks.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #19 from Jatin Kumar <jatin.iitdelhi@gmail.com> ---
I am not sure if I am correct but `add` doesn't look like any privileged or
special instruction so if running with KVM it should run directly on the
hardware and without needing emulation. You can try some privilege instructions
like wrmsr, rdmsr etc.
Please correct me if I am wrong.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #20 from Jidong Xiao <jidong.xiao@gmail.com> ---
Thanks Jatin, I actually tried some other instructions, like accessing cr
register. I use the following program:

linux:~/code/cvedr # cat getcr.c
#include <stdio.h>

main(){

        asm __volatile__ (
                "mov %cr0, %eax\n"
                "mov %cr3, %ebx\n"
        );
        printf("test cr\n");
}

In x86_emulate_insn(), there is a piece of code like this (the printk is added
by me.)
==================================
        case 0x20: /* mov cr, reg */
                printk(KERN_ERR "DEBUG: Passed %s %d
\n",__FUNCTION__,__LINE__);
                if (c->modrm_mod != 3)
                        goto cannot_emulate;
                c->regs[c->modrm_rm] =
                                realmode_get_cr(ctxt->vcpu, c->modrm_reg);
                c->dst.type = OP_NONE;  /* no writeback */
                break;
==================================
My results show that the above c program does not trigger this printk statement
neither. I assume accessing cr should be a privileged operation, right? So I am
really curious that how to trigger these printk statements?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #21 from Jatin Kumar <jatin.iitdelhi@gmail.com> ---
The way I would approach this problem is to first check if there is a VM_EXIT
while executing this instruction. You should be able to find that in
vmx_vcpu_run() function (I guess in KVM code). Put a printk there and print the
instruction as well which caused VM_EXIT.

Futher you should try to investigate if the control flow is trying to emulate
this instruction and then your printk's should be triggered.

CAUTION: I haven't spent much time digging KVM code, but I am confident about
the VM_EXIT part. Still after putting in some effort, you may want to wait for
Paolo's reply :)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #22 from Jidong Xiao <jidong.xiao@gmail.com> ---
Alright, I added a printk statement in the wrmsr case, like this:

        case 0x30:
                printk(KERN_ERR "DEBUG: Passed %s %d
\n",__FUNCTION__,__LINE__);
                /* wrmsr */
                msr_data = (u32)c->regs[VCPU_REGS_RAX]
                        | ((u64)c->regs[VCPU_REGS_RDX] << 32);
                rc = kvm_set_msr(ctxt->vcpu, c->regs[VCPU_REGS_RCX], msr_data);
                if (rc) {
                        kvm_inject_gp(ctxt->vcpu, 0);
                        c->eip = kvm_rip_read(ctxt->vcpu);
                }
                rc = X86EMUL_CONTINUE;
                c->dst.type = OP_NONE;
                break;

And i run a c program to access msr, like this:

linux:~/code/cvedr # cat accessmsr.c
#include <stdio.h>

main(){
        asm volatile("wrmsr");
        printf("test msr\n");
}

linux:~/code/cvedr # ./accessmsr
Segmentation fault

Even though my c program in the guest os got a segmentation fault, from the
host os level, I don't see the printk statement is triggered, so this is not
about privileged instructions or unprivileged instructions. I think there must
be some other conditions to trigger these printk statements.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #23 from Jidong Xiao <jidong.xiao@gmail.com> ---
Jatin, do you mean that if one instruction triggers a VM_EXIT, then it won't
trigger the emulate code? So why the emulate code tries to emulate those
privileged instructions, like wrmsr, rdmsr, mov cr to reg, mov dr to reg, I
think these instructions will trigger VM_EXIT anyway, but we see the kvm
emulator still attempt to emulate them.
(In reply to Jatin Kumar from comment #21)
> The way I would approach this problem is to first check if there is a
> VM_EXIT while executing this instruction. You should be able to find that in
> vmx_vcpu_run() function (I guess in KVM code). Put a printk there and print
> the instruction as well which caused VM_EXIT.
> 
> Futher you should try to investigate if the control flow is trying to
> emulate this instruction and then your printk's should be triggered.
> 
> CAUTION: I haven't spent much time digging KVM code, but I am confident
> about the VM_EXIT part. Still after putting in some effort, you may want to
> wait for Paolo's reply :)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #24 from Jatin Kumar <jatin.iitdelhi@gmail.com> ---
> linux:~/code/cvedr # ./accessmsr
> Segmentation fault
You got a seg fault because it is privileged instruction and I run it in kernel
mode not user mode. But you are likely to be running your program in user mode.
Try something like out instruction.

> Even though my c program in the guest os got a segmentation fault, from the
> host os level, I don't see the printk statement is triggered, so this is not
> about privileged instructions or unprivileged instructions. I think there
> must be some other conditions to trigger these printk statements.
After getting the VM_EXIT it must be first checking if it requires to emulate
or should generate a fault. In this case it would have generated the fault
without emulating it.

Again, you can try it with 'out' instruction.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #25 from Jidong Xiao <jidong.xiao@gmail.com> ---
(In reply to Jatin Kumar from comment #24)
> > linux:~/code/cvedr # ./accessmsr
> > Segmentation fault
> You got a seg fault because it is privileged instruction and I run it in
> kernel mode not user mode. But you are likely to be running your program in
> user mode. Try something like out instruction.
>  
> > Even though my c program in the guest os got a segmentation fault, from the
> > host os level, I don't see the printk statement is triggered, so this is not
> > about privileged instructions or unprivileged instructions. I think there
> > must be some other conditions to trigger these printk statements.
> After getting the VM_EXIT it must be first checking if it requires to
> emulate or should generate a fault. In this case it would have generated the
> fault without emulating it.
> 
> Again, you can try it with 'out' instruction.

Alright, Jatin, I wrote a kernel module, like this:

linux:~/code/module # cat testout.c
#include <linux/module.h>
#include <linux/kernel.h>

int init_module(void)
{
        printk(KERN_ALERT "Hello world 1.\n");
        asm __volatile__("outb %al, %dx \n");

   return 0;
}


void cleanup_module(void)
{
  printk(KERN_ALERT "Goodbye world 1.\n");
}

I compiled this module and installed with `insmod testout.ko`, and nothing
happens. Noted that the emulator code includes the following lines:

        case 0xee: /* out al,dx */
        case 0xef: /* out (e/r)ax,dx */
                printk(KERN_ERR "DEBUG: Passed %s %d
\n",__FUNCTION__,__LINE__);
                port = c->regs[VCPU_REGS_RDX];
                io_dir_in = 0;

I added that printk statement, and still, when I was loading the kernel module
in the guest OS, this printk statement was not invoked.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #26 from Jatin Kumar <jatin.iitdelhi@gmail.com> ---
In arch/x86/kvm/vmx.c file, there is a complete list of kvm_vmx_exit_handlers,
you can find there appropriate handlers for each of the exit reasons. There is
a handle_io() handler. Can you try adding a printk to this. I am not sure but
it may be going to x86_emulate_instruction() which then calls
x86_emulate_insn() if required.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #27 from Jidong Xiao <jidong.xiao@gmail.com> ---
Jatin, after I have added printk in handle_io, it looks like the printk will be
called very frequently, I don't need to run any own program in the guest OS,
the printk message in handle_io() is kept showing and never stops. Therefore I
cannot tell whether a out instruction would also trigger the emulator or not.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #28 from Paolo Bonzini <bonzini@gnu.org> ---
As Jatin said, most of the time the emulator is not invoked for instructions
that require intervention of the hypervisor.  Instead, the processor provides
enough "pre-decoded" information about the instruction, so KVM doesn't need to
do it all again in the emulator.

Instructions that invoke the emulator are those that invoke ports or
memory-mapped devices.  However "out" is a fast path, so it doesn't invoke the
emulator.  You see it all the time because the kernel uses it to interact with
many devices (such as the keyboard controller, disk controller or network
cards).

I suggest that you try reproducing the problem with "sti".   The one with "out"
is known and documented in the source code.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #29 from Jidong Xiao <jidong.xiao@gmail.com> ---
Okay, I will try the sti instruction.

Jatin, since your title says that there is something wrong with the sti
instruction, but looking at your sample code, it does not include a sti
instruction, and all you were talking about is the "out" instruction. So you
are reporting two different problems or it's just one problem?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #30 from Jidong Xiao <jidong.xiao@gmail.com> ---
Jatin,

I have the following program and I did test the single step for sti command,

(gdb) disas main
Dump of assembler code for function main:
   0x08048414 <+0>:     push   %ebp
   0x08048415 <+1>:     mov    %esp,%ebp
   0x08048417 <+3>:     and    $0xfffffff0,%esp
   0x0804841a <+6>:     sub    $0x10,%esp
=> 0x0804841d <+9>:     sti
   0x0804841e <+10>:    movl   $0x8048500,(%esp)
   0x08048425 <+17>:    call   0x8048350 <puts@plt>
   0x0804842a <+22>:    leave
   0x0804842b <+23>:    ret
End of assembler dump.
(gdb) nexti

Program received signal SIGSEGV, Segmentation fault.
0x0804841d in main ()
(gdb)

And what I have got is a segmentation fault. Since you said you triggered the
bug via a kernel module, how did you do the single step test for a kernel
module?

In the meanwhile I am studying kvm-unit-test, since Paolo said it provides a
single step test platform.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #31 from Jidong Xiao <jidong.xiao@gmail.com> ---
Hi, Paolo,

I added a sti instruction in kvm-unit-tests:x86/debug.c, like this:
        asm volatile(
                "pushf\n\t"
                "pop %%rax\n\t"
                "sti\n\t"
                "or $(1<<8),%%rax\n\t"
                "push %%rax\n\t"
                "lea (%%rip),%0\n\t"
                "popf\n\t"
                "and $~(1<<8),%%rax\n\t"
                "push %%rax\n\t"
                "popf\n\t"
                : "=g" (start) : : "rax");
        report("single step",
               n == 3 &&
               bp_addr[0] == start+1+6 && dr6[0] == 0xffff4ff0 &&
               bp_addr[1] == start+1+6+1 && dr6[1] == 0xffff4ff0 &&
               bp_addr[2] == start+1+6+1+1 && dr6[2] == 0xffff4ff0);

And then make, and run the test, it looks like everything is okay.
linux:~/code/kvm-unit-tests # /usr/local2/bin/qemu-system-x86_64 -enable-kvm
-device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4
-kernel ./x86/debug.flat
enabling apic
PASS: #BP
PASS: hw breakpoint
PASS: single step
PASS: hw watchpoint


I also tried to add a cli before sti, and the test also passed.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #32 from Jatin Kumar <jatin.iitdelhi@gmail.com> ---
(In reply to Jidong Xiao from comment #29)
> Okay, I will try the sti instruction.
> 
> Jatin, since your title says that there is something wrong with the sti
> instruction, but looking at your sample code, it does not include a sti
> instruction, and all you were talking about is the "out" instruction. So you
> are reporting two different problems or it's just one problem?

Hello Jidong,
I am sorry for the confusion. Yes these are two independent problems.
1. Single stepping an STI instruction (which as you said is already fixed)
2. Single stepping an OUT instruction, which works but causes the next
instruction to be executed without debug trap.

I am developing a small OS which runs in kernel mode most of the time and I
need to single step the code to a particular point in execution.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #33 from Jidong Xiao <jidong.xiao@gmail.com> ---
Jatin, thanks for the clarification. Are you using kgdb to perform single step
kernel code? I am still wondering how did you do the single step execution for
kernel level code.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 65561] KVM:Entry failed on Single stepping sti instruction

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=65561

--- Comment #34 from Jatin Kumar <jatin.iitdelhi@gmail.com> ---
(In reply to Jidong Xiao from comment #33)
> Jatin, thanks for the clarification. Are you using kgdb to perform single
> step kernel code? I am still wondering how did you do the single step
> execution for kernel level code.

As I mentioned its not a full fledge operating system, just a dummy one. So I
just set the DEBUG TRAP flag in eflags before the code section I want to single
step and disable it after wards. And in the interrupt handler I have some
simple code (say printf) for handling the single step debug interrupt.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.