Etherboot problems

Etherboot problems

[Simon Kagstrom]

Hello!

I'm trying to get etherboot running with KVM (version 2 from today),
but have been unsucessful so far. It works well in plain qemu, but not
with KVM.

I'm running etherboot 5.4.2, ROM rtl8139:rtl8139 [0x10ec,0x8139] built
as a "ISO bootable image without legacy floppy emulation (.iso)" on
http://rom-o-matic.net/5.4.2/. When I run it I get

   $ qemu-system-x86_64 -cdrom ~/eb-rtl8139.iso
   ...
   unhandled vm exit:  0x2
   rax 0000000060000011 rbx 0000000000000e51 rcx 0000000000000000 rdx 0000000000000000
   rsi 0000000000106347 rdi 0000000000016347 rsp 0000000000008fbe rbp 0000000000098fbe
   r8  0000000000000000 r9  0000000000000000 r10 0000000000000000 r11 0000000000000000
   r12 0000000000000000 r13 0000000000000000 r14 0000000000000000 r15 0000000000000000
   rip 0000000000000e8a rflags 00010006
   Aborted

. From browsing the Xen sources, I think the VM exit 0x2 is a
triple-fault. I've disassembled the .iso with hte, and the code that
runs last is

   ...
   00000e75 660fb7ec                       movzx       ebp, sp
   00000e79 6601c5                         add         ebp, eax            
   00000e7c 660f01975200                   lgdt        single ptr [bx+0x52]
   00000e82 0f20c0                         mov         eax, cr0
   00000e85 0c01                           or          al, 0x1
   00000e87 0f22c0                         mov         cr0, eax
   00000e8a 66ffaf6a00                     jmp         far ptr [bx+0x6a]
   ...

I'm not an expert on Intel-syntax 16-bit assembly, but I guess the
last instruction is something like

   jmp  *0x6a(%bx)

. One thing which I'm sure of is if the vm exit was caused by the move
to %cr0 or the jump.

Any takes? I can supply the .iso if someone would like to play with it

// Simon

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: Etherboot problems

[Avi Kivity]

Simon Kagstrom wrote:
> Hello!
>
> I'm trying to get etherboot running with KVM (version 2 from today),
> but have been unsucessful so far. It works well in plain qemu, but not
> with KVM.
>
> I'm running etherboot 5.4.2, ROM rtl8139:rtl8139 [0x10ec,0x8139] built
> as a "ISO bootable image without legacy floppy emulation (.iso)" on
> http://rom-o-matic.net/5.4.2/. When I run it I get
>
>    $ qemu-system-x86_64 -cdrom ~/eb-rtl8139.iso
>    ...
>    unhandled vm exit:  0x2
>    rax 0000000060000011 rbx 0000000000000e51 rcx 0000000000000000 rdx 0000000000000000
>    rsi 0000000000106347 rdi 0000000000016347 rsp 0000000000008fbe rbp 0000000000098fbe
>    r8  0000000000000000 r9  0000000000000000 r10 0000000000000000 r11 0000000000000000
>    r12 0000000000000000 r13 0000000000000000 r14 0000000000000000 r15 0000000000000000
>    rip 0000000000000e8a rflags 00010006
>    Aborted
>
> . From browsing the Xen sources, I think the VM exit 0x2 is a
> triple-fault. I've disassembled the .iso with hte, and the code that
> runs last is
>
>    ...
>    00000e75 660fb7ec                       movzx       ebp, sp
>    00000e79 6601c5                         add         ebp, eax            
>    00000e7c 660f01975200                   lgdt        single ptr [bx+0x52]
>    00000e82 0f20c0                         mov         eax, cr0
>    00000e85 0c01                           or          al, 0x1
>    00000e87 0f22c0                         mov         cr0, eax
>    00000e8a 66ffaf6a00                     jmp         far ptr [bx+0x6a]
>    ...
>
> I'm not an expert on Intel-syntax 16-bit assembly, but I guess the
> last instruction is something like
>
>    jmp  *0x6a(%bx)
>
> . One thing which I'm sure of is if the vm exit was caused by the move
> to %cr0 or the jump.
>
> Any takes? I can supply the .iso if someone would like to play with it
>   

I've generated the .iso (nice interface) and will try to reproduce it.

Thanks for testing -- we really need those wierd guests to test the 
corner cases.


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: Etherboot problems

[Avi Kivity]

Avi Kivity wrote:
>>    00000e75 660fb7ec                       movzx       ebp, sp
>>    00000e79 6601c5                         add         ebp, 
>> eax               00000e7c 660f01975200                   lgdt        
>> single ptr [bx+0x52]
>>    00000e82 0f20c0                         mov         eax, cr0
>>    00000e85 0c01                           or          al, 0x1
>>    00000e87 0f22c0                         mov         cr0, eax
>>    00000e8a 66ffaf6a00                     jmp         far ptr [bx+0x6a]
>>    ...
>>
>> I'm not an expert on Intel-syntax 16-bit assembly, but I guess the
>> last instruction is something like
>>
>>    jmp  *0x6a(%bx)
>>
>> . One thing which I'm sure of is if the vm exit was caused by the move
>> to %cr0 or the jump.
>>
>> Any takes? I can supply the .iso if someone would like to play with it
>>   
>
> I've generated the .iso (nice interface) and will try to reproduce it.
>
> Thanks for testing -- we really need those wierd guests to test the 
> corner cases.
>

Very likely this is due to VT's inability to virtualize real mode 
correctly.  Is anyone in the mood to extend x86_emulate.c to support 
real mode?


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: Etherboot problems

[Simon Kagstrom]

At Thu, 09 Nov 2006 19:49:05 +0200,
Avi Kivity wrote:
> 
> Avi Kivity wrote:
> >>    00000e75 660fb7ec                       movzx       ebp, sp
> >>    00000e79 6601c5                         add         ebp,  eax              
> >>    00000e7c 660f01975200                   lgdt        single ptr [bx+0x52]
> >>    00000e82 0f20c0                         mov         eax, cr0
> >>    00000e85 0c01                           or          al, 0x1
> >>    00000e87 0f22c0                         mov         cr0, eax
> >>    00000e8a 66ffaf6a00                     jmp         far ptr [bx+0x6a]
> >>    ...
> > I've generated the .iso (nice interface) and will try to reproduce it.
> >
> > Thanks for testing -- we really need those wierd guests to test the 
> > corner cases.
> 
> Very likely this is due to VT's inability to virtualize real mode 
> correctly.  Is anyone in the mood to extend x86_emulate.c to support 
> real mode?

I forgot to add that I've tested it in Xen as well, and it works
there. Maybe someone has an AMD at hand to test it on?

But I also suspected that this was the reason why it failed. I ran a
diff between x86_emulate.c in Xen and KVM but couldn't immediately see
any differences which would explain this.


One thing though: when KVM says

> >    unhandled vm exit:  0x2
> >    rax 0000000060000011 rbx 0000000000000e51 rcx 0000000000000000 rdx 0000000000000000
> >    rsi 0000000000106347 rdi 0000000000016347 rsp 0000000000008fbe rbp 0000000000098fbe
> >    r8  0000000000000000 r9  0000000000000000 r10 0000000000000000 r11 0000000000000000
> >    r12 0000000000000000 r13 0000000000000000 r14 0000000000000000 r15 0000000000000000
> >    rip 0000000000000e8a rflags 00010006

, I suppose this mean that the "jmp far ptr ..." instruction couldn't
be executed (maybe because of the GDT setup before).

I've tried to debug the program with gdb (qemu -s -S), but I cannot
examine the code from gdb (either "raw" or KVM). Well, well.

// Simon

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: Etherboot problems

[Anthony Liguori]

Simon Kagstrom wrote:
>> Very likely this is due to VT's inability to virtualize real mode 
>> correctly.  Is anyone in the mood to extend x86_emulate.c to support 
>> real mode?
> 
> I forgot to add that I've tested it in Xen as well, and it works
> there. Maybe someone has an AMD at hand to test it on?
> 
> But I also suspected that this was the reason why it failed. I ran a
> diff between x86_emulate.c in Xen and KVM but couldn't immediately see
> any differences which would explain this.


Xen doesn't use x86_emulate.c for real mode.  It uses vmxassist which 
uses vm86 mode (yes, there are many many emulators in Xen).

Xen 3.0.4 is planning on dropping vmxassist in favor of CPU state hand 
off between QEMU and Xen.  This would be much easier in KVM ATM as it 
doesn't require state synchronization for hardware emulation.

Regards,

Anthony Liguori

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: Etherboot problems

[Simon Kagstrom]

[-- Attachment #1: Type: text/plain, Size: 1382 bytes --]

At Thu, 09 Nov 2006 19:49:05 +0200,
Avi Kivity wrote:
> Very likely this is due to VT's inability to virtualize real mode 
> correctly.  Is anyone in the mood to extend x86_emulate.c to support 
> real mode?

I found what caused the problems for the etherboot startup and it now
works fine (well, I've not tested downloading anything over the
network, but at least it gets that far). I've started grub as well,
and that works as before even with this fix.

However, I'm not sure what the correct solution to the problem is.

The crash occurs after etherboot has switched to protected mode and
does a long jump to the new code segment:

	/* Load GDT */
   1:	DATA32 lgdt %ds:(r2p_gdt-1b)(%bx)	/* Load GDT */

	/* Switch to protected mode */
   2:	movl	%cr0, %eax
   3:	orb	$CR0_PE, %al
   4:	movl	%eax, %cr0

	/* flush prefetch queue, and reload %cs:%eip */
   5:	DATA32 ljmp %ds:*(r2p_paddr-1b)(%bx)
	.code32

The problem is that KVM zeroes all segment registers directly on a
switch to protected mode (i.e., on instruction 4 above) - and %ds
being zero causes a crash on the ljmp above.

What I've done is to remove the zeroing of all segment registers
except %ss, which is shown in the trivial patch below. I don't
remember enough of transfer-to-protected-mode assembly to know the
correct solution to this (the %ss zeroing needs to be kept for
example, not sure why).

// Simon


[-- Attachment #2: keep-segments.patch --]
[-- Type: application/octet-stream, Size: 462 bytes --]

--- kvm-2/kernel/kvm_main.c	2006-11-09 12:16:43.000000000 +0100
+++ kvm-2-ska/kernel/kvm_main.c	2006-11-10 16:24:56.000000000 +0100
@@ -714,10 +714,6 @@
 	}
 
 	FIX_PMODE_DATASEG(SS, vcpu->rmode.ss);
-	FIX_PMODE_DATASEG(ES, vcpu->rmode.es);
-	FIX_PMODE_DATASEG(DS, vcpu->rmode.ds);
-	FIX_PMODE_DATASEG(GS, vcpu->rmode.gs);
-	FIX_PMODE_DATASEG(FS, vcpu->rmode.fs);
 
 	vmcs_write16(GUEST_CS_SELECTOR,
 		     vmcs_read16(GUEST_CS_SELECTOR) & ~SELECTOR_RPL_MASK);

[-- Attachment #3: Type: text/plain, Size: 373 bytes --]

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

[-- Attachment #4: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: Etherboot problems

[Avi Kivity]

Simon Kagstrom wrote:
> At Thu, 09 Nov 2006 19:49:05 +0200,
> Avi Kivity wrote:
>   
>> Very likely this is due to VT's inability to virtualize real mode 
>> correctly.  Is anyone in the mood to extend x86_emulate.c to support 
>> real mode?
>>     
>
> I found what caused the problems for the etherboot startup and it now
> works fine (well, I've not tested downloading anything over the
> network, but at least it gets that far). I've started grub as well,
> and that works as before even with this fix.
>
> However, I'm not sure what the correct solution to the problem is.
>
> The crash occurs after etherboot has switched to protected mode and
> does a long jump to the new code segment:
>
> 	/* Load GDT */
>    1:	DATA32 lgdt %ds:(r2p_gdt-1b)(%bx)	/* Load GDT */
>
> 	/* Switch to protected mode */
>    2:	movl	%cr0, %eax
>    3:	orb	$CR0_PE, %al
>    4:	movl	%eax, %cr0
>
> 	/* flush prefetch queue, and reload %cs:%eip */
>    5:	DATA32 ljmp %ds:*(r2p_paddr-1b)(%bx)
> 	.code32
>
> The problem is that KVM zeroes all segment registers directly on a
> switch to protected mode (i.e., on instruction 4 above) - and %ds
> being zero causes a crash on the ljmp above.
>
> What I've done is to remove the zeroing of all segment registers
> except %ss, which is shown in the trivial patch below. I don't
> remember enough of transfer-to-protected-mode assembly to know the
> correct solution to this (the %ss zeroing needs to be kept for
> example, not sure why).
>   

VT does a lot of checks when entering guest mode.  These checks are 
stricter than what the processor can handle, and stricter than what some 
software expects.  The upshot is that virtualizing real mode and the 
transition between modes can't be made to work 100%.

The long term solution is to emulate rather than virtualize while in 
these states.  Short term, I'll see if there's some workaround we can 
apply here (your patch probably makes other guests fail; I'll check this 
though).

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: Etherboot problems

[Simon Kagstrom]

At Fri, 10 Nov 2006 18:11:43 +0200,
Avi Kivity wrote:
> 
> Simon Kagstrom wrote:
> > What I've done is to remove the zeroing of all segment registers
> > except %ss, which is shown in the trivial patch below. I don't
> > remember enough of transfer-to-protected-mode assembly to know the
> > correct solution to this (the %ss zeroing needs to be kept for
> > example, not sure why).
> 
> VT does a lot of checks when entering guest mode.  These checks are 
> stricter than what the processor can handle, and stricter than what some 
> software expects.  The upshot is that virtualizing real mode and the 
> transition between modes can't be made to work 100%.

OK. Just out of curiosity: What is the reason for the zeroing of the
segment selectors? I've been looking a bit in the developers manual
(volume 3a), and section 9.9.1 says

  7. After entering protected mode, the segment registers continue to
     hold the contents they had in real-address mode. The JMP or CALL
     instruction in step 4 resets the CS register.

and thereafter the segment registers should be reloaded. It seems to
me that KVM does this directly when the PE flag is set (and
enter_pmode is called), which causes etherboot to not work.

> The long term solution is to emulate rather than virtualize while in 
> these states.  Short term, I'll see if there's some workaround we can 
> apply here (your patch probably makes other guests fail; I'll check this 
> though).

I see. Grub and Linux starts with the patch, but I understand that
it's not a real solution to the problem (but since I need etherboot,
I'm attempted to continue using it for a while :-)).

// Simon

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: Etherboot problems

[Avi Kivity]

Simon Kagstrom wrote:
> At Fri, 10 Nov 2006 18:11:43 +0200,
> Avi Kivity wrote:
>   
>> Simon Kagstrom wrote:
>>     
>>> What I've done is to remove the zeroing of all segment registers
>>> except %ss, which is shown in the trivial patch below. I don't
>>> remember enough of transfer-to-protected-mode assembly to know the
>>> correct solution to this (the %ss zeroing needs to be kept for
>>> example, not sure why).
>>>       
>> VT does a lot of checks when entering guest mode.  These checks are 
>> stricter than what the processor can handle, and stricter than what some 
>> software expects.  The upshot is that virtualizing real mode and the 
>> transition between modes can't be made to work 100%.
>>     
>
> OK. Just out of curiosity: What is the reason for the zeroing of the
> segment selectors? I've been looking a bit in the developers manual
> (volume 3a), and section 9.9.1 says
>
>   7. After entering protected mode, the segment registers continue to
>      hold the contents they had in real-address mode. The JMP or CALL
>      instruction in step 4 resets the CS register.
>
> and thereafter the segment registers should be reloaded. It seems to
> me that KVM does this directly when the PE flag is set (and
> enter_pmode is called), which causes etherboot to not work.
>
>   

The problem is 22.3.1.2, "Checks on Guest Segment Registers".  When 
transitioning to _real_ mode, we have to hack the segment registers so 
that VT will accept them.  When we switch back, the hacked segments are 
no longer suitable for protected mode.

The best way forward is likely to remember the segments during a 
real->protected or protected->real mode switch, and on the switch back, 
if a segment selector has not been changed, reload it from the saved state.

Long term, we need not to use VT for real mode at all.

>> The long term solution is to emulate rather than virtualize while in 
>> these states.  Short term, I'll see if there's some workaround we can 
>> apply here (your patch probably makes other guests fail; I'll check this 
>> though).
>>     
>
> I see. Grub and Linux starts with the patch, but I understand that
> it's not a real solution to the problem (but since I need etherboot,
> I'm attempted to continue using it for a while :-)).
>   

Can you check Windows?  I'd apply the patch if it doesn't break that.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642