FreeBSD boot loader on VT-x based Xen system

FreeBSD boot loader on VT-x based Xen system

[Randy Thelen]

Folks --

I'm trying to run FreeBSD 6.0 on a VT-x based Xen system.  I'm  
experiencing a problem where the boot loader is crashing before the  
FreeBSD kernel (proper) is loaded.

The sequence of operations is that the VMX domain is constructed, the  
BIOS is executed, GRUB loads, I type the instructions to load FreeBSD  
(a boot loader) and that code begins to execute and then crashes.

Here's a console log that we captured:

> GNU GRUB  version 0.95  (638K lower / 261112K upper memory)
> [ Minimal BASH-like line editing is supported.  For the first word,  
> TAB
> lists possible command completions.  Anywhere else TAB lists the  
> possible
> completions of a device/filename.]
>
> grub> root (hd0,1,a)
>     Filesystem type is ufs2, partition type 0xa5
>
> grub> kernel /boot/loader
> [FreeBSD-a.out, loadaddr=0x200000, text=0x1000, data=0x32000,  
> bss=0x0, entry =0x200000]
>
> grub> boot
> BTX loader 1.00  Starting in protected mode (base mem=9fc00)
> Arguments passed (esp=67e88):
> <howto=80000000 bootdev=a0300000 junk=0 0 0 bootinfo=67ec4>
> Relocated bootinfo (size=48) to 9fbb8
> Relocated arguments (size=18) to 9fba0
> BTX version is 1.01
> Relocated kernel (size=780) to 9000
> Client base address is a000
> Client format is ELF
> text segment: offset=1000 vaddr=0 filesz=2a3b0 memsz=2a3b0
> data segment: offset=2c000 vaddr=2c000 filesz=53cc memsz=b4c0
> Loading complete
>
> int=0000000d  err=00000000  efl=00010006  eip=000090ac
> eax=00094c00  ebx=00202820  ecx=00000033  edx=0000a000
> esi=00209701  edi=00001f98  ebp=0000a000  esp=000017f0
> cs=0008  ds=0000  es=0000    fs=0020  gs=0020  ss=0010
> cs:eip=ff 35 0c 90 00 00 51 51-51 51 52 b1 07 6a 00 e2
>        fc 61 07 1f 0f a1 0f a9-cf fa bc 00 18 00 00 0f
> ss:esp=2b 00 00 00 02 02 00 00-00 4c 09 00 33 00 00 00
>        0a 69 6e 74 3d 30 30 30-30 30 30 30 64 20 20 65
> BTX halted

The instructions that were being executed can be found, in a FreeBSD  
source tree, at:

/usr/src/sys/boot/i386/btx/btx/btx.S

Here they are for your perusal:

                 .code32
init.8:         xorl %ecx,%ecx                  # Zero
                 movb $SEL_SDATA,%cl             # To 32-bit
                 movw %cx,%ss                    #  stack
/*
  * Launch user task.
  */
                 movb $SEL_TSS,%cl               # Set task
                 ltr %cx                         #  register
                 movl $MEM_USR,%edx              # User base address
                 movzwl %ss:BDA_MEM,%eax         # Get free memory
                 shll $0xa,%eax                  # To bytes
                 subl $0x1000,%eax               # Less arg space
                 subl %edx,%eax                  # Less base
                 movb $SEL_UDATA,%cl             # User data selector
                 pushl %ecx                      # Set SS
                 pushl %eax                      # Set ESP
                 push $0x202                     # Set flags (IF set)
                 push $SEL_UCODE                 # Set CS
                 pushl btx_hdr+0xc               # Set EIP

The error occurs with the last instruction shown above: pushl btx_hdr 
+0xc.

Here's a disassembly of those instructions:

0000907f (02) 31c9                     XOR ECX, ECX
00009081 (02) b1 10                    MOV CL, 0x10
00009083 (02) 8ed1                     MOV SS, CX
00009085 (02) b1 38                    MOV CL, 0x38
00009087 (03) 0f00d9                   LTR CX
0000908a (05) ba 00a00000              MOV EDX, 0xa000
0000908f (08) 36 0fb705 13040000       MOVZX EAX, SS:[0x413]
00009097 (03) c1e0 0a                  SHL EAX, 0xa
0000909a (05) 2d 00100000              SUB EAX, 0x1000
0000909f (02) 29d0                     SUB EAX, EDX
000090a1 (02) b1 33                    MOV CL, 0x33
000090a3 (01) 51                       PUSH ECX
000090a4 (01) 50                       PUSH EAX
000090a5 (05) 68 02020000              PUSH 0x202
000090aa (02) 6a 2b                    PUSH 0x2b
000090ac (06) ff35 0c000000            PUSH DWORD [0xc]

And, we can see that the instruction stream at 90ac matches the  
instruction stream in the eip dump in the exception display.  (Note  
that I'm disassembling the btx.o file, and so the linker has not  
fixed up the reference to btx_hdr+0xc to be 900c, instead we're  
seeing just 0xc.  Note too, that this explains why I believe I can  
replace the reference to btx_hdr+0xc with an explicit reference to  
the code segment register; see below.)

My guess is that there's a bug in VMXAssist.  I'd like some  
corroboration of that point, if possible.  In fact, because the code  
had long jumped to the above code sequence to enter 32bit mode, are  
we still in VMXAssist?  Or have we entered into pure execution mode?

It appears that the problem is based on an incorrect data segment  
register value.  The code segment register contains 8, which points  
to a valid code segment descriptor.  But, the ds register is 0, and  
that's never good.  How or why this code works on real hardware is  
undoubtedly a side effect of ... I dunno a bug, legacy, or a design  
that's not clear to me.  At any rate, If I replace the push  
instruction with:

                 pushl %cs:btx_hdr+0xc

referencing the code segment selector, then we get past the exception  
noted above and into our next problem.  I'm just curious about this  
particular bug.  Can somebody lend some insight or direction in  
tracking this down?  I'm more than happy to do lots of leg work on  
tracking this down, but a little direction would go a long way to help.

-- Randy

Re: FreeBSD boot loader on VT-x based Xen system

[Andy Clayton]

Randy Thelen wrote:
> Folks --
>
> I'm trying to run FreeBSD 6.0 on a VT-x based Xen system.  I'm 
> experiencing a problem where the boot loader is crashing before the 
> FreeBSD kernel (proper) is loaded.
>

I don't think I can help you much as far as fixing, but earlier I opened 
a bug report in Bugzilla regarding this issue at 
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=622
I will try to help any way I can but I have next to no experience with 
assembly.  Time to learn, perhaps.

Good luck for now and ask if you would like some help (with something I 
can actually help with...),
Andy Clayton

Re: FreeBSD boot loader on VT-x based Xen system

[Andy Clayton]

Andy Clayton wrote:
> Randy Thelen wrote:
>> Folks --
>>
>> I'm trying to run FreeBSD 6.0 on a VT-x based Xen system.  I'm 
>> experiencing a problem where the boot loader is crashing before the 
>> FreeBSD kernel (proper) is loaded.
>>
>
> I don't think I can help you much as far as fixing, but earlier I 
> opened a bug report in Bugzilla regarding this issue at 
> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=622
> I will try to help any way I can but I have next to no experience with 
> assembly.  Time to learn, perhaps.
>
> Good luck for now and ask if you would like some help (with something 
> I can actually help with...),
> Andy Clayton
>
Hey there is a patch that was mailed to -devel by Kevin Tronkowski today 
called "[Xen-devel] [PATCH] Extensions to vmxassist to handle additional 
mode switch cases".  It sounds promising. 

If I may quote:

Attached is a patch which extends vmxassist to handle two cases 
related to mode switching found while experimenting with different 
boot loaders.

The first case is use of the JMP instruction with memory location 
operands to complete a switch to protected mode. This patch adds 
emulation for this form of the JMP instruction to vmxassist.

The second case is where boot loader code does not save/restore a 
non-zero SS register across a protected mode traversal. 
Zeroing the SS register in vmxassist results in all sorts of problems 
in the domU after returning back to real mode. This patch stores 
segment register values before entering protected mode and correctly 
restores the old values (instead of an incorrect zero value) when 
reentering real mode. 

The first case is what I was looking at.  I'll see if I can check this 
out shortly...

Andy Clayton

Re: FreeBSD boot loader on VT-x based Xen system

[Q3aiml]

Andy Clayton wrote:
> Andy Clayton wrote:
>> Randy Thelen wrote:
>>> Folks --
>>>
>>> I'm trying to run FreeBSD 6.0 on a VT-x based Xen system.  I'm 
>>> experiencing a problem where the boot loader is crashing before the 
>>> FreeBSD kernel (proper) is loaded.
>>>
>>
>> I don't think I can help you much as far as fixing, but earlier I 
>> opened a bug report in Bugzilla regarding this issue at 
>> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=622
>> I will try to help any way I can but I have next to no experience 
>> with assembly.  Time to learn, perhaps.
>>
>> Good luck for now and ask if you would like some help (with something 
>> I can actually help with...),
>> Andy Clayton
>>
> Hey there is a patch that was mailed to -devel by Kevin Tronkowski 
> today called "[Xen-devel] [PATCH] Extensions to vmxassist to handle 
> additional mode switch cases".  It sounds promising.
>
Wow, I really should catch up on all posts here before replying.  
Yeah... you already posted something.  Ignore me for now, heh.