Cs/ 14346 breaks 64bit hv

Cs/ 14346 breaks 64bit hv

[Woller, Thomas]

Hi Keir,
Appears as if c/s 14346 has broken booting pae and 64bit HVM linux
guests, on 64bit hv on AMD-V.  

Seems like grub can't find a bootable device, and errors out.

We haven't tried VT boxes at this point.  We just uncovered this issue
and will investigate more.  Looks like 32bit hypervisor works ok, but
haven't tested thoroughly.

We'll look at the patch, but if you have any thoughts. Appreciate taking
a closer look at the modification.

  --Tom

thomas.woller@amd.com
AMD Corporation
5204 E. Ben White Blvd. UBC1
Austin, Texas 78741
+1-512-602-0059

RE: Cs/ 14346 breaks 64bit hv

[Woller, Thomas]

Initial email was incorrect.  The issue is not with c/s 14346 but with
14345.  c/s 14344 is able to boot pae/64b HVM AMD-V guests on AMD-V
without issue, while c/s 14345 fails.

[snip from 14345 changeset]
 #endif
-  /* Debugging info */
-  printf("Booting from %x:%x\n", bootseg, bootip);
   
   /* Jump to the boot vector */
 ASM_START
[end code snippet]

Removing the printf causes pae and 64bit HVM AMD-V guests to fail to
boot on 64bit hypervisor.
If I add just the single "printf" line back into the rombios.c file,
then guests boot without issues.
Very strange!  
Tested with only changing the hvmloader, starting with an installed
14344 base, and used the 14345 hvmloader, result is grub failure.


Took a look at the rombios.s code generated in both cases (14344 and
14345) and seems like the "jump to the boot vector" code is not executed
on the failing case.  There is a "default: return" in the function
immediately above an "#if BX_TCGBIOS", and then the non-executed asm
code. 
Is bcc not understanding that the asm code following should be executed?
Seems like there is added a default "return" so that the "jump to boot
vector" code is never executed?  appears to be the case..but not sure.  
Code snippets are below.  Using gcc 3.3.5, but will try 4.x compiler,
although looks like bcc is used to generate the rombios.s file if I read
the Makefile correctly.

Any thoughts appreciated.

[code from rombios.c]
  default: return;
  }

#if BX_TCGBIOS
  tcpa_ipl((Bit32u)bootseg);               /* specs: 8.2.3 steps 4 and 5
*/
#endif
/* 14345 removes this: printf("Booting from %x:%x\n", bootseg, bootip);
*/
  
  /* Jump to the boot vector */
ASM_START
    mov  bp, sp
    ;; Build an iret stack frame that will take us to the boot vector.
    ;; iret pops ip, then cs, then flags, so push them in the opposite
order.
    pushf
    mov  ax, _int18_function.bootseg + 0[bp] 
    push ax
    mov  ax, _int18_function.bootip + 0[bp] 
    push ax

[code from 14344 that boots ]
push	-$A[bp]
push	-8[bp]
mov	bx,#.668
push	bx
mov	ax,*2
push	ax
call	_bios_printf
add	sp,*8
! 6170 
! 6171 
! 6172  #asm
!BCC_ASM
_int18_function.bootip	set	$12
_int18_function.seq_nr	set	$20
_int18_function.bootchk	set	$16
_int18_function.bootseg	set	$14
_int18_function.ebda_seg	set	$1A
_int18_function.status	set	$10
_int18_function.bootdrv	set	$17
_int18_function.bootdev	set	$18
_int18_function.e	set	0
    mov bp, sp
    ;; Build an iret stack frame that will take us to the boot vector.
    ;; iret pops ip, then cs, then flags, so push them in the opposite
order.
    pushf
    mov ax, _int18_function.bootseg + 0[bp]
    push ax
    mov ax, _int18_function.bootip + 0[bp]


[code from 14345 that does not boot ]
mov	-$A[bp],ax
! 6161     break;
jmp .654
! 6162 
! 6163   default: return;
.667:
mov	sp,bp
pop	bp
ret
! 6164   }
! 6165 
! 6166 
! 6167 
! 6168 
! 6169 
! 6170 
! 6171 
! 6172  #asm
!BCC_ASM
_int18_function.bootip	set	$12
_int18_function.seq_nr	set	$20
_int18_function.bootchk	set	$16
_int18_function.bootseg	set	$14
_int18_function.ebda_seg	set	$1A
_int18_function.status	set	$10
_int18_function.bootdrv	set	$17
_int18_function.bootdev	set	$18
_int18_function.e	set	0
    mov bp, sp
    ;; Build an iret stack frame that will take us to the boot vector.
    ;; iret pops ip, then cs, then flags, so push them in the opposite
order.
    pushf
    mov ax, _int18_function.bootseg + 0[bp]
    push ax
    mov ax, _int18_function.bootip + 0[bp]
    push ax


tom

 

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
> Woller, Thomas
> Sent: Tuesday, March 13, 2007 6:01 PM
> To: xen-devel@lists.xensource.com
> Subject: [Xen-devel] Cs/ 14346 breaks 64bit hv
> 
> Hi Keir,
> Appears as if c/s 14346 has broken booting pae and 64bit HVM 
> linux guests, on 64bit hv on AMD-V.  
> 
> Seems like grub can't find a bootable device, and errors out.
> 
> We haven't tried VT boxes at this point.  We just uncovered 
> this issue and will investigate more.  Looks like 32bit 
> hypervisor works ok, but haven't tested thoroughly.
> 
> We'll look at the patch, but if you have any thoughts. 
> Appreciate taking a closer look at the modification.
> 
>   --Tom
> 
> thomas.woller@amd.com
> AMD Corporation
> 5204 E. Ben White Blvd. UBC1
> Austin, Texas 78741
> +1-512-602-0059
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 
> 

Re: Cs/ 14346 breaks 64bit hv

[Tim Deegan]

At 01:47 -0500 on 14 Mar (1173836821), Woller, Thomas wrote:
> Took a look at the rombios.s code generated in both cases (14344 and
> 14345) and seems like the "jump to the boot vector" code is not executed
> on the failing case.  There is a "default: return" in the function
> immediately above an "#if BX_TCGBIOS", and then the non-executed asm
> code. 
> Is bcc not understanding that the asm code following should be executed?
> Seems like there is added a default "return" so that the "jump to boot
> vector" code is never executed?  appears to be the case..but not sure.  

That would be terrifying.  Presumably there's a jmp past that ret
somewhere.  I got the impression from your first email that some
guests do boot, and that can only happen through that iret. 

Does adding an explicit "return;" in C after the asm block change the
behaviour?

> Code snippets are below.  Using gcc 3.3.5, but will try 4.x compiler,
> although looks like bcc is used to generate the rombios.s file if I read
> the Makefile correctly.

Yep.  What version of bcc are you using? (bcc -v says 0.16.14 for me)

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@xensource.com>, XenSource UK Limited
Registered office c/o EC2Y 5EB, UK; company number 05334508

Re: Cs/ 14346 breaks 64bit hv

[Tim Deegan]

At 09:34 +0000 on 14 Mar (1173864844), Tim Deegan wrote:
> Yep.  What version of bcc are you using? (bcc -v says 0.16.14 for me)

...and can you send me a full copy of rombios.s from the build?

I can confirm that 64bit and pae linux guests (though that shouldn't
make a difference) boot fine on my 64bit AMD machine on -unstable tip.

Tim.

-- 
Tim Deegan <Tim.Deegan@xensource.com>, XenSource UK Limited
Registered office c/o EC2Y 5EB, UK; company number 05334508

RE: Cs/ 14346 breaks 64bit hv

[Woller, Thomas]

[-- Attachment #1: Type: text/plain, Size: 1957 bytes --]

> That would be terrifying.  Presumably there's a jmp past that 
> ret somewhere.  I got the impression from your first email 
> that some guests do boot, and that can only happen through that iret. 
I put an int 3 as the first instruction in the ASM block, but didn't hit
it. But maybe int3 handler just does a return.  
I double checked and 32bit guest on 32bit hv works, but the test was
performed with a build from a system with bcc 0.16.17, which is not the
version that we are seeing failures from.  So, currently failures only
occur with the older bcc version.

> Yep.  What version of bcc are you using? (bcc -v says 0.16.14 for me)
Well, I am using SLES9.3 installed base, and bcc -v, says "no input
file", a man of bcc doesn't show any way to get the version :P, hmmm.
Here is file, and ls -l info:
/usr/bin/bcc: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV),
for GNU/Linux 2.2.5, dynamically linked (uses shared libs), stripped
-rwxr-xr-x  1 root root 16304 Mar 19  2005 /usr/bin/bcc
I'll find a more recent system to build on with newer bcc.  Our build
server xen is also based on SLES9.3 with same bcc, and failure occurs
from xen builds generated from these also.

> 
> Does adding an explicit "return;" in C after the asm block 
> change the behaviour?
No. didn't help, code is the same in generated rombios.s.  I put in a
"call" instead and move the ASM boot code up, but same behavior/code
generation.  Looks like there has to be some "C" code after the switch
or the asm code doesn't get called.

> ...and can you send me a full copy of rombios.s from the build?
Attached are 2 rombios.s files in tar.bz2, appended the changeset to the
filename. Let me know if you don't get these, our firewall has been
doing some strange things lately.

Sure does look like the code generated isn't correct with older bcc..
But a few more tests will confirm or deny.

Thanks for looking at this!

Tom


[-- Attachment #2: rombios_files_001.tar.bz2 --]
[-- Type: application/octet-stream, Size: 111064 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Cs/ 14346 breaks 64bit hv

[Keir Fraser]

On 14/3/07 15:00, "Woller, Thomas" <thomas.woller@amd.com> wrote:

> Sure does look like the code generated isn't correct with older bcc..
> But a few more tests will confirm or deny.

If this is the case I think we should explicitly check for a recent enough
bcc during the build, and print a warning (and not build) the rombios if the
dev86 tools are too old. We don't want to have to be worrying about broken
toolchains if we can avoid it.

 -- Keir

Re: Cs/ 14346 breaks 64bit hv

[Tim Deegan]

At 10:00 -0500 on 14 Mar (1173866413), Woller, Thomas wrote:
> No. didn't help, code is the same in generated rombios.s.  I put in a
> "call" instead and move the ASM boot code up, but same behavior/code
> generation.  Looks like there has to be some "C" code after the switch
> or the asm code doesn't get called.

Yes, it looks like the asm code is being attached to the previous line
of C code, which in the 14345 case is the "return" statement inside the
switch!  My 0.16.14 bcc does the right thing there.

Is there a sensible way of getting a version number out of your bcc/as86?

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@xensource.com>, XenSource UK Limited
Registered office c/o EC2Y 5EB, UK; company number 05334508

RE: Cs/ 14346 breaks 64bit hv

[Woller, Thomas]

> Yes, it looks like the asm code is being attached to the 
> previous line of C code, which in the 14345 case is the 
> "return" statement inside the switch!  My 0.16.14 bcc does 
> the right thing there.
Yea, I installed bcc 0.16.17 and using this bcc version's generated
hvmloader, allows booting of the guest.  And the code in rombios.s looks
correct now.

> 
> Is there a sensible way of getting a version number out of 
> your bcc/as86?
I can't seem to get a version out of bcc, as86, or ld86 that seems to
have come from this dev86-0.16-0-187 package installed when I installed
SLES9.3.  looking around at an objdump isn't yielding any useful info on
first glance.

Hmmm. well, we are going to upgrade our buildserver and any personal
build boxes.  So, I think that we are good with our build process after
we upgrade.  Not sure what else to do here.

Thanks for the lookover, I thought I was misinterpreting the generated
code for a bit there.

Tom