Xen 3.2 and Big Real Mode support?

Xen 3.2 and Big Real Mode support?

[Guillaume Thouvenin]

Hello,

 I read in the announce of Xen 3.2.0 released that it has "preliminary
support for a wider range of bootloaders in fully virtualised (HVM)
guests, using full emulation of x86 'real mode'" .

 I'd like to know what is the level of the emulation of x86 'real
mode'? I ask that because I tried to install OpenSuse 10.3 (I used
the iso file openSUSE-10.3-GM-x86_64-mini.iso) with Xen 3.2-testing
from mercurial and the boot failed. I got the following messages:

"...
 Booting from CD-Rom...

 ISOLINUX 3.31 0x46f43c1e Copyright (C) 1994-2005 H. Peter Anvin
 Loading...
"

and it hangs after the "...". Logs don't show any specifics problems.
As OpenSuse 10.3 is known to have some 'real mode' issues due to the
presence of the gfx bootloader I asking myself what is the state of the
art of 'real mode' emulation into Xen?

Do you have some tests to validate the real mode emulation?


Thanks for the help
Best Regards,

Guillaume

Re: Xen 3.2 and Big Real Mode support?

[Keir Fraser]

You need to build Xen with 'vmxassist=n'. If you boot a debug build of Xen
and look at the HVM guest output that gets written to the Xen console in
that case as the HVM guest boots, if there is any reference to VMXASSIST
then you do not have an appropriate build of Xen. If there is no reference
to VMXASSIST, and you see the line "Invoking ROMBIOS ...", then you are
good.

If you pull latest xen-unstable (eventually-to-be-3.3) then vmxassist is
disabled by default, and a few more instructions are correctly emulated.

 -- Keir

On 23/1/08 14:32, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net>
wrote:

> Hello,
> 
>  I read in the announce of Xen 3.2.0 released that it has "preliminary
> support for a wider range of bootloaders in fully virtualised (HVM)
> guests, using full emulation of x86 'real mode'" .
> 
>  I'd like to know what is the level of the emulation of x86 'real
> mode'? I ask that because I tried to install OpenSuse 10.3 (I used
> the iso file openSUSE-10.3-GM-x86_64-mini.iso) with Xen 3.2-testing
> from mercurial and the boot failed. I got the following messages:
> 
> "...
>  Booting from CD-Rom...
> 
>  ISOLINUX 3.31 0x46f43c1e Copyright (C) 1994-2005 H. Peter Anvin
>  Loading...
> "
> 
> and it hangs after the "...". Logs don't show any specifics problems.
> As OpenSuse 10.3 is known to have some 'real mode' issues due to the
> presence of the gfx bootloader I asking myself what is the state of the
> art of 'real mode' emulation into Xen?
> 
> Do you have some tests to validate the real mode emulation?
> 
> 
> Thanks for the help
> Best Regards,
> 
> Guillaume

Re: Xen 3.2 and Big Real Mode support?

[Samuel Thibault]

Hello,

Guillaume Thouvenin, le Wed 23 Jan 2008 15:32:52 +0100, a écrit :
>  I'd like to know what is the level of the emulation of x86 'real
> mode'? I ask that because I tried to install OpenSuse 10.3 (I used
> the iso file openSUSE-10.3-GM-x86_64-mini.iso) with Xen 3.2-testing
> from mercurial and the boot failed. I got the following messages:
> 
> "...
>  Booting from CD-Rom...
> 
>  ISOLINUX 3.31 0x46f43c1e Copyright (C) 1994-2005 H. Peter Anvin
>  Loading...
> "
> 
> and it hangs after the "...". Logs don't show any specifics problems.
> As OpenSuse 10.3 is known to have some 'real mode' issues due to the
> presence of the gfx bootloader I asking myself what is the state of the
> art of 'real mode' emulation into Xen?

That's the kind of bug that should get fixed by this real mode emulation
indeed.  Note that that precise bug (gfxboot) only appears on Intel
machines with VMX.  AMD's SVM doesn't have that issue.

Samuel

Re: Xen 3.2 and Big Real Mode support?

[Guillaume Thouvenin]

On Wed, 23 Jan 2008 14:35:44 +0000
Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:

> If you pull latest xen-unstable (eventually-to-be-3.3) then vmxassist is
> disabled by default, and a few more instructions are correctly emulated.

Thanks for your help. So I tried to install opensuse from
openSUSE-10.3-GM-x86_64-mini.iso on my Intel Xeon and it seems that at
least one instruction is not emulated in real-mode. I'm working on the
problem.

Regards,
Guillaume

(XEN) HVM1: HVM Loader
(XEN) HVM1: Detected Xen v3.3-unstable
(XEN) HVM1: Writing SMBIOS tables ...
(XEN) HVM1: Loading ROMBIOS ...
(XEN) HVM1: 9004 bytes of ROMBIOS high-memory extensions:
(XEN) HVM1:   Relocating to 0x3fff9c00-0x3fffbf2c ... done
(XEN) irq.c:236: Dom1 PCI link 0 changed 0 -> 5
(XEN) HVM1: PCI-ISA link 0 routed to IRQ5
(XEN) irq.c:236: Dom1 PCI link 1 changed 0 -> 10
(XEN) HVM1: PCI-ISA link 1 routed to IRQ10
(XEN) irq.c:236: Dom1 PCI link 2 changed 0 -> 11
(XEN) HVM1: PCI-ISA link 2 routed to IRQ11
(XEN) irq.c:236: Dom1 PCI link 3 changed 0 -> 5
(XEN) HVM1: PCI-ISA link 3 routed to IRQ5
(XEN) HVM1: pci dev 01:2 INTA->IRQ10
(XEN) HVM1: pci dev 03:0 INTA->IRQ5
(XEN) HVM1: pci dev 04:0 INTA->IRQ5
(XEN) HVM1: pci dev 02:0 bar 10 size 02000000: f0000008
(XEN) HVM1: pci dev 03:0 bar 14 size 01000000: f2000008
(XEN) HVM1: pci dev 02:0 bar 14 size 00001000: f3000000
(XEN) HVM1: pci dev 03:0 bar 10 size 00000100: 0000c001
(XEN) HVM1: pci dev 04:0 bar 10 size 00000100: 0000c101
(XEN) HVM1: pci dev 04:0 bar 14 size 00000100: f3001000
(XEN) HVM1: pci dev 01:1 bar 20 size 00000010: 0000c201
(XEN) HVM1: Creating MP tables ...
(XEN) HVM1: Loading Cirrus VGABIOS ...
(XEN) HVM1: Loading ACPI ...
(XEN) HVM1: BIOS map:
(XEN) HVM1:  c0000-c7fff: VGA BIOS
(XEN) HVM1:  e9000-e9143: SMBIOS tables
(XEN) HVM1:  ea000-eb1df: ACPI tables
(XEN) HVM1:  f0000-fffff: Main BIOS
(XEN) HVM1: Invoking ROMBIOS ...
(XEN) HVM1:  rombios.c,v 1.138 2005/05/07 15:55:26 vruppert Exp $
(XEN) HVM1: VGABios $Id: vgabios.c,v 1.61 2005/05/24 16:50:50 vruppert Exp $
(XEN) HVM1: HVMAssist BIOS, 1 cpu, $Revision: 1.138 $ $Date: 2005/05/07 15:55:26 $
(XEN) HVM1:
(XEN) HVM1: ata0-0: PCHS=16383/16/63 translation=lba LCHS=1024/255/63
(XEN) HVM1: ata0 master: QEMU HARDDISK ATA-7 Hard-Disk (10240 MBytes)
(XEN) HVM1: ata0  slave: Unknown device
(XEN) HVM1: ata1 master: QEMU CD-ROM ATAPI-4 CD-Rom/DVD-Rom
(XEN) HVM1: ata1  slave: Unknown device
(XEN) HVM1:
(XEN) HVM1:
(XEN) HVM1:
(XEN) HVM1: Press F10 to select boot device.
(XEN) HVM1: Booting from CD-Rom...
(XEN) realmode.c:545:d1 Failed to emulate insn.
(XEN) realmode.c:599:d1 Real-mode emulation failed @ 4004:0000b071: 0f 17 0f 01 17 0f
(XEN) domain_crash_sync called from realmode.c:600
(XEN) Domain 1 (vcpu#0) crashed on cpu#6:
(XEN) ----[ Xen-3.3-unstable  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    6
(XEN) RIP:    0028:[<000000000000b071>]
(XEN) RFLAGS: 0000000000000017   CONTEXT: hvm
(XEN) rax: 0000000000004d0b   rbx: 0000000000800000   rcx: 0000000000004f00
(XEN) rdx: 00000000000013ca   rsi: 0000000000055e1c   rdi: 0000000000055e1d
(XEN) rbp: 000000000000200b   rsp: 00000000ffff007a   r8:  0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000000000010   cr4: 0000000000000000
(XEN) cr3: 0000000000000000   cr2: 0000000000000000
(XEN) ds: 0020   es: 0020   fs: 0020   gs: 0020   ss: 0020   cs: 0028
[

Re: Xen 3.2 and Big Real Mode support?

[Keir Fraser]

On 31/1/08 09:42, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net>
wrote:

> (XEN) realmode.c:545:d1 Failed to emulate insn.
> (XEN) realmode.c:599:d1 Real-mode emulation failed @ 4004:0000b071: 0f 17 0f
> 01 17 0f
> (XEN) domain_crash_sync called from realmode.c:600
> (XEN) Domain 1 (vcpu#0) crashed on cpu#6:

The failing instruction is 0F 17, which is MOVHPS. That's an SSE
instruction, and it's a bit unlikely to be the very first one we would see.
I may try to repro this later.

 -- Keir

Re: Re: Xen 3.2 and Big Real Mode support?

[Keir Fraser]

On 31/1/08 09:48, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:

> On 31/1/08 09:42, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net>
> wrote:
> 
>> (XEN) realmode.c:545:d1 Failed to emulate insn.
>> (XEN) realmode.c:599:d1 Real-mode emulation failed @ 4004:0000b071: 0f 17 0f
>> 01 17 0f
>> (XEN) domain_crash_sync called from realmode.c:600
>> (XEN) Domain 1 (vcpu#0) crashed on cpu#6:
> 
> The failing instruction is 0F 17, which is MOVHPS. That's an SSE
> instruction, and it's a bit unlikely to be the very first one we would see.
> I may try to repro this later.

A reasonably simple test would be to modify the CPUID emulation intercept in
realmode.c to mask out SSE feature bits.

 -- Keir

Re: Re: Xen 3.2 and Big Real Mode support?

[Guillaume Thouvenin]

On Thu, 31 Jan 2008 09:53:51 +0000
Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:

> On 31/1/08 09:48, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
> > On 31/1/08 09:42, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net>
> > wrote:
> > 
> >> (XEN) realmode.c:545:d1 Failed to emulate insn.
> >> (XEN) realmode.c:599:d1 Real-mode emulation failed @ 4004:0000b071: 0f 17 0f
> >> 01 17 0f
> >> (XEN) domain_crash_sync called from realmode.c:600
> >> (XEN) Domain 1 (vcpu#0) crashed on cpu#6:
> > 
> > The failing instruction is 0F 17, which is MOVHPS. That's an SSE
> > instruction, and it's a bit unlikely to be the very first one we would see.
> > I may try to repro this later.
> 
> A reasonably simple test would be to modify the CPUID emulation intercept in
> realmode.c to mask out SSE feature bits.

Ok thanks for the hint.

So I will mask bit 25 and 26 of edx (that indicate respectively SSE and
SEE2 features) and bit 0 of ecx (that indicates SSE3 feature). I will
see if it allows me to boot the installation CD.

-- 
Guillaume Thouvenin

Re: Re: Xen 3.2 and Big Real Mode support?

[Keir Fraser]

On 31/1/08 10:18, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net>
wrote:

> On Thu, 31 Jan 2008 09:53:51 +0000
> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
>> On 31/1/08 09:48, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:
>> 
>>> On 31/1/08 09:42, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net>
>>> wrote:
> 
> Ok thanks for the hint.
> 
> So I will mask bit 25 and 26 of edx (that indicate respectively SSE and
> SEE2 features) and bit 0 of ecx (that indicates SSE3 feature). I will
> see if it allows me to boot the installation CD.

I reproduced the problem. CPUID is not executed before the crash. Also a
disassembly of that area of memory does not look like sane 16-bit code. So I
think something has gone wrong before we get to the (probably bogus) MOVHPS
instruction.

 -- Keir

Re: Re: Xen 3.2 and Big Real Mode support?

[Guillaume Thouvenin]

On Thu, 31 Jan 2008 10:36:31 +0000
Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:

> I reproduced the problem. CPUID is not executed before the crash. Also a
> disassembly of that area of memory does not look like sane 16-bit code. So I
> think something has gone wrong before we get to the (probably bogus) MOVHPS
> instruction.

A question here. You reproduce the problem by starting a domain and
booting it on the openSUSE-10.3-GM-x86_64-mini.iso CD-ROM? How do you
disassemble the area of memory that produces the problem? Are you using
"objdump" on the bootloader found on the iso file or are you using some
other trick?


Thanks for your help,

-- 
Guillaume Thouvenin

Re: Re: Xen 3.2 and Big Real Mode support?

[Keir Fraser]

[-- Attachment #1: Type: text/plain, Size: 861 bytes --]

On 31/1/08 13:40, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net>
wrote:

> On Thu, 31 Jan 2008 10:36:31 +0000
> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
>> I reproduced the problem. CPUID is not executed before the crash. Also a
>> disassembly of that area of memory does not look like sane 16-bit code. So I
>> think something has gone wrong before we get to the (probably bogus) MOVHPS
>> instruction.
> 
> A question here. You reproduce the problem by starting a domain and
> booting it on the openSUSE-10.3-GM-x86_64-mini.iso CD-ROM? How do you
> disassemble the area of memory that produces the problem? Are you using
> "objdump" on the bootloader found on the iso file or are you using some
> other trick?

I have a utility to scrape another domain's memory to a file, and then I run
ndisasm on that. The utility is attached.

 -- Keir


[-- Attachment #2: xenmem.c --]
[-- Type: application/octet-stream, Size: 1451 bytes --]

/******************************************************************************
 * xenmem.c
 *
 * Dump guest memory pages to a binary file.
 */

#include <time.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <signal.h>
#include <string.h>
#include <inttypes.h>
#include <xenctrl.h>

int main(int argc, char **argv)
{
    int xc_handle, domid, bytes;
    unsigned long start, end, _start, _end;
    unsigned char *p;

    if ( argc != 4 )
    {
        fprintf(stderr, "usage: xenmem <domid> <start> <end>\n");
        exit(-1);
    }

    domid = atoi(argv[1]);
    if (domid==0) {
      fprintf(stderr, "cannot trace dom0\n");
      exit(-1);
    }

    start = strtoul(argv[2], NULL, 0);
    end   = strtoul(argv[3], NULL, 0);

    fprintf(stderr, "Dom %d: %08lx - %08lx\n", domid, start, end);
    
    xc_handle = xc_interface_open();

    _start = start & ~0xffful;
    _end = (end + 0xffful) & ~0xffful;

    p = xc_map_foreign_range(xc_handle, domid, _end-_start,
                             PROT_READ, _start>>12);
    if ( p == NULL )
    {
        fprintf(stderr, "failed to map page.\n");
        exit(-1);
    }

    bytes = write(1, p + (start - _start), end - start);
    fprintf(stderr, "Wrote %d bytes to stdout\n", bytes);

    munmap(p, _end-_start);

    xc_interface_close(xc_handle);

    return 0;
}

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: Xen 3.2 and Big Real Mode support?

[Keir Fraser]

On 31/1/08 13:40, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net>
wrote:

>> I reproduced the problem. CPUID is not executed before the crash. Also a
>> disassembly of that area of memory does not look like sane 16-bit code. So I
>> think something has gone wrong before we get to the (probably bogus) MOVHPS
>> instruction.
> 
> A question here. You reproduce the problem by starting a domain and
> booting it on the openSUSE-10.3-GM-x86_64-mini.iso CD-ROM? How do you
> disassemble the area of memory that produces the problem? Are you using
> "objdump" on the bootloader found on the iso file or are you using some
> other trick?

By the way, this is now fixed with tip of the xen-unstable tree (changeset
16980), obtainable from http://xenbits.xensource.com/staging/xen-unstable.hg

 -- Keir

Re: Re: Xen 3.2 and Big Real Mode support?

[Guillaume Thouvenin]

On Tue, 05 Feb 2008 16:06:28 +0000
Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:

> On 31/1/08 13:40, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net>
> wrote:
> 
> >> I reproduced the problem. CPUID is not executed before the crash. Also a
> >> disassembly of that area of memory does not look like sane 16-bit code. So I
> >> think something has gone wrong before we get to the (probably bogus) MOVHPS
> >> instruction.
> > 
> > A question here. You reproduce the problem by starting a domain and
> > booting it on the openSUSE-10.3-GM-x86_64-mini.iso CD-ROM? How do you
> > disassemble the area of memory that produces the problem? Are you using
> > "objdump" on the bootloader found on the iso file or are you using some
> > other trick?
> 
> By the way, this is now fixed with tip of the xen-unstable tree (changeset
> 16980), obtainable from http://xenbits.xensource.com/staging/xen-unstable.hg

Waow. I don't understand everything (and especially how you find that
the problem was here) but it works now.

 I have another question, why did you remove VMXASSIST? I saw that you
replaced all invocation to VMXASSIST by only one call to
vmx_goto_realmode() in arch/x86/hvm/svm/x86_64/exits.S. Are there some
links to any discussion about this choice?

Thanks for your help,
Guillaume

Re: Re: Xen 3.2 and Big Real Mode support?

[Keir Fraser]

On 6/2/08 08:20, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net>
wrote:

>> By the way, this is now fixed with tip of the xen-unstable tree (changeset
>> 16980), obtainable from http://xenbits.xensource.com/staging/xen-unstable.hg
> 
> Waow. I don't understand everything (and especially how you find that
> the problem was here) but it works now.
> 
>  I have another question, why did you remove VMXASSIST? I saw that you
> replaced all invocation to VMXASSIST by only one call to
> vmx_goto_realmode() in arch/x86/hvm/svm/x86_64/exits.S. Are there some
> links to any discussion about this choice?

The new emulation replaces vmxassist. There are fundamental limitations to
teh vmxassist approach which mean that it could never execute, for example,
the openSuSE install CDs because of their use of 'big real' mode. This is
talked about in a few threads on xen-devel, e.g.:
http://lists.xensource.com/archives/html/xen-devel/2006-06/msg00005.html
It's bit hard to understand without more context though!

Roughly speaking the main pro for emulation is more accurate execution
leading to working with a wider range of bootloaders. It's also less code,
and I understand the code (since I wrote it) so I can maintain it better in
future. The main con is that it may be a bit slower than vmxassist, since
vmxassist did direct execution of most real-mode instructions rather than
emulating all of them. But actually most time is spent emulating I/O in both
cases, so I think the speed difference is not that important.

Now that emulation seems to work reliably in xen-unstable (apart from
openSuSE 10.1 CD, argh!) I will remove vmxassist entirely. Also I will
backport the emulation fixes to 3.2-testing so that real-mode emulation will
work properly in 3.2.1. I will keep VMXASSIST as the default in 3.2 branch
though.

 -- Keir

Re: Re: Xen 3.2 and Big Real Mode support?

[Keir Fraser]

On 6/2/08 08:20, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net>
wrote:

>> By the way, this is now fixed with tip of the xen-unstable tree (changeset
>> 16980), obtainable from http://xenbits.xensource.com/staging/xen-unstable.hg
> 
> Waow. I don't understand everything (and especially how you find that
> the problem was here) but it works now.

I found the bug because I tracked down the real-mode -> protected-mode
transition code in the SuSE bootloader and it did something like this at
start of protected mode:
  mov %ss,%eax
  shl $4,%eax
  add %eax,%esp
  mov <protected mode flat segment>,%bx
  mov %bx,%ss
  .....

The problem was that the bottom bits of %ss got cleared on exit from real
mode, to satisfy vmenter checks that the processor does. But this deliberate
corruption of state can of course affect program execution and in this case
we end up with a bad stack pointer! So the fix had to be to emulate far
enough into protected mode that %cs and %ss both get reloaded with valid
protected-mode segment data.

 -- Keir