Re: Quarantined: Booting F21-Alpha-x86 iso image on Xen 4.4 with: io.c:206:d240 MMIO emulation failed @ 0008:ffff245f: 38 7d 2d 3d 02 83 ff ff da 95

Re: Quarantined: Booting F21-Alpha-x86 iso image on Xen 4.4 with: io.c:206:d240 MMIO emulation failed @ 0008:ffff245f: 38 7d 2d 3d 02 83 ff ff da 95

[Andrew Cooper]

On 24/09/2014 22:10, Konrad Rzeszutek Wilk wrote:
> Hey,
>
> Tomorrow is Fedora 21 Alpha test day and I decided to try it
> one day earlier. The PV guests install just fine, but to my
> surprise the HVM ones crash. It did not matter what kind of
> ISO image I put in, it always crashed at:

<snip>

> (d240) Invoking SeaBIOS ...
> (XEN) io.c:206:d240 MMIO emulation failed @ 0008:ffff245f: 38 7d 2d 3d 02 83 ff ff da 95
> (XEN) hvm.c:1346:d240 Triple fault on VCPU0 - invoking HVM shutdown action 1.
>
> Which seem to be:
>
>    0:   38 7d 2d                cmp    %bh,0x2d(%rbp)
>    3:   3d 02 83 ff ff          cmp    $0xffff8302,%eax
>    8:   da                      .byte 0xda
>    9:   95                      xchg   %eax,%ebp

Hmm - %bh and %rbp in the same instruction looks wrong.  It would appear
that it is not, but I do believe you have told objdump that it is 64bit
raw x86.

Given that it is in the middle of SeaBIOS, I presume it is actually
32bit code, which would change %rbp to %ebp but keep all other decoding
identical.  At the very least, Xen could help us out by indicating which
processor mode it believes the guest is currently in, which would avoid
any ambiguity in the correct instruction decoding.

Either way, the presence of the .byte in the disassembly (whether 32 or
64bit) indicates a misaligned instruction instruction pointer.  Can you
disassemble SeaBIOS and work out what 0xffff245f is supposed to be, and
whether it is indeed misaligned?

>
> If I change the guest config to use QEMU traditional it all works
> fine.

Which means the difference between SeaBIOS and ROMBIOS, which I suspect
is the relevant point, rather than purely a Qemu issue.

At a guess, I would say that this is a bug in SeaBIOS which causes a
jump to a bogus address, which blows up some point later because of
0x2d(%[er]bp) being an MMIO address which is unclaimed by Qemu (or
invalid in the p2m), at which point Xen injects a #GP which turns into a
triple fault if an IDT has not yet suitably been set up.

Can you put a "dump_execution_state();" in hvm_triple_fault() to gather
rather more guest register state at the point of the crash?

~Andrew

Re: Quarantined: Booting F21-Alpha-x86 iso image on Xen 4.4 with: io.c:206:d240 MMIO emulation failed @ 0008:ffff245f: 38 7d 2d 3d 02 83 ff ff da 95

[Jan Beulich]

>>> On 25.09.14 at 02:11, <andrew.cooper3@citrix.com> wrote:
> On 24/09/2014 22:10, Konrad Rzeszutek Wilk wrote:
>> (d240) Invoking SeaBIOS ...
>> (XEN) io.c:206:d240 MMIO emulation failed @ 0008:ffff245f: 38 7d 2d 3d 02 83 
> ff ff da 95
>> (XEN) hvm.c:1346:d240 Triple fault on VCPU0 - invoking HVM shutdown action 1.
>>
>> Which seem to be:
>>
>>    0:   38 7d 2d                cmp    %bh,0x2d(%rbp)
>>    3:   3d 02 83 ff ff          cmp    $0xffff8302,%eax
>>    8:   da                      .byte 0xda
>>    9:   95                      xchg   %eax,%ebp
> 
> Hmm - %bh and %rbp in the same instruction looks wrong.  It would appear
> that it is not, but I do believe you have told objdump that it is 64bit
> raw x86.

Even if it was 64-bit code, %rbp being used as the base address of
a memory access in no way contradicts the use of %bh.

However, grepping through the SeaBIOS sources doesn't reveal
anything hinting at an instruction like this being present. Nor can I
find such a byte sequence in the binaries I have here (albeit if this
is compiled rather than assembly code, differences due to
different compilers being used may make it impossible for me to
find this).

> Either way, the presence of the .byte in the disassembly (whether 32 or
> 64bit) indicates a misaligned instruction instruction pointer.

Depending on whether this was on an AMD or Intel system, the
bytes past the ones constituting the first instruction may not be
valid. Which is supported by bytes 4-7 resembling the upper half of
a hypervisor address (which could be a leftover on the stack).

>> If I change the guest config to use QEMU traditional it all works
>> fine.
> 
> Which means the difference between SeaBIOS and ROMBIOS, which I suspect
> is the relevant point, rather than purely a Qemu issue.
> 
> At a guess, I would say that this is a bug in SeaBIOS which causes a
> jump to a bogus address, which blows up some point later because of
> 0x2d(%[er]bp) being an MMIO address which is unclaimed by Qemu (or
> invalid in the p2m), at which point Xen injects a #GP which turns into a
> triple fault if an IDT has not yet suitably been set up.

Could also be a build problem, considering that Konrad's own rebuilt
hvmloader didn't exhibit the issue.

Jan

Re: Quarantined: Booting F21-Alpha-x86 iso image on Xen 4.4 with: io.c:206:d240 MMIO emulation failed @ 0008:ffff245f: 38 7d 2d 3d 02 83 ff ff da 95

[Konrad Rzeszutek Wilk]

On Thu, Sep 25, 2014 at 10:49:23AM +0100, Jan Beulich wrote:
> >>> On 25.09.14 at 02:11, <andrew.cooper3@citrix.com> wrote:
> > On 24/09/2014 22:10, Konrad Rzeszutek Wilk wrote:
> >> (d240) Invoking SeaBIOS ...
> >> (XEN) io.c:206:d240 MMIO emulation failed @ 0008:ffff245f: 38 7d 2d 3d 02 83 
> > ff ff da 95
> >> (XEN) hvm.c:1346:d240 Triple fault on VCPU0 - invoking HVM shutdown action 1.
> >>
> >> Which seem to be:
> >>
> >>    0:   38 7d 2d                cmp    %bh,0x2d(%rbp)
> >>    3:   3d 02 83 ff ff          cmp    $0xffff8302,%eax
> >>    8:   da                      .byte 0xda
> >>    9:   95                      xchg   %eax,%ebp
> > 
> > Hmm - %bh and %rbp in the same instruction looks wrong.  It would appear
> > that it is not, but I do believe you have told objdump that it is 64bit
> > raw x86.
> 
> Even if it was 64-bit code, %rbp being used as the base address of
> a memory access in no way contradicts the use of %bh.
> 
> However, grepping through the SeaBIOS sources doesn't reveal
> anything hinting at an instruction like this being present. Nor can I
> find such a byte sequence in the binaries I have here (albeit if this
> is compiled rather than assembly code, differences due to
> different compilers being used may make it impossible for me to
> find this).
> 
> > Either way, the presence of the .byte in the disassembly (whether 32 or
> > 64bit) indicates a misaligned instruction instruction pointer.
> 
> Depending on whether this was on an AMD or Intel system, the
> bytes past the ones constituting the first instruction may not be
> valid. Which is supported by bytes 4-7 resembling the upper half of
> a hypervisor address (which could be a leftover on the stack).
> 
> >> If I change the guest config to use QEMU traditional it all works
> >> fine.
> > 
> > Which means the difference between SeaBIOS and ROMBIOS, which I suspect
> > is the relevant point, rather than purely a Qemu issue.
> > 
> > At a guess, I would say that this is a bug in SeaBIOS which causes a
> > jump to a bogus address, which blows up some point later because of
> > 0x2d(%[er]bp) being an MMIO address which is unclaimed by Qemu (or
> > invalid in the p2m), at which point Xen injects a #GP which turns into a
> > triple fault if an IDT has not yet suitably been set up.
> 
> Could also be a build problem, considering that Konrad's own rebuilt
> hvmloader didn't exhibit the issue.

It was, but not an obvious one. The seabios-bin RPM was built with 'CONFIG_XEN=n'
and the Xen RPM was built with: "--with-system-seabios=/usr/share/seabios/bios.bin"
which means that hvmloader slurped up an seabios BIOS binary that was not
built for Xen support.

Changing config.seabios-128k to have CONFIG_XEN=n to CONFIG_XEN=y and rebuilding
both seabios and Xen RPMs fixed the issue.

This is also tracked here:
https://bugzilla.redhat.com/show_bug.cgi?id=1146260

> 
> Jan
>