Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

[Atom2 <ariel.atom2@web2web.at>]

Am 16.11.15 um 16:31 schrieb Konrad Rzeszutek Wilk:
>>>> Your analysis was absolutely spot on. After re-thinking this for a
>>>> moment, I thought going down that route first would make a lot of sense
>>>> as PV guests still do work and one of the differences to HVM domUs is
>>>> that the former do _not_ require SeaBIOS. Looking at my log files of
>>>> installed packages confirmed an upgrade from SeaBIOS 1.7.5 to 1.8.2 in
>>>> the relevant timeframe which obviously had not made it to the hvmloader
>>>> of xen-4.5.1 as I did not re-compile xen after the upgrade of SeaBIOS.
>>>>
>>>> So I re-compiled xen-4.5.1 (obviously now using the installed SeaBIOS
>>>> 1.8.2) and the same error as with xen-4.5.2 popped up - and that seemed
>>>> to strongly indicate that there indeed might be an issue with SeaBIOS as
>>>> this probably was the only variable that had changed from the original
>>>> install of xen-4.5.1.
> I recall seeing this way back in Fedora 20 days. I narrowed it down the
> SeaBIOS version that was a standalone package to not have CONFIG_XEN.
>
> Having that fixed in the SeaBIOS package fixed it.
Hi Konrad, Doug, Andrew (specifically added to this part of the thread)!
Konrad, you might have found an interesting point. I did have a look at 
the ebuild for the failing version and in there I found the following 
comment:
====== comment from ebuild =======
     # Upstream hasn't released a new binary.  We snipe ours from Fedora 
for now.
     # http://code.coreboot.org/p/seabios/downloads/get/bios.bin-${PV}.gz
====== end comment from ebuild =======
which might in fact underline that there might be an issue similar to 
what you described above.

What is also pretty interesting is the fact that the old (working) 
SeaBIOS version 1.7.5 installed as "bios.bin" under /usr/share/seabios 
is actually 262.144 bytes in size whereas the new (invalid) SeaBIOS 
1.8.2 installed in the same location is only half as big: 131.072 bytes.

I checked at the download site and the 1.8.2 binary version is indeed 
not available from http://code.coreboot.org/p/seabios/downloads/. But 
both the binary versions for 1.7.5 and 1.8.0 are available and both are 
acutually 262.144 bytes in size, so I'd be very surprised if the 1.8.2 
version is really only half that size. By the way, the old working 
version (according to the ebuild) was directly downloaded from the above 
url and also shows an identical SHA1 digest to that version available 
for download there.

To me this looks as if something is really wrong here. If anybody of you 
has access to a 1.8.2 version, could you please confirm whether there's 
really that big a size difference between the 1.7.5 and the 1.8.2 
versions? Or is that difference probably attributable to the missing 
CONFIG_XEN option?

Andrew: I havent't gotten around to run the debug version of the 
hypervisor again, but if the current suspicion turns out to be true, 
there's probably not much value in that anyways. Would you agree?

Thanks Atom2
>>>> My next step was to downgrade SeaBIOS to 1.7.5 and to re-compile
>>>> xen-4.5.1. Voila, the system was again up and running. While still
>>>> having SeaBIOS 1.7.5 installed, I also re-compiled xen-4.5.2 and ... you
>>>> probably guessed it ... the problem was gone: The system boots up with
>>>> no issues and everything is fine again.
>>>>
>>>> So in a nutshell: There seems to be a problem with SeaBIOS 1.8.2
>>>> preventing HVM doamins from successfully starting up. I don't know what
>>>> this is triggered from, if this is specific to my hardware or whether
>>>> something else in my environment is to blame.
>>>>
>>>> In any case, I am again more than happy to provide data / run a few
>>>> tests should you wish to get to the grounds of this.
>>>>
>>>> I do owe you a beer (or any other drink) should you ever be at my
>>>> location (i.e. Vienna, Austria).
>>>>
>>>> Many thanks again for your analysis and your first class support. Xen
>>>> and their people absolutely rock!
>>>>
>>>> Atom2
>>> I'm a little late to the thread but can you send me (you can do it
>>> off-list if you'd like) the USE flags you used for xen, xen-tools and
>>> seabios? Also emerge --info. You can kill two birds with one stone by
>>> using emerge --info xen.
>> Hi Doug,
>> here you go:
>> USE flags:
>> app-emulation/xen-4.5.2-r1::gentoo  USE="-custom-cflags -debug -efi -flask
>> -xsm"
>> app-emulation/xen-tools-4.5.2::gentoo  USE="hvm pam pygrub python qemu
>> screen system-seabios -api -custom-cflags -debug -doc -flask (-ocaml) -ovmf
>> -static-libs -system-qemu" PYTHON_TARGETS="python2_7"
>> sys-firmware/seabios-1.7.5::gentoo  USE="binary"
>> emerge --info: Please see the attached file
>>> I'm not too familiar with the xen ebuilds but I was pretty sure that
>>> xen-tools is what builds hvmloader and it downloads a copy of SeaBIOS
>>> and builds it so that it remains consistent. But obviously your
>>> experience shows otherwise.
>> You are right, it's xen-tools that builds hvmloader. If I remember
>> correctly, the "system-seabios" USE flag (for xen-tools) specifies whether
>> sys-firmware/seabios is used and the latter downloads SeaBIOS in it's binary
>> form provided its "binary" USE flag is set. At least that's my
>> understanding.
>>> I'm looking at some ideas to improve SeaBIOS packaging on Gentoo and
>>> your info would be helpful.
>> Great. Whatever makes gentoo and xen stronger will be awesome. What
>> immediately springs to mind is to create a separate hvmloader package and
>> slot that (that's just an idea and probably not fully thought through, but
>> ss far as I understood Andrew, it would then be possible to specify the
>> specific firmware version [i.e. hvmloader] to use on xl's command line by
>> using firmware_override="full/path/to/firmware").
>>
>> I also found out that an upgrade to sys-firmware/seabios obviously does not
>> trigger an automatic re-emerge of xen-tools and thus hvmloader. Shouldn't
>> this also happen automatically as xen-tools depends on seabios?
> No and yes. You can use 'seabios' also with a standalone QEMU (so not
> hardware virtualization).
> en.org/xen-devel