Re: HVM support for e820_host (Was: Bug: Limitation of <=2GB RAM in domU persists with 4.3.0)

[Gordan Bobic <gordan@bobich.net>]

I spoke too soon - even with e820_host=0, the same error occurs. What 
did I break? The code in question is this:

if (libxl_defbool_val(d_config->b_info.e820_host)) {
     ret = libxl__e820_alloc(gc, domid, d_config);
     if (ret) {
         LIBXL__LOG_ERRNO(gc->owner, LIBXL__LOG_ERROR,
                 "Failed while collecting E820 with: %d (errno:%d)\n",
                 ret, errno);
     }
}

With e820_host=0, that outer black should evaluate to false, should it 
not? In libxl_create.c, if I am understanding the code correctly, 
e820_host is defaulted to false, too. What am I missing?

Gordan

On 09/03/2013 09:35 PM, Gordan Bobic wrote:
> First attempt at a test run predictably failed. I added e820_host=1 to a
> VM config and tried starting it:
>
> [root@normandy ~]# xl create /etc/xen/edi
> Parsing config from /etc/xen/edi
> libxl: error: libxl_x86.c:307:libxl__arch_domain_create: Failed while
> collecting E820 with: -3 (errno:-1)
>
> libxl: error: libxl_create.c:901:domcreate_rebuild_done: cannot
> (re-)build domain: -3
> libxl: error: libxl_dm.c:1300:libxl__destroy_device_model: could not
> find device-model's pid for dom 1
> libxl: error: libxl.c:1415:libxl__destroy_domid:
> libxl__destroy_device_model failed for 1
>
> xl-edi.log, qemu-dm-edi.log attached.
> Both actually look identical to previous logs before the patch.
>
> Is this something that is clearly a consequence of the patch being
> incomplete? Or did I break something?
>
> Gordan
>
> On 09/03/2013 08:47 PM, Gordan Bobic wrote:
>> On 09/03/2013 03:59 PM, Konrad Rzeszutek Wilk wrote:
>>
>>>>>> 2) Further, I'm finding myself motivated to write that
>>>>>> auto-set (as opposed to hard coded) vBAR=pBAR patch discussed
>>>>>> briefly a week or so ago (have an init script read the BAR
>>>>>> info from dom0 and put it in xenstore, plus a patch to
>>>>>> make pBAR=vBAR reservations built dynamically rather than
>>>>>> statically, based on this data. Now, I'm quite fluent in C,
>>>>>> but my familiarity with Xen soruce code is nearly non-existant
>>>>>> (limited to studying an old unsupported patch every now and then
>>>>>> in order to make it apply to a more recent code release).
>>>>>> Can anyone help me out with a high level view WRT where
>>>>>> this would be best plumbed in (which files and the flow of
>>>>>> control between the affected files)?
>>>>>
>>>>> hvmloader probably and the libxl e820 code. What from a
>>>>> high view needs to happen is that:
>>>>> 1). Need to relax the check in libxl for e820_hole
>>>>>     to also do it for HVM guests. Said code just iterates over the
>>>>>     host E820 and sanitizes it a bit and makes a E820 hypercall to
>>>>>     set it for the guest.
>> [snip]
>>
>> OK, I have attached a preliminary patch against 4.3.0 for the libxl
>> part. It compiles. I haven't tried running it to see if it actually
>> works or does something, but my packages build.
>>
>> Please let me know if I've missed anything. On it's own, I don't think
>> this patch will do much (apart from maybe break HVM hosts with
>> e820_host=1 set).
>>
>>>>> 2). Figure out whether the E820 hypercall (which sets the E820
>>>>>     layout for a guest) can be run on HVM guests. I think it
>>>>>     could not and Mukesh in his PVH patches posted a patch
>>>>>     to enable that - "..Move e820 fields out of pv_domain struct"
>>
>> Is this already in 4.3.0 or is this an out-of-tree patch? Do you have a
>> link to it handy?
>>
>>>>> 2). Hvmloader should do an E820 get machine memory hypercall
>>>>>    to see if there is anything there. If there is - that means
>>>>>     the toolstack has request a "new" type of E820. Iterate
>>>>>     over the E820 and make it look like that.
>>>>>     You can look in the Linux arch/x86/xen/setup.c to see how
>>>>>     it does that.
>>>>>
>>>>>    The complication there is that hvmloader needs to to fit the
>>>>>    ACPI code (the guest type one) and such.
>>>>>    Presumarily you can just re-use the existing spaces that
>>>>>    the host has marked as E820_RESERVED or E820_ACPI..
>>>>
>>>> Yup, I get it. Not only that, but it should also ideally (not
>>>> strictly necessary, but it'd be handy) map the IOMEM for devices
>>>> it is passed so that pBAR=vBAR (as opposed to just leaving all
>>>> the host e820 reserved areas well alone - which would work for
>>>> most things).
>>>
>>> Yes. That is an extra complication that could be done in subsequent
>>> patches. But in theory if you have the E820 mirrored from the host the
>>> pBAR=vBAR should be easy enough as the values from the host BARs can
>>> easily fit in the E820 gaps.
>>
>> Agreed. Let's leave the pBAR=vBAR part for a separate patch set. I'll
>> have to figure out a sensible way to query the IOMEM regions for each of
>> the devices passed to the VM and make sure they are in the same hole.
>>
>>>>>    Then there is the SMBIOS would need to move and the BIOS
>>>>>    might need to be relocated - but I think those are relocatable
>>>>>   in some form.
>>
>> [bit above left for later reference]
>>
>>>>> Well, I am more than happy to help you with this.
>>>>
>>>> Thanks, much appreciated. :)
>>>
>>> Yeeey! Vict^H^H^H^volunteer :-)! <manically laughter in the background>
>>>
>>> I am also reachable on IRC (FreeNode mostly) as either darnok or konrad
>>> if that would be more convient to discuss this.
>>
>> Thanks. I'll keep that in mind. :)
>>
>> Gordan
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>>
>