Re: HVM support for e820_host (Was: Bug: Limitation of <=2GB RAM in domU persists with 4.3.0)

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Tue, Sep 03, 2013 at 09:49:40PM +0100, Gordan Bobic wrote:
> I spoke too soon - even with e820_host=0, the same error occurs.
> What did I break? The code in question is this:
> 
> if (libxl_defbool_val(d_config->b_info.e820_host)) {
>     ret = libxl__e820_alloc(gc, domid, d_config);
>     if (ret) {
>         LIBXL__LOG_ERRNO(gc->owner, LIBXL__LOG_ERROR,
>                 "Failed while collecting E820 with: %d (errno:%d)\n",
>                 ret, errno);
>     }
> }
> 
> With e820_host=0, that outer black should evaluate to false, should
> it not? In libxl_create.c, if I am understanding the code correctly,
> e820_host is defaulted to false, too. What am I missing?

Just sent you an email but I believe what is failing is:

241     rc = xc_domain_set_memory_map(ctx->xch, domid, map, nr);                    

You can add some extra LIBXL__LOG_ERRNO to check each 'rc' to see
which one of them failed.

Hm, perhaps it might make sense to actually have the libxl__e820_alloc
also use the LIBXL__LOG_ERRNO to log more details..

> 
> Gordan
> 
> On 09/03/2013 09:35 PM, Gordan Bobic wrote:
> >First attempt at a test run predictably failed. I added e820_host=1 to a
> >VM config and tried starting it:
> >
> >[root@normandy ~]# xl create /etc/xen/edi
> >Parsing config from /etc/xen/edi
> >libxl: error: libxl_x86.c:307:libxl__arch_domain_create: Failed while
> >collecting E820 with: -3 (errno:-1)
> >
> >libxl: error: libxl_create.c:901:domcreate_rebuild_done: cannot
> >(re-)build domain: -3
> >libxl: error: libxl_dm.c:1300:libxl__destroy_device_model: could not
> >find device-model's pid for dom 1
> >libxl: error: libxl.c:1415:libxl__destroy_domid:
> >libxl__destroy_device_model failed for 1
> >
> >xl-edi.log, qemu-dm-edi.log attached.
> >Both actually look identical to previous logs before the patch.
> >
> >Is this something that is clearly a consequence of the patch being
> >incomplete? Or did I break something?
> >
> >Gordan
> >
> >On 09/03/2013 08:47 PM, Gordan Bobic wrote:
> >>On 09/03/2013 03:59 PM, Konrad Rzeszutek Wilk wrote:
> >>
> >>>>>>2) Further, I'm finding myself motivated to write that
> >>>>>>auto-set (as opposed to hard coded) vBAR=pBAR patch discussed
> >>>>>>briefly a week or so ago (have an init script read the BAR
> >>>>>>info from dom0 and put it in xenstore, plus a patch to
> >>>>>>make pBAR=vBAR reservations built dynamically rather than
> >>>>>>statically, based on this data. Now, I'm quite fluent in C,
> >>>>>>but my familiarity with Xen soruce code is nearly non-existant
> >>>>>>(limited to studying an old unsupported patch every now and then
> >>>>>>in order to make it apply to a more recent code release).
> >>>>>>Can anyone help me out with a high level view WRT where
> >>>>>>this would be best plumbed in (which files and the flow of
> >>>>>>control between the affected files)?
> >>>>>
> >>>>>hvmloader probably and the libxl e820 code. What from a
> >>>>>high view needs to happen is that:
> >>>>>1). Need to relax the check in libxl for e820_hole
> >>>>>    to also do it for HVM guests. Said code just iterates over the
> >>>>>    host E820 and sanitizes it a bit and makes a E820 hypercall to
> >>>>>    set it for the guest.
> >>[snip]
> >>
> >>OK, I have attached a preliminary patch against 4.3.0 for the libxl
> >>part. It compiles. I haven't tried running it to see if it actually
> >>works or does something, but my packages build.
> >>
> >>Please let me know if I've missed anything. On it's own, I don't think
> >>this patch will do much (apart from maybe break HVM hosts with
> >>e820_host=1 set).
> >>
> >>>>>2). Figure out whether the E820 hypercall (which sets the E820
> >>>>>    layout for a guest) can be run on HVM guests. I think it
> >>>>>    could not and Mukesh in his PVH patches posted a patch
> >>>>>    to enable that - "..Move e820 fields out of pv_domain struct"
> >>
> >>Is this already in 4.3.0 or is this an out-of-tree patch? Do you have a
> >>link to it handy?
> >>
> >>>>>2). Hvmloader should do an E820 get machine memory hypercall
> >>>>>   to see if there is anything there. If there is - that means
> >>>>>    the toolstack has request a "new" type of E820. Iterate
> >>>>>    over the E820 and make it look like that.
> >>>>>    You can look in the Linux arch/x86/xen/setup.c to see how
> >>>>>    it does that.
> >>>>>
> >>>>>   The complication there is that hvmloader needs to to fit the
> >>>>>   ACPI code (the guest type one) and such.
> >>>>>   Presumarily you can just re-use the existing spaces that
> >>>>>   the host has marked as E820_RESERVED or E820_ACPI..
> >>>>
> >>>>Yup, I get it. Not only that, but it should also ideally (not
> >>>>strictly necessary, but it'd be handy) map the IOMEM for devices
> >>>>it is passed so that pBAR=vBAR (as opposed to just leaving all
> >>>>the host e820 reserved areas well alone - which would work for
> >>>>most things).
> >>>
> >>>Yes. That is an extra complication that could be done in subsequent
> >>>patches. But in theory if you have the E820 mirrored from the host the
> >>>pBAR=vBAR should be easy enough as the values from the host BARs can
> >>>easily fit in the E820 gaps.
> >>
> >>Agreed. Let's leave the pBAR=vBAR part for a separate patch set. I'll
> >>have to figure out a sensible way to query the IOMEM regions for each of
> >>the devices passed to the VM and make sure they are in the same hole.
> >>
> >>>>>   Then there is the SMBIOS would need to move and the BIOS
> >>>>>   might need to be relocated - but I think those are relocatable
> >>>>>  in some form.
> >>
> >>[bit above left for later reference]
> >>
> >>>>>Well, I am more than happy to help you with this.
> >>>>
> >>>>Thanks, much appreciated. :)
> >>>
> >>>Yeeey! Vict^H^H^H^volunteer :-)! <manically laughter in the background>
> >>>
> >>>I am also reachable on IRC (FreeNode mostly) as either darnok or konrad
> >>>if that would be more convient to discuss this.
> >>
> >>Thanks. I'll keep that in mind. :)
> >>
> >>Gordan
> >>
> >>
> >>_______________________________________________
> >>Xen-devel mailing list
> >>Xen-devel@lists.xen.org
> >>http://lists.xen.org/xen-devel
> >>
> >
>