Re: Bug: Limitation of <=2GB RAM in domU persists with 4.3.0

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Tue, Sep 03, 2013 at 02:53:06PM +0100, Gordan Bobic wrote:
> On Mon, 29 Jul 2013 14:04:31 -0400, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> 
> Hi Konrad,
> 
> Apologies it took me a month to get back to this.

Hey Gordan,

That is OK. Time flies fast!
> 
> >>2) Further, I'm finding myself motivated to write that
> >>auto-set (as opposed to hard coded) vBAR=pBAR patch discussed
> >>briefly a week or so ago (have an init script read the BAR
> >>info from dom0 and put it in xenstore, plus a patch to
> >>make pBAR=vBAR reservations built dynamically rather than
> >>statically, based on this data. Now, I'm quite fluent in C,
> >>but my familiarity with Xen soruce code is nearly non-existant
> >>(limited to studying an old unsupported patch every now and then
> >>in order to make it apply to a more recent code release).
> >>Can anyone help me out with a high level view WRT where
> >>this would be best plumbed in (which files and the flow of
> >>control between the affected files)?
> >
> >hvmloader probably and the libxl e820 code. What from a
> >high view needs to happen is that:
> > 1). Need to relax the check in libxl for e820_hole
> >    to also do it for HVM guests. Said code just iterates over the
> >    host E820 and sanitizes it a bit and makes a E820 hypercall to
> >    set it for the guest.
> 
> I'm looking at the libxl code at the moment.
> 
> In cases where e820_host is seen as PV specific, would the
> correct thing to do be to move it out of the PV/HVM specific
> blocks so it applies to both?

Yes.
> 
> In libxl/libxl_x86.c/libxl__e820_alloc
> 
> I have thus far changed the code to remove the PV check,
> and having moved e820_host option to be common to both VM
> types, I changed the 820 related instances from
> b_info->u.pv.e820_host
> to
> b_info->e820_host
> 
> Is this the correct/preferred way this should be handled?

Yes.
> Or would it be better to make e820_host be in both PV and
> HVM options, and refer to it as such
> (u.pv.e820_host / u.hvm.e820_host) ?

No. Lets make it work across the board.
> 
> The e820 sanitizer is called with b_info->u.pv.slack_memkb
> parameter. What does parameter actually mean? I googled
> it and couldn't find any documentation specific to it, and
> it doesn't appear to be documented as settable in the config
> file. What would the equivalent be in case of HVM?

0.

If my memory serves me right it is  some amount of memory that
a PV guests that it does not use normally. It is used by the
frontend and backend driver to communicate. Kind of like a shadow
memory. But only ancient kernels use it but those still have to be
supported.
> 
> > 2). Figure out whether the E820 hypercall (which sets the E820
> >    layout for a guest) can be run on HVM guests. I think it
> >    could not and Mukesh in his PVH patches posted a patch
> >    to enable that - "..Move e820 fields out of pv_domain struct"
> > 2). Hvmloader should do an E820 get machine memory hypercall
> >   to see if there is anything there. If there is - that means
> >    the toolstack has request a "new" type of E820. Iterate
> >    over the E820 and make it look like that.
> >    You can look in the Linux arch/x86/xen/setup.c to see how
> >    it does that.
> >
> >   The complication there is that hvmloader needs to to fit the
> >   ACPI code (the guest type one) and such.
> >   Presumarily you can just re-use the existing spaces that
> >   the host has marked as E820_RESERVED or E820_ACPI..
> 
> Yup, I get it. Not only that, but it should also ideally (not
> strictly necessary, but it'd be handy) map the IOMEM for devices
> it is passed so that pBAR=vBAR (as opposed to just leaving all
> the host e820 reserved areas well alone - which would work for
> most things).

Yes. That is an extra complication that could be done in subsequent
patches. But in theory if you have the E820 mirrored from the host the
pBAR=vBAR should be easy enough as the values from the host BARs can
easily fit in the E820 gaps.

> 
> >   Then there is the SMBIOS would need to move and the BIOS
> >   might need to be relocated - but I think those are relocatable
> >  in some form.
> 
> OK, I'll look at that once I have a workable patch for the libxl
> part.

Aye.
> 
> >>The added bonus of this (if it can be made to work) is that
> >>it might just make unmodified GeForce cards work, too,
> >>which probably makes it worthwhile on it's own.
> >
> >Well, I am more than happy to help you with this.
> 
> Thanks, much appreciated. :)

Yeeey! Vict^H^H^H^volunteer :-)! <manically laughter in the background>

I am also reachable on IRC (FreeNode mostly) as either darnok or konrad
if that would be more convient to discuss this.
> 
> Gordan