3GB memory restriction for pv domU with PCI

3GB memory restriction for pv domU with PCI

[Bruce Edge]

On Fri, Jun 11, 2010 at 6:38 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Fri, Jun 11, 2010 at 09:09:36AM +0300, Pasi Kärkkäinen wrote:
>> On Thu, Jun 10, 2010 at 04:25:54PM -0700, Bruce Edge wrote:

>> > Here's the domU boot log:

>> >         memory       = 4096
>
> And also there is a bug where we can't do PCI in PV guests when guest
> has more than 3GB.
>

Is there any expectation that this 3GB restriction will be resolved in
the short term ?

Does anyone have any pointers to the specific problem ?

Thanks

-Bruce

Re: 3GB memory restriction for pv domU with PCI

[Konrad Rzeszutek Wilk]

On Fri, Jun 11, 2010 at 08:54:05AM -0700, Bruce Edge wrote:
> On Fri, Jun 11, 2010 at 6:38 AM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > On Fri, Jun 11, 2010 at 09:09:36AM +0300, Pasi Kärkkäinen wrote:
> >> On Thu, Jun 10, 2010 at 04:25:54PM -0700, Bruce Edge wrote:
> 
> >> > Here's the domU boot log:
> 
> >> >         memory       = 4096
> >
> > And also there is a bug where we can't do PCI in PV guests when guest
> > has more than 3GB.
> >
> 
> Is there any expectation that this 3GB restriction will be resolved in
> the short term ?

What is "short-term" ? This week = no. This month = likely.

> 
> Does anyone have any pointers to the specific problem ?

http://old.nabble.com/-PATCH-More-PCIFront--td26273977.html
http://lists.xensource.com/archives/html/xen-devel/2009-12/msg01112.html

Re: 3GB memory restriction for pv domU with PCI

[Bruce Edge]

On Fri, Jun 11, 2010 at 9:42 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Fri, Jun 11, 2010 at 08:54:05AM -0700, Bruce Edge wrote:
>> On Fri, Jun 11, 2010 at 6:38 AM, Konrad Rzeszutek Wilk
>> <konrad.wilk@oracle.com> wrote:
>> > On Fri, Jun 11, 2010 at 09:09:36AM +0300, Pasi Kärkkäinen wrote:
>> >> On Thu, Jun 10, 2010 at 04:25:54PM -0700, Bruce Edge wrote:
>>
>> >> > Here's the domU boot log:
>>
>> >> >         memory       = 4096
>> >
>> > And also there is a bug where we can't do PCI in PV guests when guest
>> > has more than 3GB.
>> >
>>
>> Is there any expectation that this 3GB restriction will be resolved in
>> the short term ?
>
> What is "short-term" ? This week = no. This month = likely.

It looks like this is still a restriction. Is there any plan to allow
pvops domU's to use more than 4GB ?

How much and who do we have to pay to get it in sooner?
If this ends up being our only show stopper for pvops, I'll start
selling lab equipment out the back door to pay for it. :-)

-Bruce

>
>>
>> Does anyone have any pointers to the specific problem ?
>
> http://old.nabble.com/-PATCH-More-PCIFront--td26273977.html
> http://lists.xensource.com/archives/html/xen-devel/2009-12/msg01112.html
>

Re: 3GB memory restriction for pv domU with PCI

[Konrad Rzeszutek Wilk]

On Wed, Oct 13, 2010 at 05:10:32PM -0700, Bruce Edge wrote:
> On Fri, Jun 11, 2010 at 9:42 AM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > On Fri, Jun 11, 2010 at 08:54:05AM -0700, Bruce Edge wrote:
> >> On Fri, Jun 11, 2010 at 6:38 AM, Konrad Rzeszutek Wilk
> >> <konrad.wilk@oracle.com> wrote:
> >> > On Fri, Jun 11, 2010 at 09:09:36AM +0300, Pasi Kärkkäinen wrote:
> >> >> On Thu, Jun 10, 2010 at 04:25:54PM -0700, Bruce Edge wrote:
> >>
> >> >> > Here's the domU boot log:
> >>
> >> >> >         memory       = 4096
> >> >
> >> > And also there is a bug where we can't do PCI in PV guests when guest
> >> > has more than 3GB.
> >> >
> >>
> >> Is there any expectation that this 3GB restriction will be resolved in
> >> the short term ?
> >
> > What is "short-term" ? This week = no. This month = likely.
> 
> It looks like this is still a restriction. Is there any plan to allow
> pvops domU's to use more than 4GB ?

Jeremy wrote up some new shinny code where the memory can be fragmented
(so that we can now stick in a big PCI hole). His patches are in
xen/balloon. But I haven't taken those and fiddled with them and changed
the toolstack (xm or xl) to actually take advantage of this.

> 
> How much and who do we have to pay to get it in sooner?
> If this ends up being our only show stopper for pvops, I'll start
> selling lab equipment out the back door to pay for it. :-)

Sheeshh. you are missing the mark. Instead of selling the lab equipment
you _send_ the equipment - that is like a X-mas present for geeks: shiny
new toy! :-)

But jokes aside, I haven't touched this since this last email. And I am
not going to get to it this week nor the next. Then there is one week
where (25th->29th) where I can play with this. After that, Linux Kernel
Summit and the Linux Plumbers Conference is in - so it will be quiet.

If this is a show-stopper right now, you could come with a hack where
the resource check is just ignored.

Re: 3GB memory restriction for pv domU with PCI

[Bruce Edge]

On Thu, Oct 14, 2010 at 7:23 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Wed, Oct 13, 2010 at 05:10:32PM -0700, Bruce Edge wrote:
>> On Fri, Jun 11, 2010 at 9:42 AM, Konrad Rzeszutek Wilk
>> <konrad.wilk@oracle.com> wrote:
>> > On Fri, Jun 11, 2010 at 08:54:05AM -0700, Bruce Edge wrote:
>> >> On Fri, Jun 11, 2010 at 6:38 AM, Konrad Rzeszutek Wilk
>> >> <konrad.wilk@oracle.com> wrote:
>> >> > On Fri, Jun 11, 2010 at 09:09:36AM +0300, Pasi Kärkkäinen wrote:
>> >> >> On Thu, Jun 10, 2010 at 04:25:54PM -0700, Bruce Edge wrote:
>> >>
>> >> >> > Here's the domU boot log:
>> >>
>> >> >> >         memory       = 4096
>> >> >
>> >> > And also there is a bug where we can't do PCI in PV guests when guest
>> >> > has more than 3GB.
>> >> >
>> >>
>> >> Is there any expectation that this 3GB restriction will be resolved in
>> >> the short term ?
>> >
>> > What is "short-term" ? This week = no. This month = likely.
>>
>> It looks like this is still a restriction. Is there any plan to allow
>> pvops domU's to use more than 4GB ?
>
> Jeremy wrote up some new shinny code where the memory can be fragmented
> (so that we can now stick in a big PCI hole). His patches are in
> xen/balloon. But I haven't taken those and fiddled with them and changed
> the toolstack (xm or xl) to actually take advantage of this.
>
>>
>> How much and who do we have to pay to get it in sooner?
>> If this ends up being our only show stopper for pvops, I'll start
>> selling lab equipment out the back door to pay for it. :-)
>
> Sheeshh. you are missing the mark. Instead of selling the lab equipment
> you _send_ the equipment - that is like a X-mas present for geeks: shiny
> new toy! :-)
>
> But jokes aside, I haven't touched this since this last email. And I am
> not going to get to it this week nor the next. Then there is one week
> where (25th->29th) where I can play with this. After that, Linux Kernel
> Summit and the Linux Plumbers Conference is in - so it will be quiet.

It's not the last nail in the coffin right now, but it will be if it's
not in there by the end of the year at the latest.
We'll have to fallback to hvm which no one wants to do because we'll
lose all the new goodies that come along with pvops.

We are in a pv-ops vs. hvm showdown mode right now. The remaining
sticking points are this and the 2MB contig alloc for the tachyon 2 MB
block for the FC SEST. I feel like if we don't get pvops in now, it'll
be a hard sell later on.

Anyway, thanks for the info and I'll keep my fingers crossed that 5
days is all it takes.

>
> If this is a show-stopper right now, you could come with a hack where
> the resource check is just ignored.
>
I assume one could do this only if not using PCI passthrough?

-Bruce

Re: 3GB memory restriction for pv domU with PCI

[Konrad Rzeszutek Wilk]

> sticking points are this and the 2MB contig alloc for the tachyon 2 MB
> block for the FC SEST. I feel like if we don't get pvops in now, it'll
> be a hard sell later on.

The 2MB contig alloc ought to work with the pci_alloc_coherent function.

> 
> Anyway, thanks for the info and I'll keep my fingers crossed that 5
> days is all it takes.
> 
> >
> > If this is a show-stopper right now, you could come with a hack where
> > the resource check is just ignored.
> >
> I assume one could do this only if not using PCI passthrough?

Well, it is basically relaxing the criteria. In the olden days (2.6.18)
the System RAM region wa not accounted for. Nowadays it is. So when
a PCI device requests a region that the kernel thinks is System RAM
it tells it to buzz of. In 2.6.18, the kernel would have no trouble
giving you that region. So the hack is just to relax the criteria
and maybe just do:

 if (strncmp(r->name,"System RAM") == 0) {
	return OK;
 }

Or something like that in the resource.c file. I do't remember the
details of that file, but that looked like one of those quick-and-fast
ways of doing this. The right way is to actually make the guest aware of the
PCI hole the Dom0 has and just shift the memory as required. It sounds
easy :-)