tmem ephemeral page discarding

tmem ephemeral page discarding

[James Harper]

How does Xen manage ephemeral pages? Is there a way to manage the priority of pages in the ephemeral store? In my use case, I would prefer that an old page was automatically discarded to make room for a new page, and that pages that have been retrieved already are more likely to be discarded than a page that has not yet been retrieved.

Thanks

James

Re: tmem ephemeral page discarding

[Konrad Rzeszutek Wilk]

On Sun, Jun 02, 2013 at 04:06:18AM +0000, James Harper wrote:
> How does Xen manage ephemeral pages? Is there a way to manage the priority of pages in the ephemeral store? In my use case, I would prefer that an old page was automatically discarded to make room for a new page, and that pages that have been retrieved already are more likely to be discarded than a page that has not yet been retrieved.
> 

In other words an LRU system with bounds. Right now the bounds are the amount of memory available and 
when Xen needs more memory the bounds gets decreased - which will discard the least used ephermeal pages.

There is a tmem subop to actually set the cap (TMEMC_SET_CAP), which sets the cap value. However
I am not seeing in the code anybody using it. Hmmm

> Thanks
> 
> James
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: tmem ephemeral page discarding

[Dan Magenheimer]

> From: Konrad Rzeszutek Wilk
> Sent: Monday, June 03, 2013 6:22 AM
> To: James Harper
> Cc: xen-devel@lists.xen.org; Dan Magenheimer
> Subject: Re: [Xen-devel] tmem ephemeral page discarding
> 
> On Sun, Jun 02, 2013 at 04:06:18AM +0000, James Harper wrote:
> > How does Xen manage ephemeral pages? Is there a way to manage the priority of pages in the ephemeral
> store? In my use case, I would prefer that an old page was automatically discarded to make room for a
> new page, and that pages that have been retrieved already are more likely to be discarded than a page
> that has not yet been retrieved.
> 
> In other words an LRU system with bounds. Right now the bounds are the amount of memory available and
> when Xen needs more memory the bounds gets decreased - which will discard the least used ephermeal
> pages.

It sounds like Xen tmem matches your use case.  Ephemeral pages that have been retrieved
are actually removed from tmem, i.e. ephemeral gets are "destructive", i.e. an ephemeral
get implies a flush.  (Note: This isn't true for shared ephemeral pages, but I don't
think you care about clustered filesystems so ignore that case for now.)

And ephemeral pages (as Konrad replied) are kept in a LRU queue across all
(tmem-aware) guests and the LRU ephemeral page across all guests is
discarded.  Or if a weight limit is exceeded, the LRU ephemeral
page belonging to the current guest is discarded.

> There is a tmem subop to actually set the cap (TMEMC_SET_CAP), which sets the cap value. However
> I am not seeing in the code anybody using it. Hmmm

See discussion in "Weights and caps" in docs/misc/tmem-internals.html (or on the web at
https://oss.oracle.com/projects/tmem/dist/documentation/internals/xen4-internals-v01.html )

The current cap non-implementation is per-guest, but I never found
a good use model for it so never implemented it, outside of the
toolstack control interface.  Caps exist only because "weights+caps"
make sense for a CPU scheduler and I thought both might make sense
also for a "memory scheduler".

If you have a use model for it (or a use model for across-all-guests
cap), it shouldn't be hard to implement.

Hope that helps!
Dan

Re: tmem ephemeral page discarding

[James Harper]

> 
> And ephemeral pages (as Konrad replied) are kept in a LRU queue across all
> (tmem-aware) guests and the LRU ephemeral page across all guests is
> discarded.  Or if a weight limit is exceeded, the LRU ephemeral
> page belonging to the current guest is discarded.
> 

Yep that sounds about perfect.

I can't seem to get it working under 64 bit windows at the moment... keeps returning -ENOSYS and I'm wondering if windows and linux differ in their structure alignment or something. I remember this being a problem with vbd.

Btw, the choice of 'new' as a struct name in the tmem struct isn't necessarily wise... it might never come up but 'new' is a reserved word in c++.

James

Re: tmem ephemeral page discarding

[Dan Magenheimer]

> From: James Harper [mailto:james.harper@bendigoit.com.au]
> Sent: Monday, June 03, 2013 7:42 PM
> To: Dan Magenheimer; Konrad Wilk
> Cc: xen-devel@lists.xen.org
> Subject: RE: [Xen-devel] tmem ephemeral page discarding
> 
> >
> > And ephemeral pages (as Konrad replied) are kept in a LRU queue across all
> > (tmem-aware) guests and the LRU ephemeral page across all guests is
> > discarded.  Or if a weight limit is exceeded, the LRU ephemeral
> > page belonging to the current guest is discarded.
> >
> 
> Yep that sounds about perfect.
> 
> I can't seem to get it working under 64 bit windows at the moment... keeps returning -ENOSYS and I'm
> wondering if windows and linux differ in their structure alignment or something. I remember this being
> a problem with vbd.

Any chance you've not enabled it?  Tmem is disabled by default
and you need to provide "tmem" as a boot parameter to xen to
enable it.  Tmem calls from a guest will fail if it is
disabled.

> Btw, the choice of 'new' as a struct name in the tmem struct isn't necessarily wise... it might never
> come up but 'new' is a reserved word in c++.

Hmmm... Where are you seeing it?  I do see it in the tmem
code as a local parameter ("new" is used in code in other
places in Xen as well), but not in a public structure definition.

Thanks,
Dan

Re: tmem ephemeral page discarding

[James Harper]

> > I can't seem to get it working under 64 bit windows at the moment... keeps
> > returning -ENOSYS and I'm
> > wondering if windows and linux differ in their structure alignment or
> > something. I remember this being
> > a problem with vbd.
> 
> Any chance you've not enabled it?  Tmem is disabled by default
> and you need to provide "tmem" as a boot parameter to xen to
> enable it.  Tmem calls from a guest will fail if it is
> disabled.

I botched a cut&paste when adding the new x64 hypercall to GPLPV. Oops. All good now

> > Btw, the choice of 'new' as a struct name in the tmem struct isn't
> necessarily wise... it might never
> > come up but 'new' is a reserved word in c++.
> 
> Hmmm... Where are you seeing it?  I do see it in the tmem
> code as a local parameter ("new" is used in code in other
> places in Xen as well), but not in a public structure definition.
> 

I was looking at very old source and now I see it was resolved over 2 years ago http://xenbits.xen.org/hg/xen-unstable.hg/rev/0353037c6b95

Thanks

James