tmem docs

tmem docs

[James Harper]

Where is the best place to read about tmem hypercall abi? Should I start with the Linux implementation?

Thanks

James

Re: tmem docs

[Pasi Kärkkäinen]

On Mon, May 27, 2013 at 09:14:09AM +0000, James Harper wrote:
> Where is the best place to read about tmem hypercall abi? Should I start with the Linux implementation?
> 

I'm not really answering your question here.. but if you didn't see these links, 
take a look, they might help:
http://lists.xen.org/archives/html/xen-devel/2013-05/msg01892.html

-- Pasi

Re: tmem docs

[James Harper]

> On Mon, May 27, 2013 at 09:14:09AM +0000, James Harper wrote:
> > Where is the best place to read about tmem hypercall abi? Should I start
> > with the Linux implementation?
> >
> 
> I'm not really answering your question here.. but if you didn't see these links,
> take a look, they might help:
> http://lists.xen.org/archives/html/xen-devel/2013-05/msg01892.html
> 

That all tells me about how great tmem is, and some of the interfaces exposed by Linux, but I can't find anything about the interfaces exposed by Xen that Windows would need to use. It seems that the interface is fairly thin though, and I have a few Windows related problems to solve first before I can even think about making use of tmem.

Thanks

James

Re: tmem docs

[Konrad Rzeszutek Wilk]

On Tue, May 28, 2013 at 12:26:15AM +0000, James Harper wrote:
> > On Mon, May 27, 2013 at 09:14:09AM +0000, James Harper wrote:
> > > Where is the best place to read about tmem hypercall abi? Should I start
> > > with the Linux implementation?
> > >
> > 
> > I'm not really answering your question here.. but if you didn't see these links,
> > take a look, they might help:
> > http://lists.xen.org/archives/html/xen-devel/2013-05/msg01892.html
> > 
> 
> That all tells me about how great tmem is, and some of the interfaces exposed by Linux, but I can't find anything about the interfaces exposed by Xen that Windows would need to use. It seems that the interface is fairly thin though, and I have a few Windows related problems to solve first before I can even think about making use of tmem.

Let's CC Dan here. He has a stash of documents on that.

But looking at the include/public/tmem.h it is very slim on comments. The ones
that are of interest to you (I think) are:

TMEM_DESTROY_POOL, TMEM_NEW_POOL, TMEM_PUT_PAGE, and TMEM_GET_PAGE.

> 
> Thanks
> 
> James
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: tmem docs

[Dan Magenheimer]

> From: Konrad Rzeszutek Wilk
> Subject: Re: [Xen-devel] tmem docs
> 
> On Tue, May 28, 2013 at 12:26:15AM +0000, James Harper wrote:
> > > On Mon, May 27, 2013 at 09:14:09AM +0000, James Harper wrote:
> > > > Where is the best place to read about tmem hypercall abi? Should I start
> > > > with the Linux implementation?
> > >
> > > I'm not really answering your question here.. but if you didn't see these links,
> > > take a look, they might help:
> > > http://lists.xen.org/archives/html/xen-devel/2013-05/msg01892.html
> >
> > That all tells me about how great tmem is, and some of the interfaces exposed by Linux, but I can't
> find anything about the interfaces exposed by Xen that Windows would need to use. It seems that the
> interface is fairly thin though, and I have a few Windows related problems to solve first before I can
> even think about making use of tmem.
> 
> Let's CC Dan here. He has a stash of documents on that.
> 
> But looking at the include/public/tmem.h it is very slim on comments. The ones
> that are of interest to you (I think) are:
> 
> TMEM_DESTROY_POOL, TMEM_NEW_POOL, TMEM_PUT_PAGE, and TMEM_GET_PAGE.
> 
> > Thanks
> > James

Hi James --

I'll be working for a few more weeks and would be happy
to answer any questions I can.  However, I am no longer on the
xen-devel email list so you will need to cc me directly.

I'm gathering you are trying to connect a Windows kernel to the
Xen tmem ABI?  There have been various ideas proposed about
connecting up Linux _userland_ apps to tmem and it may be possible
to do that for Windows as well.  However, the existing ABI
is probably best suited for a guest kernel, and I've never
had access to Windows kernel source so I'm not in a position
to judge how easy or how hard it would be to modify Windows
kernel source to adapt it.  KY Srinivasan is currently working
on cross-guest memory capacity optimization for Linux on HyperV
and has expressed some interest in tmem for Windows so you
may want to contact him to see if he's given it any thought.
(kys@microsoft.com) KY was an early Xen contributor also.

The tmem ABI spec is here:

https://oss.oracle.com/projects/tmem/dist/documentation/api/tmemspec-v001.pdf 

As you can see by the date, that spec version is very old (2008)
but, on quick scan, it is still 99% accurate.  The main difference
I see is that (page 2, line 18) object identifiers are now
192-bit instead of 64-bit.  See reason for that change here:
http://lists.xen.org/archives/html/xen-devel/2010-09/msg00069.html 

> But looking at the include/public/tmem.h it is very slim on comments. The ones
> that are of interest to you (I think) are:
> TMEM_DESTROY_POOL, TMEM_NEW_POOL, TMEM_PUT_PAGE, and TMEM_GET_PAGE.

Also, TMEM_FLUSH_PAGE and TMEM_FLUSH_OBJECT.  Note that
TMEM_READ, TMEM_WRITE, TMEM_XCHG, and TMEM_NEW_PAGE were
never used by Linux.  Also TMEM_CONTROL is only described
by the implementation... it is only used by tools in dom0
so you may not care about it for Windows.  And you can
ignore shared+persistent pools (never implemented) and,
unless there is some cool Windows cluster-filesystem, you
can ignore shared+ephemeral pools too (only implemented
on linux for ocfs2).

The core of the linux implementation of the ABI can be found in
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/xen/tmem.c 
Look for the functions with name starting "xen_tmem_"
This is where the Linux concepts "cleancache" and "frontswap"
get translated into the Xen tmem ABI.

>> interface is fairly thin though

Yes, everything funnels through a single hypercall.

>> I have a few Windows related problems to solve first before I can

Let me know if/how I can help!  The hardest part is
probably proper placement of flush calls into the
OS kernel memory management code to ensure coherency.

Dan

P.S....

> > That all tells me about how great tmem is

Sorry about that... I had to spend far far too much time selling
the value of tmem at the cost of insufficient time improving it. :-(
And I suck at sales :-(

Re: tmem docs

[Konrad Rzeszutek Wilk]

On Tue, May 28, 2013 at 09:20:28AM -0700, Dan Magenheimer wrote:
> > From: Konrad Rzeszutek Wilk
> > Subject: Re: [Xen-devel] tmem docs
> > 
> > On Tue, May 28, 2013 at 12:26:15AM +0000, James Harper wrote:
> > > > On Mon, May 27, 2013 at 09:14:09AM +0000, James Harper wrote:
> > > > > Where is the best place to read about tmem hypercall abi? Should I start
> > > > > with the Linux implementation?
> > > >
> > > > I'm not really answering your question here.. but if you didn't see these links,
> > > > take a look, they might help:
> > > > http://lists.xen.org/archives/html/xen-devel/2013-05/msg01892.html
> > >
> > > That all tells me about how great tmem is, and some of the interfaces exposed by Linux, but I can't
> > find anything about the interfaces exposed by Xen that Windows would need to use. It seems that the
> > interface is fairly thin though, and I have a few Windows related problems to solve first before I can
> > even think about making use of tmem.
> > 
> > Let's CC Dan here. He has a stash of documents on that.
> > 
> > But looking at the include/public/tmem.h it is very slim on comments. The ones
> > that are of interest to you (I think) are:
> > 
> > TMEM_DESTROY_POOL, TMEM_NEW_POOL, TMEM_PUT_PAGE, and TMEM_GET_PAGE.
> > 
> > > Thanks
> > > James
> 
> Hi James --
> 
> I'll be working for a few more weeks and would be happy
> to answer any questions I can.  However, I am no longer on the
> xen-devel email list so you will need to cc me directly.
> 
> I'm gathering you are trying to connect a Windows kernel to the
> Xen tmem ABI?  There have been various ideas proposed about
> connecting up Linux _userland_ apps to tmem and it may be possible
> to do that for Windows as well.  However, the existing ABI
> is probably best suited for a guest kernel, and I've never
> had access to Windows kernel source so I'm not in a position
> to judge how easy or how hard it would be to modify Windows
> kernel source to adapt it.  KY Srinivasan is currently working
> on cross-guest memory capacity optimization for Linux on HyperV
> and has expressed some interest in tmem for Windows so you
> may want to contact him to see if he's given it any thought.
> (kys@microsoft.com) KY was an early Xen contributor also.
> 
> The tmem ABI spec is here:
> 
> https://oss.oracle.com/projects/tmem/dist/documentation/api/tmemspec-v001.pdf 

Awesome.
> 
> As you can see by the date, that spec version is very old (2008)
> but, on quick scan, it is still 99% accurate.  The main difference
> I see is that (page 2, line 18) object identifiers are now
> 192-bit instead of 64-bit.  See reason for that change here:
> http://lists.xen.org/archives/html/xen-devel/2010-09/msg00069.html 
> 

Is there a source for the spec so I can modify it (And also
put the spec in the Xen codebase).