Re: [Resend] Cross Memory Attach v3 [PATCH]

[Christopher Yeoh <cyeoh@au1.ibm.com>]

On Sun, 20 Mar 2011 18:55:32 -0700
Andrew Morton <akpm@linux-foundation.org> wrote:
> On Mon, 21 Mar 2011 12:20:18 +1030 Christopher Yeoh
> <cyeoh@au1.ibm.com> wrote:
> 
> > On Thu, 17 Mar 2011 12:54:27 -0700
> > Andrew Morton <akpm@linux-foundation.org> wrote:
> > > On Thu, 17 Mar 2011 15:40:26 +1030
> > > Christopher Yeoh <cyeoh@au1.ibm.com> wrote:
> > > 
> > > > > Thinking out loud: if we had a way in which a process can add
> > > > > and remove a local anonymous page into pagecache then other
> > > > > processes could access that page via mmap.  If both processes
> > > > > map the file with a nonlinear vma they they can happily sit
> > > > > there flipping pages into and out of the shared mmap at
> > > > > arbitrary file offsets. The details might get hairy ;) We
> > > > > wouldn't want all the regular mmap semantics of
> > > > 
> > > > Yea, its the complexity of trying to do it that way that
> > > > eventually lead me to implementing it via a syscall and
> > > > get_user_pages instead, trying to keep things as simple as
> > > > possible.
> > > 
> > > The pagecache trick potentially gives zero-copy access, whereas
> > > the proposed code is single-copy.  Although the expected benefits
> > > of that may not be so great due to TLB manipulation overheads.
> > > 
> > > I worry that one day someone will come along and implement the
> > > pagecache trick, then we're stuck with obsolete code which we
> > > have to maintain for ever.
> > 
> > Perhaps I don't understand what you're saying correctly but I think
> > that one problem with the zero copy page flipping approach is that
> > there is no guarantee with the data that the MPI apps want to send 
> > resides in a page or pages all by itself.
> 
> Well.  The applications could of course be changed.  But if the
> applications are changeable then they could be changed to use
> MAP_SHARED memory sharing and we wouldn't be having this discussion,
> yes?

Yup, the applications can't be changed.

> But yes, I'm assuming that it will be acceptable for the sending app
> to expose some memory (up to PAGE_SIZE-1) below and above the actual
> payload which is to be transferred.

So in addition to this restriction and the TLB manipulation overhead
you mention, I believe that in practice if you need to use the data soon
(as opposed to just sending it out a network interface for example)
then the gain you get for zero copy vs single copy is not as high as
you might expect except for quite large sizes of data. The reason being
that that with page flipping the data will be cache cold whereas if you
have done a single copy it will be hot.

Rusty (CC'd) has experience in this area and can explain it better than
me :-)

My feeling is that waiting for a perfect solution (which has its own
problems such as the page size/alignment restrictions and high
complexity for implementation) we'll be putting off a good solution for
a long time.

Regards,

Chris
-- 
cyeoh@au.ibm.com

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>