Re: [Resend] Cross Memory Attach v3 [PATCH]

[Christopher Yeoh <cyeoh@au1.ibm.com>]

On Sun, 20 Mar 2011 18:55:32 -0700
Andrew Morton <akpm@linux-foundation.org> wrote:
> > > The pagecache trick potentially gives zero-copy access, whereas
> > > the proposed code is single-copy.  Although the expected benefits
> > > of that may not be so great due to TLB manipulation overheads.
> > > 
> > > I worry that one day someone will come along and implement the
> > > pagecache trick, then we're stuck with obsolete code which we
> > > have to maintain for ever.
> > 
> > Perhaps I don't understand what you're saying correctly but I think
> > that one problem with the zero copy page flipping approach is that
> > there is no guarantee with the data that the MPI apps want to send 
> > resides in a page or pages all by itself.
> 
> Well.  The applications could of course be changed.  But if the
> applications are changeable then they could be changed to use
> MAP_SHARED memory sharing and we wouldn't be having this discussion,
> yes?
> 
> (Why can't the applications be changed to use existing shared memory
> capabilities, btw?)

An MPI application commonly doesn't know in advance when allocating
memory if the data it will eventually be sending will be to a local
node or remote node process.  It will depend on the configuration of the
cluster that you run the application on and parameters when you start
it up (eg how many processes per node to start etc), and exactly how
the program ends up executing.

So short of allocating everything to be shared memory just in case you
want intranode communication we can't use shared memory
cooperatively like that to reduce copies. Shared memory *is*
often used for intranode communication, but in a copy-in to shared
memory on the sender and copy-out on the receiver side.

We did originally do some early hacking on hpcc where we did allocate
everything from a shared memory pool just to see what sort of
theoretical gain we could have from a single-copy model, but its not a
solution we can use in general.

Regards,

Chris
-- 
cyeoh@au.ibm.com

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>