Re: [fuse-devel] [PATCH] FUSE/CUSE: implement direct mmap support

[Tejun Heo <tj@kernel.org>]

Hello, Miklos.

On 02/09/2010 11:59 PM, Miklos Szeredi wrote:
> On Tue, 09 Feb 2010, Tejun Heo wrote:
> Okay, I'm a bit confused about these offsets.
> 
> Client asks to map a file at an offset.  Server receives offset, may
> change it (but only by multiple of SHMLBA) then returns it to the
> kernel.  The returned offset globally identifies not only the mapped
> region but the page within the region.  Sounds neat.
> 
> But then fuse_do_mmap() goes and changes vma->vm_pgoff, which will
> show up in /proc/PID/maps for example, which is really not nice.
>
> Can't this page ID rather be put in vma->vm_private_data?

Yeah, sure, it can be put anywhere.  I just never thought about it
being visible under /proc.  Will move it inside fuse_dmmap_vm.

> Also can we take this page ID abstraction a step further, and say that
> the ID has nothing to do with the original offset, the only
> requirement is that it'd globally identify all direct mapped pages.
> 
> And the coherency requirements would be satisfied by the
> fuse_dev_mmap() code.  Haven't looked into what that would take, but
> it sounds doable.

The coherency requirement is not really between the address and offset
but virtual addresses different maps which may end up sharing the same
physical page.  On an architecture with virtually indexed but
physically tagged caches, virtual addresses which map to the same page
must end up hitting the same cache group; otherwise, the processor
wouldn't be able to determine whether the two addresses point to the
same page (as only the tags inside the same cache group are matched
against the physical address).

This is achieved by enforcing address to offset alignment.  IOW, all
maps are forced to be SHMLBA aligned to offset so that all maps are
SHMLBA aligned to each other.  And without knowing which maps are
gonna end up sharing which pages beforehand, enforcing SHMLBA
alignment against something is pretty much the only way to achieve
this and that's the reason why something as low level as processor
cache orgnization is exported to userland as SHMLBA for shared memory
in the first place.

FUSE dmmap has exactly the same problem as it basically is a shared
memory implementation with slightly different interface.  Unless we
can know which maps are gonna be shared beforehand, and we can't know
this either for between the client and server or between clients, the
only way to make those shared mappings fall into the same cache slot
is aligning them to address space offsets.

So, I don't think it's feasible to do the address matching from inside
the kernel without a lot of convolution.  The only way to do it would
be adjusting the mapping addresses but this has at least two problems
- 1. the VM layer isn't made that way and virtual address is
determined by vm generic and arch code before ->mmap() is invoked and
can't be changed after that.  2. the SHMLBA alignment is good in the
sense that it gives the userland something to rely on when determining
mapping address for address hint or fixed mapping.  I don't think it
would be wise to break such assumptions which hold for *ALL* existing
memory mappings.

Thanks.

-- 
tejun