Re: [PATCH 00/20] Add support for shared PTEs across processes

[David Hildenbrand <david@redhat.com>]

> API
> ===
> 
> mshare does not introduce a new API. It instead uses existing APIs
> to implement page table sharing. The steps to use this feature are:
> 
> 1. Mount msharefs on /sys/fs/mshare -
>          mount -t msharefs msharefs /sys/fs/mshare
> 
> 2. mshare regions have alignment and size requirements. Start
>     address for the region must be aligned to an address boundary and
>     be a multiple of fixed size. This alignment and size requirement
>     can be obtained by reading the file /sys/fs/mshare/mshare_info
>     which returns a number in text format. mshare regions must be
>     aligned to this boundary and be a multiple of this size.
> 
> 3. For the process creating an mshare region:
>          a. Create a file on /sys/fs/mshare, for example -
>                  fd = open("/sys/fs/mshare/shareme",
>                                  O_RDWR|O_CREAT|O_EXCL, 0600);
> 
>          b. Establish the starting address and size of the region
>                  struct mshare_info minfo;
> 
>                  minfo.start = TB(2);
>                  minfo.size = BUFFER_SIZE;
>                  ioctl(fd, MSHAREFS_SET_SIZE, &minfo)

We could set the size using ftruncate, just like for any other file. It 
would have to be the first thing after creating the file, and before we 
allow any other modifications.

Idealy, we'd be able to get rid of the "start", use something resaonable 
(e.g., TB(2)) internally, and allow processes to mmap() it at different 
(suitably-aligned) addresses.

I recall we discussed that in the past. Did you stumble over real 
blockers such that we really must mmap() the file at the same address in 
all processes? I recall some things around TLB flushing, but not sure. 
So we might have to stick to an mmap address for now.

When using fallocate/stat to set/query the file size, we could end up with:

/*
  * Set the address where this file can be mapped into processes. Other
  * addresses are not supported for now, and mmap will fail. Changing the
  * mmap address after mappings were already created is not supported.
  */
MSHAREFS_SET_MMAP_ADDRESS
MSHAREFS_GET_MMAP_ADDRESS


> 
>          c. Map some memory in the region
>                  struct mshare_create mcreate;
> 
>                  mcreate.addr = TB(2);

Can we use the offset into the virtual file instead? We should be able 
to perform that translation internally fairly easily I assume.

>                  mcreate.size = BUFFER_SIZE;
>                  mcreate.offset = 0;
>                  mcreate.prot = PROT_READ | PROT_WRITE;
>                  mcreate.flags = MAP_ANONYMOUS | MAP_SHARED | MAP_FIXED;
>                  mcreate.fd = -1;
> 
>                  ioctl(fd, MSHAREFS_CREATE_MAPPING, &mcreate)

Would examples with multiple mappings work already in this version?

Did you experiment with other mappings (e.g., ordinary shared file 
mappings), and what are the blockers to make that fly?

> 
>          d. Map the mshare region into the process
>                  mmap((void *)TB(2), BUF_SIZE, PROT_READ | PROT_WRITE,
>                          MAP_SHARED, fd, 0);
> 
>          e. Write and read to mshared region normally.
> 
> 4. For processes attaching an mshare region:
>          a. Open the file on msharefs, for example -
>                  fd = open("/sys/fs/mshare/shareme", O_RDWR);
> 
>          b. Get information about mshare'd region from the file:
>                  struct mshare_info minfo;
> 
>                  ioctl(fd, MSHAREFS_GET_SIZE, &minfo);
> 
>          c. Map the mshare'd region into the process
>                  mmap(minfo.start, minfo.size,
>                          PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
> 
> 5. To delete the mshare region -
>                  unlink("/sys/fs/mshare/shareme");
> 

I recall discussions around cgroup accounting, OOM handling etc. I 
thought the conclusion was that we need an "mshare process" where the 
memory is accounted to, and once that process is killed (e.g., OOM), it 
must tear down all mappings/pages etc.

How does your design currently look like in that regard? E.g., how can 
OOM handling make progress, how is cgroup accounting handled?

-- 
Cheers,

David / dhildenb