[LSF/MM TOPIC] Pipe + splice problems

[LSF/MM TOPIC] Pipe + splice problems

[Pavel Emelyanov]

Hi,

While working on the checkpoint-restore project and looking at
what other Parallels people do with FUSE, I have met several
drawbacks of pipes. I admit, that most of them are well known for
years, but would like to summarize them and refresh the topic.

So the major problems are

* Pipe as page buffer lacks the random access feature

  In FUSE-based FS-es like CEPH or Gluster with complex internal
  data flows it's common case when data arrive in pipe, but FS
  is willing to forward the pages from pipe tail into one descriptor
  while keeping those from head in memory waiting for unblocked
  another. Pipes do not allow this.

* Pipe's locking is one big mutex

  For pipes with more than one page in buffers this results in
  readers-vs-writers contention and, subsequently, pipe works
  slower than a UNIX socket

* The pipe->mem vmsplice always copies data

  There are cases in C/R when we have pages in pipes that cane
  be mapped in tasks' address spaces, but vmslice doesn't allow
  for that.

* No pipe -> AIO splicing

  The pipe -> FS always goes through page cache, while AIO is
  more preferable in some scenarios

* No fallocate analogue for pipe

  People report many calls to __alloc_page in profiling logs when 
  heavily working with pipes


And a couple of minor issues

* Pipe requires 2 FDs to work with

  With this using pipe as generic page-buffer is difficult due to
  nr_files limitation.

* No sendpage for UNIX sockets results in pipe->unix data copy


Thanks,
Pavel