Why "splice" doesn't splice between two non-pipes?

Why "splice" doesn't splice between two non-pipes?

[Askar Safin]

Why "splice" doesn't splice between two non-pipes? I think such splice would be very useful (not only file-to-socket). For example, for dd, cp, cp -r. Also, it seems, the kernel already supports splice between non-pipes (do_splice_direct function), this function is just not available to userspace. (The patch is just an idea, not tested. I don't know how to actually add this feature.)

==
Askar Safin

diff --git a/fs/splice.c b/fs/splice.c
index d37431d..ffb95db 100644
--- a/fs/splice.c
+++ b/fs/splice.c
@@ -1394,7 +1394,7 @@ static long do_splice(struct file *in, loff_t __user *off_in,
                return ret;
        }
 
-       return -EINVAL;
+       return do_splice_direct(in, off_in, out, off_out, len, flags);
 }
 
 /*

Re: Why "splice" doesn't splice between two non-pipes?

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 932 bytes --]

On Fri, 27 Jun 2014 20:34:42 +0400 Askar Safin <safinaskar@mail.ru> wrote:

> Why "splice" doesn't splice between two non-pipes? I think such splice would be very useful (not only file-to-socket). For example, for dd, cp, cp -r. Also, it seems, the kernel already supports splice between non-pipes (do_splice_direct function), this function is just not available to userspace. (The patch is just an idea, not tested. I don't know how to actually add this feature.)
> 
> ==
> Askar Safin
> 
> diff --git a/fs/splice.c b/fs/splice.c
> index d37431d..ffb95db 100644
> --- a/fs/splice.c
> +++ b/fs/splice.c
> @@ -1394,7 +1394,7 @@ static long do_splice(struct file *in, loff_t __user *off_in,
>                 return ret;
>         }
>  
> -       return -EINVAL;
> +       return do_splice_direct(in, off_in, out, off_out, len, flags);
>  }
>  
>  /*

Consider using "sendfile" instead of "splice".

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Why "splice" doesn't splice between two non-pipes?

[Askar Safin]

Thanks, NeilBrown.

Current "man sendfile" from kernel man pages contradicts to itself. The section "DESCRIPTION" says: "In Linux kernels before 2.6.33, out_fd must refer to a socket.  Since Linux 2.6.33 it can be any file", and AFAIK this is right current info. But the section "NOTES" says: "In Linux 2.4 and earlier, out_fd could also refer to a regular file" and this confuses. Also, the "NOTES" says: "The Linux-specific splice(2) call supports transferring data between arbitrary files (e.g., a pair of sockets)". This is wrong, because one splice call cannot splice together two sockets. So, please fix the man. Probably you should double-check whole man and make sure that whole man is up-to-date.

Also, it seems "splice" and "sendfile" both are not universal. I. e. none of this calls can move info from any fd to any other fd. "splice" needs one fd to be pipe. "sendfile" needs input fd to be mmap-capable. So, why there is no such universal syscall?

I think there is a way to splice any fds: just use pipe in the middle, i. e. performing in the kernel "splice-pipe-splice". And this will be not-slower (and probably faster) than two splices and one pipe created in userspace.

Let's imagine I want to create ultra-fast dd (or cat) for recent Linux kernels, which can dd any fds (sockets, terminals, files, pipes, etc). I think then I should do the following:
1. Try to do splice on this fds. If this fails go to step 2.
2. Try to do sendfile on this fds. If this fails go to step 3.
3. makepipe and fork. Then two splices in this two processes. (Am I right? Or I should use threads? Or I should use async splices in one thread [Is there such splices?]? Or select?) If this fails go to step 4.
4. read-write loop.

Is my plan right?
If yes, why so difficult? Why not just create some syscall, which can perform all this in kernel? (Of course, I don't mean real creating new syscall, this may be just fixing splice or sendfile.)

==
Askar Safin