splice question

splice question

[David Härdeman]

I'm currently toying with splice. The test program I have sends a MPEG2 
transport stream from a dvb card and sends it to N clients via http.

Since the dvb frontend driver doesn't support splice 
(linux/drivers/media/dvb/dvb-core/dmxdev.c I guess, anyone interested in 
adding splice support to it?) I had to do a normal read() to a 
user-space buffer and then write that to a pipe.

Once the data is in the pipe, my idea was to tee() from the pipe to each 
client socket using nonblocking ops, and then consume the data by 
splicing it to /dev/null.

The problem is that tee() doesn't support sockets. Is this a limitation 
that would be easy to fix?

Otherwise I guess I'd have to add a second pipe, then (in a loop)
tee() from the first to the second pipe and then splice from the 
second pipe to a socket. Doesn't sound very elegant and would need quite 
a lot of extra syscalls.

-- 
David Härdeman

Re: splice question

[Linus Torvalds]

On Sun, 12 Aug 2007, David H?rdeman wrote:
> 
> Once the data is in the pipe, my idea was to tee() from the pipe to each
> client socket using nonblocking ops, and then consume the data by splicing it
> to /dev/null.
> 
> The problem is that tee() doesn't support sockets. Is this a limitation that
> would be easy to fix?

It's very intentional. 

You should think of "tee()" as a memcpy() on kernel buffers.

And what are kernel buffers? It's not a socket.

The "kernel buffer" is simply just another name for a pipe.

So tee() *duplicates* the data in pipe, and then you can use "splice()" on 
the duplicated data to actually send it off somewhere else (eg a socket).

(Or any other pipe operation, for that matter - you can read() it into 
user space etc).

> Otherwise I guess I'd have to add a second pipe, then (in a loop)
> tee() from the first to the second pipe and then splice from the second pipe
> to a socket. Doesn't sound very elegant and would need quite a lot of extra
> syscalls.

You really should think of this as a memcpy(), and you'll be in the right 
mindframe. The system calls themselves are cheap.

			Linus

Re: splice question

[David Härdeman]

On Sun, Aug 12, 2007 at 12:41:54PM -0700, Linus Torvalds wrote:
>On Sun, 12 Aug 2007, David H?rdeman wrote:
>> Otherwise I guess I'd have to add a second pipe, then (in a loop)
>> tee() from the first to the second pipe and then splice from the second pipe
>> to a socket. Doesn't sound very elegant and would need quite a lot of extra
>> syscalls.
>
>You really should think of this as a memcpy(), and you'll be in the right 
>mindframe. The system calls themselves are cheap.

Ok, I've implemented it using two pipes, and it works. But it does seem 
a bit wasteful...in case one client is not keeping up, the data will 
have to be tee():ed first from pipe1 to pipe2, only to then find out 
that the splice() from pipe2 to socket only does a partial transfer 
after which the data in pipe2 has to be thrown away and then the loop 
starts over with the next client.

A tee() from pipe1 to the socket could (I imagine) realize immediately 
that the socket does not have enough buffer space and return EWOULDBLOCK 
and avoid at least one copy?

-- 
David Härdeman