PROPOSAL: extend pipe() to support NULL argument.

PROPOSAL: extend pipe() to support NULL argument.

[Changli Gao]

Linux also uses pipes as references of kernel buffers. In that case,
only one RW file descriptor will be enough, instead of inventing a new
system call, I think extending the existing pipe() is a better idea.
If pipe is passed with a NULL pointer, one RW file descriptor is
returned. I have checkd the other architectures except of x86, it
seems that not all of the platforms implement this system call with
the prototype: int pipe(int fd[2]); and changing glibc is needed. Is
it possible to extend pipe() in this way?

-- 
Regards，
Changli Gao(xiaosuo@gmail.com)

Re: PROPOSAL: extend pipe() to support NULL argument.

[Amerigo Wang]

On Thu, Jul 02, 2009 at 03:36:41PM +0800, Changli Gao wrote:
>Linux also uses pipes as references of kernel buffers. In that case,
>only one RW file descriptor will be enough, instead of inventing a new
>system call, I think extending the existing pipe() is a better idea.
>If pipe is passed with a NULL pointer, one RW file descriptor is
>returned. I have checkd the other architectures except of x86, it
>seems that not all of the platforms implement this system call with
>the prototype: int pipe(int fd[2]); and changing glibc is needed. Is
>it possible to extend pipe() in this way?
>

Why? Why do you need this feature?

Re: PROPOSAL: extend pipe() to support NULL argument.

[Changli Gao]

On Thu, Jul 2, 2009 at 5:13 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
> On Thu, Jul 02, 2009 at 03:36:41PM +0800, Changli Gao wrote:
>
> Why? Why do you need this feature?
>
Are you familiar with splice() and tee()? They both use pipes as kernel buffers.


-- 
Regards，
Changli Gao(xiaosuo@gmail.com)

Re: PROPOSAL: extend pipe() to support NULL argument.

[Amerigo Wang]

On Thu, Jul 02, 2009 at 05:18:49PM +0800, Changli Gao wrote:
>On Thu, Jul 2, 2009 at 5:13 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>> On Thu, Jul 02, 2009 at 03:36:41PM +0800, Changli Gao wrote:
>>
>> Why? Why do you need this feature?
>>
>Are you familiar with splice() and tee()? They both use pipes as kernel buffers.

You are not answering the question, obviously.

And you snipped too much, how can you return that fd? Using the return value?
Ah! This will probably break the user-space program...

Re: PROPOSAL: extend pipe() to support NULL argument.

[Changli Gao]

On Thu, Jul 2, 2009 at 5:46 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>>Are you familiar with splice() and tee()? They both use pipes as kernel buffers.
>
> You are not answering the question, obviously.
When you use pipes as kernel buffer handlers, two fd isn't necessary.
Using one will save half of fd resources. Is it obviously?

>
> And you snipped too much, how can you return that fd? Using the return value?
one RW file descriptor is returned. I have answered this in the first post.

> Ah! This will probably break the user-space program...
>
I don't think so. As a skillful programmer, who will trasfter pipe() a
NULL pointer? In any way, it is break sth, but not very seriously, and
won't affact any right and robust program.


-- 
Regards，
Changli Gao(xiaosuo@gmail.com)

Re: PROPOSAL: extend pipe() to support NULL argument.

[Amerigo Wang]

On Thu, Jul 02, 2009 at 06:04:02PM +0800, Changli Gao wrote:
>On Thu, Jul 2, 2009 at 5:46 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>>>Are you familiar with splice() and tee()? They both use pipes as kernel buffers.
>>
>> You are not answering the question, obviously.
>When you use pipes as kernel buffer handlers, two fd isn't necessary.
>Using one will save half of fd resources. Is it obviously?

Not really.. I can't see any reason why you use this method to save
fd's... pick read(2)/write(2).

>
>>
>> And you snipped too much, how can you return that fd? Using the return value?
>one RW file descriptor is returned. I have answered this in the first post.

No, you never say *how* it is returned.

>
>> Ah! This will probably break the user-space program...
>>
>I don't think so. As a skillful programmer, who will trasfter pipe() a
>NULL pointer? In any way, it is break sth, but not very seriously, and
>won't affact any right and robust program.

Huh? Isn't the code sample below too common?

if (pipe(...))
   perror("pipe");

Currently pipe(2) can make sure this is robust.

You are trying to break it...

Re: PROPOSAL: extend pipe() to support NULL argument.

[Changli Gao]

On Thu, Jul 2, 2009 at 6:21 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
> On Thu, Jul 02, 2009 at 06:04:02PM +0800, Changli Gao wrote:
>>On Thu, Jul 2, 2009 at 5:46 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>>>>Are you familiar with splice() and tee()? They both use pipes as kernel buffers.
>>>
>>> You are not answering the question, obviously.
>>When you use pipes as kernel buffer handlers, two fd isn't necessary.
>>Using one will save half of fd resources. Is it obviously?
>
> Not really.. I can't see any reason why you use this method to save
> fd's... pick read(2)/write(2).
In order to use splice, you must have pipe first. The common code is:

int fd[2];

pipe(fd);
splice(infd, NULL, fd[1], NULL, getpagesize(), SPLICE_F_MOVE);
splice(fd[0], NULL, outfd, NULL, getpagesize(), SPLICE_F_MOVE);
close(fd[0]);
close(fd[1]);

if pipe supports NULL argument, and returnes a RW pipe fd in that
case, the code will become:

int fd;

fd = pipe(NULL);
splice(infd, NULL, fd, NULL, getpagesize(), SPLICE_F_MOVE);
splice(fd, NULL, outfd, NULL, getpagesize(), SPLICE_F_MOVE);
close(fd);

One fd is saved.

>
>>
>>>
>>> And you snipped too much, how can you return that fd? Using the return value?
>>one RW file descriptor is returned. I have answered this in the first post.
>
> No, you never say *how* it is returned.
return value. save in %eax in i386.

>
>>
>>> Ah! This will probably break the user-space program...
>>>
>>I don't think so. As a skillful programmer, who will trasfter pipe() a
>>NULL pointer? In any way, it is break sth, but not very seriously, and
>>won't affact any right and robust program.
>
> Huh? Isn't the code sample below too common?
>
> if (pipe(...))
>   perror("pipe");
>
> Currently pipe(2) can make sure this is robust.
>
> You are trying to break it...
>

It depends on the argument for pipe, if the argument is NULL, it
breaks the code above.



-- 
Regards，
Changli Gao(xiaosuo@gmail.com)

Re: PROPOSAL: extend pipe() to support NULL argument.

[Amerigo Wang]

On Fri, Jul 03, 2009 at 08:43:39AM +0800, Changli Gao wrote:
>On Thu, Jul 2, 2009 at 6:21 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>> On Thu, Jul 02, 2009 at 06:04:02PM +0800, Changli Gao wrote:
>>>On Thu, Jul 2, 2009 at 5:46 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>>>>>Are you familiar with splice() and tee()? They both use pipes as kernel buffers.
>>>>
>>>> You are not answering the question, obviously.
>>>When you use pipes as kernel buffer handlers, two fd isn't necessary.
>>>Using one will save half of fd resources. Is it obviously?
>>
>> Not really.. I can't see any reason why you use this method to save
>> fd's... pick read(2)/write(2).
>In order to use splice, you must have pipe first. The common code is:
>
>int fd[2];
>
>pipe(fd);
>splice(infd, NULL, fd[1], NULL, getpagesize(), SPLICE_F_MOVE);
>splice(fd[0], NULL, outfd, NULL, getpagesize(), SPLICE_F_MOVE);
>close(fd[0]);
>close(fd[1]);
>
>if pipe supports NULL argument, and returnes a RW pipe fd in that
>case, the code will become:
>
>int fd;
>
>fd = pipe(NULL);
>splice(infd, NULL, fd, NULL, getpagesize(), SPLICE_F_MOVE);
>splice(fd, NULL, outfd, NULL, getpagesize(), SPLICE_F_MOVE);
>close(fd);
>
>One fd is saved.


What is a pipe in your mind? A pipe has two points, obviously,
you are making it a single point, thus you can't call it a pipe
any more...

I don't like this.

And just as what you showed above, you can close the fd's
after finishing, thus you only save one fd during these
two splice(), not useful...


>>
>> You are trying to break it...
>>
>
>It depends on the argument for pipe, if the argument is NULL, it
>breaks the code above.
>

Sure, malloc(2) can return NULL.

Re: PROPOSAL: extend pipe() to support NULL argument.

[Changli Gao]

On Fri, Jul 3, 2009 at 9:55 AM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>
>
> What is a pipe in your mind? A pipe has two points, obviously,
> you are making it a single point, thus you can't call it a pipe
> any more...
>
> I don't like this
How do you think the following code?

mkfifo("/tmp/a", O_RDWR);
fd = open("'/tmp/a", O_RDWR);

Is it right? Is it allowed by Linux/Unix/POSIX?

and when used by splice(), pipe represents a kernel buffer handler not
its original meaning.
.
>
> And just as what you showed above, you can close the fd's
> after finishing, thus you only save one fd during these
> two splice(), not useful...
splice() is a high performance system call. It maybe used in thousands
of connection environment, so one instance saves one fd, and thousands
of fd will be saved.

>
>
>>>
>>> You are trying to break it...
>>>
>>
>>It depends on the argument for pipe, if the argument is NULL, it
>>breaks the code above.
>>
>
> Sure, malloc(2) can return NULL.
>
>
You should check the return value returned by malloc brefore passing
it to pipe(), and not rely on pipe()'s check.


-- 
Regards，
Changli Gao(xiaosuo@gmail.com)

Re: PROPOSAL: extend pipe() to support NULL argument.

[Amerigo Wang]

On Fri, Jul 03, 2009 at 10:10:13AM +0800, Changli Gao wrote:
>On Fri, Jul 3, 2009 at 9:55 AM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>>
>>
>> What is a pipe in your mind? A pipe has two points, obviously,
>> you are making it a single point, thus you can't call it a pipe
>> any more...
>>
>> I don't like this
>How do you think the following code?
>
>mkfifo("/tmp/a", O_RDWR);
>fd = open("'/tmp/a", O_RDWR);
>
>Is it right? Is it allowed by Linux/Unix/POSIX?
>
>and when used by splice(), pipe represents a kernel buffer handler not
>its original meaning.

When I said 'pipe', I meant *anonymous* pipe, definitely!
There is nothing related with *named* pipe here.

You are going to a *wrong* direction.

>.
>>
>> And just as what you showed above, you can close the fd's
>> after finishing, thus you only save one fd during these
>> two splice(), not useful...
>splice() is a high performance system call. It maybe used in thousands
>of connection environment, so one instance saves one fd, and thousands
>of fd will be saved.

If you repeat your (pipe,splice,splice,close,close) sequence for
thousands times, it is still the same, nothing saves...

Re: PROPOSAL: extend pipe() to support NULL argument.

[Changli Gao]

On Fri, Jul 3, 2009 at 11:00 AM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>
> When I said 'pipe', I meant *anonymous* pipe, definitely!
> There is nothing related with *named* pipe here.
>
Pipe means *anonymous* pipe? Who defined that? What the difference
between an anonymou pipe and a named pipe? Two ends or one ends? I
don't think so. Having explicit name is the right difference between
them.

And when I said 'kernel buffer handler', I means it is not a
traditional pipe, and I just extend and reuse the pipe API to create a
kernel buffer handler for splicing use.

> You are going to a *wrong* direction.
>
What is the right direction you think? Keep the pipe() API unchanged
and keep two fds refering to the same kernel buffer? If you want to
keep the API unchanged just for compatiblity with POSIX, I agree with
you, and maybe a new system call is needed.

>
> If you repeat your (pipe,splice,splice,close,close) sequence for
> thousands times, it is still the same, nothing saves...
>
You means that?

while (<condition>) {
 pipe/splice/splice/close/close?
}

You don't know my meaning. As a proxy server, there maybe lots of
connections to maintain, and these connections will keep open for a
long time. If the data received can be sent in a relay cycle, the
kernel buffer can be reused. If not, the kernel buffer must be
reserved. When there are lots of these kinds of connections, lots of
kernel buffers must be reserved. At this time, whether two fds per
kernel buffer or one fds per kernel buffer matters.


-- 
Regards，
Changli Gao(xiaosuo@gmail.com)

Re: PROPOSAL: extend pipe() to support NULL argument.

[Amerigo Wang]

On Fri, Jul 03, 2009 at 12:08:58PM +0800, Changli Gao wrote:
>On Fri, Jul 3, 2009 at 11:00 AM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>>
>> When I said 'pipe', I meant *anonymous* pipe, definitely!
>> There is nothing related with *named* pipe here.
>>
>Pipe means *anonymous* pipe? Who defined that? What the difference
>between an anonymou pipe and a named pipe? Two ends or one ends? I
>don't think so. Having explicit name is the right difference between
>them.

I have no interests to teach you its definition.

if you re-read my email, you should conclude what 'pipe' there means,
according to the contexts.

>
>And when I said 'kernel buffer handler', I means it is not a
>traditional pipe, and I just extend and reuse the pipe API to create a
>kernel buffer handler for splicing use.


What pipe(2) creates is exactly an *anonymous* pipe.

>
>> You are going to a *wrong* direction.
>>
>What is the right direction you think? Keep the pipe() API unchanged
>and keep two fds refering to the same kernel buffer? If you want to
>keep the API unchanged just for compatiblity with POSIX, I agree with
>you, and maybe a new system call is needed.


Yes, exactly.
Inventing a new API is better than modifying pipe(2), IMO.

BUT I still don't agree that you really need this... I think you
can add a flag or something like that to an fd to do this, e.g.

   fcntl(fd, F_SETFD, FD_PIPERW);

Isn't this better?

>
>>
>> If you repeat your (pipe,splice,splice,close,close) sequence for
>> thousands times, it is still the same, nothing saves...
>>
>You means that?
>
>while (<condition>) {
> pipe/splice/splice/close/close?
>}

At least this is what I understand from your words.

>
>You don't know my meaning. As a proxy server, there maybe lots of
>connections to maintain, and these connections will keep open for a
>long time. If the data received can be sent in a relay cycle, the
>kernel buffer can be reused. If not, the kernel buffer must be
>reserved. When there are lots of these kinds of connections, lots of
>kernel buffers must be reserved. At this time, whether two fds per
>kernel buffer or one fds per kernel buffer matters.

SHow us the code, please.

Re: PROPOSAL: extend pipe() to support NULL argument.

[Changli Gao]

On Fri, Jul 3, 2009 at 1:19 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>
>
> Yes, exactly.
> Inventing a new API is better than modifying pipe(2), IMO.
>
> BUT I still don't agree that you really need this... I think you
> can add a flag or something like that to an fd to do this, e.g.
>
>   fcntl(fd, F_SETFD, FD_PIPERW);
>
> Isn't this better?
You don't understand my need. I don't want to change a RDONLY or
WRONLY pipe to a RW one, but I want to pipe() return just one RW pipe
instead. It seems you prefer pipe0() to pipe(NULL).

>
>>
>>You don't know my meaning. As a proxy server, there maybe lots of
>>connections to maintain, and these connections will keep open for a
>>long time. If the data received can be sent in a relay cycle, the
>>kernel buffer can be reused. If not, the kernel buffer must be
>>reserved. When there are lots of these kinds of connections, lots of
>>kernel buffers must be reserved. At this time, whether two fds per
>>kernel buffer or one fds per kernel buffer matters.
>
> SHow us the code, please.
>
If you don't know my need after reading my words, I don't think you
can understand the fake code. The fake code:

main thread:

while (1) {
   fd = accept();
   pthread_create(worker, fd);
}

worker thread(assume data is transfered just from client to server for
simplification):

serv_fd = connect();
while (1) {
  select(fd, RD);
  pipe = get_pipe_from_poll();
  n = splice(fd, pipe);
  while (n > 0) {
     n -= splice(pipe, serv_fd);
  }
  put_pipe_to_poll();
}
close(serv_fd);
close(fd);

If there are lots of threads blocked on splice(pipe, serv_fd), there
will be lots of kernel buffers.

-- 
Regards，
Changli Gao(xiaosuo@gmail.com)

Re: PROPOSAL: extend pipe() to support NULL argument.

[Amerigo Wang]

On Fri, Jul 03, 2009 at 01:42:12PM +0800, Changli Gao wrote:
>On Fri, Jul 3, 2009 at 1:19 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>>
>>
>> Yes, exactly.
>> Inventing a new API is better than modifying pipe(2), IMO.
>>
>> BUT I still don't agree that you really need this... I think you
>> can add a flag or something like that to an fd to do this, e.g.
>>
>>   fcntl(fd, F_SETFD, FD_PIPERW);
>>
>> Isn't this better?
>You don't understand my need. I don't want to change a RDONLY or
>WRONLY pipe to a RW one, but I want to pipe() return just one RW pipe
>instead.

This is exacly what I meant by that example.

>It seems you prefer pipe0() to pipe(NULL).

pipefd() seems better.

>>>You don't know my meaning. As a proxy server, there maybe lots of
>>>connections to maintain, and these connections will keep open for a
>>>long time. If the data received can be sent in a relay cycle, the
>>>kernel buffer can be reused. If not, the kernel buffer must be
>>>reserved. When there are lots of these kinds of connections, lots of
>>>kernel buffers must be reserved. At this time, whether two fds per
>>>kernel buffer or one fds per kernel buffer matters.
>>
>> SHow us the code, please.
>>
>If you don't know my need after reading my words, I don't think you
>can understand the fake code.

No, code is *much* better than any of your words.

>The fake code:
>
>main thread:
>
>while (1) {
>   fd = accept();
>   pthread_create(worker, fd);
>}
>
>worker thread(assume data is transfered just from client to server for
>simplification):
>
>serv_fd = connect();
>while (1) {
>  select(fd, RD);
>  pipe = get_pipe_from_poll();
>  n = splice(fd, pipe);
>  while (n > 0) {
>     n -= splice(pipe, serv_fd);
>  }
>  put_pipe_to_poll();
>}
>close(serv_fd);
>close(fd);

If saving one fd really helps here, probably you want to
save more, you will need a syscall like:

  int splice_without_new_fd(int infd, int outfd);

But splice(2) is designed to be as it is. You need to increase
your fd limit, instead of saving one by pipe().

Re: PROPOSAL: extend pipe() to support NULL argument.

[Changli Gao]

On Fri, Jul 3, 2009 at 3:15 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>
> If saving one fd really helps here, probably you want to
> save more, you will need a syscall like:
>
>  int splice_without_new_fd(int infd, int outfd);
Do you know sendfile()? Its current implementation is buggy, and will
be blocked on outfd. Anyway, the above code is just a use case, there
are other cases sendfile can't cover.

>
> But splice(2) is designed to be as it is. You need to increase
> your fd limit, instead of saving one by pipe().
>
I don't agree with you. We should save resource as much as we can, and
not work around it.


-- 
Regards，
Changli Gao(xiaosuo@gmail.com)

Re: PROPOSAL: extend pipe() to support NULL argument.

[Amerigo Wang]

On Fri, Jul 03, 2009 at 03:40:34PM +0800, Changli Gao wrote:
>On Fri, Jul 3, 2009 at 3:15 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>>
>> If saving one fd really helps here, probably you want to
>> save more, you will need a syscall like:
>>
>>  int splice_without_new_fd(int infd, int outfd);
>Do you know sendfile()? Its current implementation is buggy, and will
>be blocked on outfd. Anyway, the above code is just a use case, there
>are other cases sendfile can't cover.

So what? So you should fix it intead of inventing a new pipe() and use
splice(2)...

Wait... if splice(2) doesn't block, what is your point for saving
an fd in your code? You can do:

int splice_two_fd(int fd1, int fd2)
{
	int fds[2];
	pipe(fds);
	splice(fd1, fds[0]);//not block
	splice(fds[1], fd2);//not block
	close(fds[0]); //can be closed soon
	close(fds[1]); //ditto
}

Outside this function no new fd's are used.

>
>>
>> But splice(2) is designed to be as it is. You need to increase
>> your fd limit, instead of saving one by pipe().
>>
>I don't agree with you. We should save resource as much as we can, and
>not work around it.

You are saying splice(2) is wrong? Because it is splice(2) who
needs 3 fd's finally.

Re: PROPOSAL: extend pipe() to support NULL argument.

[Changli Gao]

On Fri, Jul 3, 2009 at 4:16 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
> On Fri, Jul 03, 2009 at 03:40:34PM +0800, Changli Gao wrote:
>>On Fri, Jul 3, 2009 at 3:15 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>>>
>>> If saving one fd really helps here, probably you want to
>>> save more, you will need a syscall like:
>>>
>>>  int splice_without_new_fd(int infd, int outfd);
>>Do you know sendfile()? Its current implementation is buggy, and will
>>be blocked on outfd. Anyway, the above code is just a use case, there
>>are other cases sendfile can't cover.
>
> So what? So you should fix it intead of inventing a new pipe() and use
> splice(2)...
I said: It just a special case. Fixing sendfile() won't fix everything.

>
> Wait... if splice(2) doesn't block, what is your point for saving
> an fd in your code? You can do:
>
> int splice_two_fd(int fd1, int fd2)
> {
>        int fds[2];
>        pipe(fds);
>        splice(fd1, fds[0]);//not block
>        splice(fds[1], fd2);//not block
>        close(fds[0]); //can be closed soon
>        close(fds[1]); //ditto
> }
>
> Outside this function no new fd's are used.
>
In fact, sendfile(2) is implemented via splice(2) infrastructure. The
logic is the same, and much clever than yours. Dive into its code, you
will find the reason why it must be blocked on outfd.

>>
>>>
>>> But splice(2) is designed to be as it is. You need to increase
>>> your fd limit, instead of saving one by pipe().
>>>
>>I don't agree with you. We should save resource as much as we can, and
>>not work around it.
>
> You are saying splice(2) is wrong? Because it is splice(2) who
> needs 3 fd's finally.
>
It isn't splice(2)'s fault. In fact, I can also work around this issue
by named pipe.



-- 
Regards，
Changli Gao(xiaosuo@gmail.com)

Re: PROPOSAL: extend pipe() to support NULL argument.

[Amerigo Wang]

On Fri, Jul 03, 2009 at 04:27:29PM +0800, Changli Gao wrote:
>On Fri, Jul 3, 2009 at 4:16 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>> On Fri, Jul 03, 2009 at 03:40:34PM +0800, Changli Gao wrote:
>>>On Fri, Jul 3, 2009 at 3:15 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>>>>
>>>> If saving one fd really helps here, probably you want to
>>>> save more, you will need a syscall like:
>>>>
>>>>  int splice_without_new_fd(int infd, int outfd);
>>>Do you know sendfile()? Its current implementation is buggy, and will
>>>be blocked on outfd. Anyway, the above code is just a use case, there
>>>are other cases sendfile can't cover.
>>
>> So what? So you should fix it intead of inventing a new pipe() and use
>> splice(2)...
>I said: It just a special case. Fixing sendfile() won't fix everything.

What do you want? Just saving an fd, isn't it?

sendfile() does this but you said it's buggy, if you fix it, what is the
rest problem?

>
>>
>> Wait... if splice(2) doesn't block, what is your point for saving
>> an fd in your code? You can do:
>>
>> int splice_two_fd(int fd1, int fd2)
>> {
>>        int fds[2];
>>        pipe(fds);
>>        splice(fd1, fds[0]);//not block
>>        splice(fds[1], fd2);//not block
>>        close(fds[0]); //can be closed soon
>>        close(fds[1]); //ditto
>> }
>>
>> Outside this function no new fd's are used.
>>
>In fact, sendfile(2) is implemented via splice(2) infrastructure. The
>logic is the same, and much clever than yours. Dive into its code, you
>will find the reason why it must be blocked on outfd.


Sure, this is done in user-space, definitely....

>
>>>
>>>>
>>>> But splice(2) is designed to be as it is. You need to increase
>>>> your fd limit, instead of saving one by pipe().
>>>>
>>>I don't agree with you. We should save resource as much as we can, and
>>>not work around it.
>>
>> You are saying splice(2) is wrong? Because it is splice(2) who
>> needs 3 fd's finally.
>>
>It isn't splice(2)'s fault. In fact, I can also work around this issue
>by named pipe.

And increase the max fd number...

Re: PROPOSAL: extend pipe() to support NULL argument.

[Changli Gao]

On Fri, Jul 3, 2009 at 5:42 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>>I said: It just a special case. Fixing sendfile() won't fix everything.
>
> What do you want? Just saving an fd, isn't it?
>
> sendfile() does this but you said it's buggy, if you fix it, what is the
> rest problem?
>
You should know why Linux invented splice(2). Not all of the problems
can be solved by sendfile(2).

>>
>>In fact, sendfile(2) is implemented via splice(2) infrastructure. The
>>logic is the same, and much clever than yours. Dive into its code, you
>>will find the reason why it must be blocked on outfd.
>
>
> Sure, this is done in user-space, definitely....
>
Did you review the code of sendfile(2)? If so, you will know the
common problem.If the data buffered in pipe can't be sent by one
splice(2) call, there are two ways:
1. reserve this pipe to reserve the data in it.
2. block on splice(pipe, outfd) until all the data in it is sent.
and sendfile(2) chooses this second one. In fact, there is  another
choice: simply close the pipe, and the data in the pipe is lost
forever. It isn't a choice, is it?

>>
>>It isn't splice(2)'s fault. In fact, I can also work around this issue
>>by named pipe.
>
> And increase the max fd number...
>
Yea, in many cases, max fd number must be enlarged. More fds means
more memory. Although memory is cheaper today, we have to do our best
to save money.


-- 
Regards，
Changli Gao(xiaosuo@gmail.com)

Re: PROPOSAL: extend pipe() to support NULL argument.

[Amerigo Wang]

On Fri, Jul 03, 2009 at 05:59:10PM +0800, Changli Gao wrote:
>On Fri, Jul 3, 2009 at 5:42 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>>>I said: It just a special case. Fixing sendfile() won't fix everything.
>>
>> What do you want? Just saving an fd, isn't it?
>>
>> sendfile() does this but you said it's buggy, if you fix it, what is the
>> rest problem?
>>
>You should know why Linux invented splice(2). Not all of the problems
>can be solved by sendfile(2).

Sure, it definitely can be seen from the interface....

>
>>>
>>>In fact, sendfile(2) is implemented via splice(2) infrastructure. The
>>>logic is the same, and much clever than yours. Dive into its code, you
>>>will find the reason why it must be blocked on outfd.
>>
>>
>> Sure, this is done in user-space, definitely....
>>
>Did you review the code of sendfile(2)? 

Not at all.

I just wanted to show what I wanted to say, not to re-implement
sendfile(2).

Re: PROPOSAL: extend pipe() to support NULL argument.

[Albert ARIBAUD]

Changli Gao a écrit :

> Yea, in many cases, max fd number must be enlarged. More fds means
> more memory. Although memory is cheaper today, we have to do our best
> to save money.

Sorry for interrupting, but I don't see how pipe could return a single fd, 
considering there are two (partly) independent ends, each being read (resp. 
written) in their own time, and an fd has only one "current read/write 
position" IIUC.

If the proposal is to have two independent positions (one for reads and one 
for writes) for a single fd, then I am not sure the gain in the number of 
fds used is worth the loss in the increased size of the fd structure.

Am I missing something?

Amicalement,
-- 
Albert.

Re: PROPOSAL: extend pipe() to support NULL argument.

[Changli Gao]

On Fri, Jul 3, 2009 at 6:28 PM, Albert ARIBAUD<albert.aribaud@free.fr> wrote:
> Changli Gao a écrit :
>
>> Yea, in many cases, max fd number must be enlarged. More fds means
>> more memory. Although memory is cheaper today, we have to do our best
>> to save money.
>
> Sorry for interrupting, but I don't see how pipe could return a single fd,
> considering there are two (partly) independent ends, each being read (resp.
> written) in their own time, and an fd has only one "current read/write
> position" IIUC.
>
> If the proposal is to have two independent positions (one for reads and one
> for writes) for a single fd, then I am not sure the gain in the number of
> fds used is worth the loss in the increased size of the fd structure.
>
> Am I missing something?
pipe doesn't support llseek.


-- 
Regards，
Changli Gao(xiaosuo@gmail.com)

Re: PROPOSAL: extend pipe() to support NULL argument.

[Albert ARIBAUD]

Changli Gao a écrit :
> On Fri, Jul 3, 2009 at 6:28 PM, Albert ARIBAUD<albert.aribaud@free.fr> wrote:
>> Changli Gao a écrit :
>>
>>> Yea, in many cases, max fd number must be enlarged. More fds means
>>> more memory. Although memory is cheaper today, we have to do our best
>>> to save money.
>> Sorry for interrupting, but I don't see how pipe could return a single fd,
>> considering there are two (partly) independent ends, each being read (resp.
>> written) in their own time, and an fd has only one "current read/write
>> position" IIUC.
>>
>> If the proposal is to have two independent positions (one for reads and one
>> for writes) for a single fd, then I am not sure the gain in the number of
>> fds used is worth the loss in the increased size of the fd structure.
>>
>> Am I missing something?
 >
> pipe doesn't support llseek.

I wasn't thinking of actively seeking a file position, but simply that reads 
and writes were independent (and both variable) in size, thus even without 
(l)seeking, each endpoint has an independent read (resp. write) position to 
track what's been read from (resp.written into) it at any time, and such a 
position is, IIUC, single for a given fd, making a fd (a struct file) 
insufficient for representing a pipe.

Amicalement,
-- 
Albert.

Re: PROPOSAL: extend pipe() to support NULL argument.

[Changli Gao]

On Mon, Jul 6, 2009 at 2:15 PM, Albert ARIBAUD<albert.aribaud@free.fr> wrote:
>>
>>
>> pipe doesn't support llseek.
>
> I wasn't thinking of actively seeking a file position, but simply that reads
> and writes were independent (and both variable) in size, thus even without
> (l)seeking, each endpoint has an independent read (resp. write) position to
> track what's been read from (resp.written into) it at any time, and such a
> position is, IIUC, single for a given fd, making a fd (a struct file)
> insufficient for representing a pipe.
>
> Amicalement,
> --
> Albert.
>
pipe doesn't refer to pos when reading or writing. Please refer to
pipe_read() and pipe_write() for more detail.


-- 
Regards，
Changli Gao(xiaosuo@gmail.com)

Re: PROPOSAL: extend pipe() to support NULL argument.

[Albert ARIBAUD]

Changli Gao a écrit :
> On Mon, Jul 6, 2009 at 2:15 PM, Albert ARIBAUD<albert.aribaud@free.fr> wrote:
>>>
>>> pipe doesn't support llseek.
>> I wasn't thinking of actively seeking a file position, but simply that reads
>> and writes were independent (and both variable) in size, thus even without
>> (l)seeking, each endpoint has an independent read (resp. write) position to
>> track what's been read from (resp.written into) it at any time, and such a
>> position is, IIUC, single for a given fd, making a fd (a struct file)
>> insufficient for representing a pipe.
>>
>> Amicalement,
>> --
>> Albert.
>>
> pipe doesn't refer to pos when reading or writing. Please refer to
> pipe_read() and pipe_write() for more detail.

Hmm. Then how does pipe avoid overwriting what it's already written at the 
in endpoint, or reading twice the same data from the out endpoint?

Amicalement,
-- 
Albert.

Re: PROPOSAL: extend pipe() to support NULL argument.

[Denys Vlasenko]

On Friday 03 July 2009 12:28, Albert ARIBAUD wrote:
> Changli Gao a écrit :
> 
> > Yea, in many cases, max fd number must be enlarged. More fds means
> > more memory. Although memory is cheaper today, we have to do our best
> > to save money.
> 
> Sorry for interrupting, but I don't see how pipe could return a single fd, 
> considering there are two (partly) independent ends, each being read (resp. 
> written) in their own time, and an fd has only one "current read/write 
> position" IIUC.
> 
> If the proposal is to have two independent positions (one for reads and one 
> for writes) for a single fd, then I am not sure the gain in the number of 
> fds used is worth the loss in the increased size of the fd structure.
> 
> Am I missing something?

As I understand it, the proposal is to make pipe(NULL) return a fd,
which can be read from and written to. Whatever you write
is remembered, and can be retrieved by reads later.

standard pipe(two_fds) returns two fds. One is only used for writes.
Another is only used for reads. Which may be wasteful in some
situations (like, if you have thousands of pipes).

There is no trick I know of which may be used to "combine"
these two fds into one, close second one and thus save one fd table slot.

-- 
vda

Re: PROPOSAL: extend pipe() to support NULL argument.

[Changli Gao]

On Tue, Jul 7, 2009 at 12:43 PM, Denys Vlasenko<vda.linux@googlemail.com> wrote:
> On Friday 03 July 2009 12:28, Albert ARIBAUD wrote:
>
> As I understand it, the proposal is to make pipe(NULL) return a fd,
> which can be read from and written to. Whatever you write
> is remembered, and can be retrieved by reads later.
>
> standard pipe(two_fds) returns two fds. One is only used for writes.
> Another is only used for reads. Which may be wasteful in some
> situations (like, if you have thousands of pipes).
>
> There is no trick I know of which may be used to "combine"
> these two fds into one, close second one and thus save one fd table slot.
>
Right!

 mkfifo(name);
 open(name, O_RDWR);
 unlink(name);

Is it a trick?


-- 
Regards，
Changli Gao(xiaosuo@gmail.com)

Re: PROPOSAL: extend pipe() to support NULL argument.

[David Newall]

Changli Gao wrote:
>  mkfifo(name);
>  open(name, O_RDWR);
>  unlink(name);
>
> Is it a trick?

This does beg the question, why is it necessary to "extend" pipe() in
this way?

Re: PROPOSAL: extend pipe() to support NULL argument.

[Zan Lynx]

David Newall wrote:
> Changli Gao wrote:
>>  mkfifo(name);
>>  open(name, O_RDWR);
>>  unlink(name);
>>
>> Is it a trick?
> 
> This does beg the question, why is it necessary to "extend" pipe() in
> this way?

Well, to me it sounds like the goal of the proposal is for very high 
load, high performance socket servers so efficiency matters.

The above is more than 3 times as expensive as one pipe(). Each syscall 
is thousands of machine cycles in overhead. In addition to that all 
three are VFS operations which as I recall are even more expensive than 
other syscalls.

But perhaps mkfifo(NULL) should be the syscall chosen to extend since 
its parameters are much like open() already, and it seems to work in the 
above sequence to create a one-FD FIFO already.

Or instead of, or in addition to a NULL name argument, its mode argument 
could be used as a flags argument to specify an anonymous fifo creation 
with FD return.
-- 
Zan Lynx
zlynx@acm.org

"Knowledge is Power.  Power Corrupts.  Study Hard.  Be Evil."

Re: PROPOSAL: extend pipe() to support NULL argument.

[Changli Gao]

On Wed, Jul 8, 2009 at 5:02 AM, Zan Lynx<zlynx@acm.org> wrote:
>
> But perhaps mkfifo(NULL) should be the syscall chosen to extend since its
> parameters are much like open() already, and it seems to work in the above
> sequence to create a one-FD FIFO already.
>
It sounds a good idea.


-- 
Regards，
Changli Gao(xiaosuo@gmail.com)

Re: PROPOSAL: extend pipe() to support NULL argument.

[David Newall]

Changli Gao wrote:
> On Wed, Jul 8, 2009 at 5:02 AM, Zan Lynx<zlynx@acm.org> wrote:
>   
>> But perhaps mkfifo(NULL) should be the syscall chosen to extend since its
>> parameters are much like open() already, and it seems to work in the above
>> sequence to create a one-FD FIFO already.
>>
>>     
> It sounds a good idea.
>   

I disagree.  I disbelieve that the cost of two fds per pipe is too high.

Re: PROPOSAL: extend pipe() to support NULL argument.

[Alan Cox]

On Thu, 2 Jul 2009 17:18:49 +0800
Changli Gao <xiaosuo@gmail.com> wrote:

> On Thu, Jul 2, 2009 at 5:13 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
> > On Thu, Jul 02, 2009 at 03:36:41PM +0800, Changli Gao wrote:
> >
> > Why? Why do you need this feature?
> >
> Are you familiar with splice() and tee()? They both use pipes as kernel buffers.

And are being used quite happily with the pipe() call. So what is the
problem ?

Re: PROPOSAL: extend pipe() to support NULL argument.

[Changli Gao]

On Thu, Jul 2, 2009 at 6:42 PM, Alan Cox<alan@lxorguk.ukuu.org.uk> wrote:
> On Thu, 2 Jul 2009 17:18:49 +0800
> Changli Gao <xiaosuo@gmail.com> wrote:
>
>> On Thu, Jul 2, 2009 at 5:13 PM, Amerigo Wang<xiyou.wangcong@gmail.com> wrote:
>> > On Thu, Jul 02, 2009 at 03:36:41PM +0800, Changli Gao wrote:
>> >
>> > Why? Why do you need this feature?
>> >
>> Are you familiar with splice() and tee()? They both use pipes as kernel buffers.
>
> And are being used quite happily with the pipe() call. So what is the
> problem ?
>
No problem! But I just want to save a fd. As one RW pipe fd is enough
for splicing.



-- 
Regards，
Changli Gao(xiaosuo@gmail.com)