Changes in function read_pipe

Changes in function read_pipe

[Carlos Rica]

Some people talked recently about renaming the function read_pipe
(sha1_file.c) to the better name read_fd.
Here I discuss other possible changes to the current version:

1. It now requires to allocate memory for the buffer before calling it,
you cannot pass it a pointer set to NULL or not initializated at all.

2. The function doesn't terminate the data with NUL, and if you
need that, you must to realloc before adding the '\0', because
the function only returns the size of the data, not the buffer size.

3. When function fails in reading (xread returns < 0), buffer is not
freed.

I'm not sure which of those issues are really important,
so I thought it was better to ask this in the list.
This is the current implementation of the function:

/*
 * reads from fd as long as possible into a supplied buffer of size bytes.
 * If necessary the buffer's size is increased using realloc()
 *
 * returns 0 if anything went fine and -1 otherwise
 *
 * NOTE: both buf and size may change, but even when -1 is returned
 * you still have to free() it yourself.
 */
int read_pipe(int fd, char** return_buf, unsigned long* return_size)
{
	char* buf = *return_buf;
	unsigned long size = *return_size;
	ssize_t iret;
	unsigned long off = 0;

	do {
		iret = xread(fd, buf + off, size - off);
		if (iret > 0) {
			off += iret;
			if (off == size) {
				size *= 2;
				buf = xrealloc(buf, size);
			}
		}
	} while (iret > 0);

	*return_buf = buf;
	*return_size = off;

	if (iret < 0)
		return -1;
	return 0;
}

Kristian Høgsberg recently sent some changes to the function
to replace the function with another one easier to call, but
with a different behavior:

 /*
- * reads from fd as long as possible into a supplied buffer of size bytes.
- * If necessary the buffer's size is increased using realloc()
+ * reads from fd as long as possible and allocates a buffer to hold
+ * the contents.  The buffer and size of the contents is returned in
+ * *return_buf and *return_size.  In case of failure, the allocated
+ * buffers are freed, otherwise, the buffer must be freed using xfree.
  *
  * returns 0 if anything went fine and -1 otherwise
- *
- * NOTE: both buf and size may change, but even when -1 is returned
- * you still have to free() it yourself.
  */
-int read_pipe(int fd, char** return_buf, unsigned long* return_size)
+int read_fd(int fd, char** return_buf, unsigned long* return_size)
 {
-	char* buf = *return_buf;
-	unsigned long size = *return_size;
+	unsigned long size = 4096;
+	char* buf = xmalloc(size);
 	ssize_t iret;
 	unsigned long off = 0;

@@ -2328,21 +2327,22 @@ int read_pipe(int fd, char** return_buf, unsigned long* return_size)
 	*return_buf = buf;
 	*return_size = off;

-	if (iret < 0)
+	if (iret < 0) {
+		free(buf);
 		return -1;
+	}
+
 	return 0;
 }

Any other ideas? read_pipe is really handy to reuse on builtins,
so we need to decide how we should call to it now before starting
to reuse it in many places.

Comments will be appreciated.

Re: Changes in function read_pipe

[Junio C Hamano]

Carlos Rica <jasampler@gmail.com> writes:

> Some people talked recently about renaming the function read_pipe
> (sha1_file.c) to the better name read_fd.
> Here I discuss other possible changes to the current version:
>
> 1. It now requires to allocate memory for the buffer before calling it,
> you cannot pass it a pointer set to NULL or not initializated at all.

It should be a trivial and easy fix to add "initial allocation"
at the beginning of the function.  While I suspect that the
optimal initial allocation would be different for different
callers, probably we would not care that much.

> 2. The function doesn't terminate the data with NUL, and if you
> need that, you must to realloc before adding the '\0', because
> the function only returns the size of the data, not the buffer size.

Among the existing callers, index_fd() does not need NUL
termination, but we are almost always allocating more than
needed when the final size is unknown, so giving an extra NUL
(of course not including that in the final result size) would
not hurt.  Other callers are 'stripspace' and 'mktag', and I
would imagine 'tag' and 'commit' when they are reading the
message from the standard input would benefit from the implicit
NUL termination.  I'd say go for it.

> 3. When function fails in reading (xread returns < 0), buffer is not
> freed.

I do not think this is a problem.  The current callers seem to
do something like this:

	if (read_pipe(fd, ...)) {
        	free(buf);
                return error(...);
	}

but not freeing, and reporting how many you read so far, would
allow readers to work on results partially read (not to be
confused with the EAGAIN which is already handled with xread()).