Re: [PATCH] write-tree performance problems

[Linus Torvalds <torvalds@osdl.org>]

On Wed, 20 Apr 2005, Chris Mason wrote:
> 
> The patch below with your current tree brings my 100 patch test down to 22 
> seconds again.

If you ever have a cache_entry bigger than 16384, your code will write 
things out in the wrong order (write the new cache without flushing the 
old buffer).

You also don't free the buffer.

Finally, if you really want to go fast, you should really try to make your
writes powers-of-two, ie fill up the buffer entirely rather than saying
"if I were to overflow, flush it now". It doesn't matter that much for
some filesystems (especially local and append-only like the patterns are
here), but it can definitely matter for the stupid ones.

But yeah, we could obviously chunk things out properly. You might want to 
just use stdio and "fwrite()", though, which does all of that for you, and 
hopefully does it right.

(I'm not a big fan of stdio for something like this, so if you want to 
create a little helper function that just does the chunking, go wild. 
Something like

	#define BUFSIZ 8192
	static char buffer[BUFSIZ];
	static unsigned long buflen;

	int ce_write(int fd, void *data, unsigned int len)
	{
		while (len) {
			unsigned int buffered = buflen;
			unsigned int partial = BUFSIZ - buflen;
			if (partial > len)
				partial = len;
			memcpy(buffer + buflen, data, partial);
			buffered += partial;
			if (buffered == BUFSIZ) {
				if (write(fd, buffer, BUFSIZ) != BUFSIZ)
					die("unable to write");
				buffered = 0;
			}
			buflen = buffered;
			len -= partial;
			data += partial;
		}
	}

	int ce_flush(int fd)
	{
		unsigned int left = buflen;
		if (left) {
			buflen = 0;
			if (write(fd, buffer, left) != left)
				die("unable to write");
		}
	}

which should be ok, and cheesily avoids the allocation overhread issues by
just having a nice static buffer.

"If you want to go fast, do it right".

Untested, as usual.

		Linus