git.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Chris Mason <mason@suse.com>
To: Linus Torvalds <torvalds@osdl.org>
Cc: git@vger.kernel.org
Subject: Re: [PATCH] write-tree performance problems
Date: Wed, 20 Apr 2005 13:23:04 -0400	[thread overview]
Message-ID: <200504201323.05447.mason@suse.com> (raw)
In-Reply-To: <Pine.LNX.4.58.0504200957030.6467@ppc970.osdl.org>

[-- Attachment #1: Type: text/plain, Size: 1390 bytes --]

On Wednesday 20 April 2005 13:06, Linus Torvalds wrote:
> On Wed, 20 Apr 2005, Chris Mason wrote:
> > At any rate, the time for a single write-tree is pretty consistent. 
> > Before it was around .5 seconds, and with this change it goes down to
> > .128s.
>
> Oh, wow.
>
> I bet your SHA1 implementation is done with hand-optimized and scheduled
> x86 MMX code or something, while my poor G5 is probably using some slow
> generic routine. As a result, it only improved by 33% for me since the
> compression was just part of the picture, but with your cheap SHA1 the
> compression costs really dominated, and so it's almost four times faster
> for you.

Aha, I was wondering why your write-tree speeds sounded so bad...this athlon 
machine is ~2years old now.

Your comments about costs for writing the index file got me thinking, so I 
benchmarked how long the update-cache takes if we don't do the sha1 of the 
index file.  There was almost no difference at all.  update-cache currently 
takes about .152 seconds

The code to write the cache calls write() for every cache entry, writing just 
a few bytes at a time.  I changed it to collect these into a 16k buffer, 
which brings me down to .044s.  This might not help as much on ext23, since 
they are faster than reiser for tiny writes.

The patch below with your current tree brings my 100 patch test down to 22 
seconds again.

-chris

[-- Attachment #2: read-cache-fast.diff --]
[-- Type: text/x-diff, Size: 1104 bytes --]

--- linus.back/read-cache.c	2005-04-20 10:14:23.268310000 -0400
+++ linus/read-cache.c	2005-04-20 13:05:13.200083672 -0400
@@ -232,11 +232,12 @@
 	SHA_CTX c;
 	struct cache_header hdr;
 	int i;
+	char *buf;
+	int len = 0;
 
 	hdr.hdr_signature = htonl(CACHE_SIGNATURE);
 	hdr.hdr_version = htonl(1);
 	hdr.hdr_entries = htonl(entries);
-
 	SHA1_Init(&c);
 	SHA1_Update(&c, &hdr, offsetof(struct cache_header, sha1));
 	for (i = 0; i < entries; i++) {
@@ -246,13 +247,31 @@
 	}
 	SHA1_Final(hdr.sha1, &c);
 
+	buf = malloc(16384);
+	if (!buf) {
+		return -1;
+	}
 	if (write(newfd, &hdr, sizeof(hdr)) != sizeof(hdr))
 		return -1;
 
 	for (i = 0; i < entries; i++) {
 		struct cache_entry *ce = cache[i];
 		int size = ce_size(ce);
-		if (write(newfd, ce, size) != size)
+		if (size > 16384) {
+			if (write(newfd, ce, size) != size)
+				return -1;
+			continue;
+		}
+		if (len + size > 16384) {
+			if (write(newfd, buf, len) != len)
+				return -1;
+			len = 0;
+		}
+		memcpy(buf + len, ce, size);
+		len += size;
+	}
+	if (len) {
+		if (write(newfd, buf, len) != len)
 			return -1;
 	}
 	return 0;

  reply	other threads:[~2005-04-20 17:26 UTC|newest]

Thread overview: 54+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2005-04-19 16:50 [PATCH] write-tree performance problems Chris Mason
2005-04-19 17:36 ` Linus Torvalds
2005-04-19 18:11   ` Chris Mason
2005-04-19 19:03     ` Linus Torvalds
2005-04-19 21:08       ` Chris Mason
2005-04-19 21:23         ` Linus Torvalds
2005-04-20  0:49           ` Chris Mason
2005-04-20  1:09             ` Linus Torvalds
2005-04-20  6:43             ` Linus Torvalds
2005-04-20  7:38               ` H. Peter Anvin
2005-04-20  9:08                 ` WARNING! Object DB conversion (was Re: [PATCH] write-tree performance problems) Linus Torvalds
2005-04-20 10:04                   ` Ingo Molnar
2005-04-20 12:11                   ` Jon Seymour
2005-04-20 13:24                     ` Martin Uecker
2005-04-20 13:35                       ` Morten Welinder
2005-04-20 13:41                       ` Jon Seymour
2005-04-20 14:30                       ` C. Scott Ananian
2005-04-20 15:19                         ` Martin Uecker
2005-04-20 15:28                           ` C. Scott Ananian
2005-04-20 15:57                             ` Martin Uecker
2005-04-20 16:33                               ` Martin Uecker
2005-04-20 13:30                   ` Blob chunking code. [First look.] C. Scott Ananian
2005-04-20 17:31                     ` Blob chunking code. [Second look] C. Scott Ananian
2005-04-20 14:13                   ` WARNING! Object DB conversion (was Re: [PATCH] write-tree performance problems) David Woodhouse
2005-04-20 14:59                     ` Linus Torvalds
2005-04-20 22:29                       ` David Woodhouse
     [not found]                   ` <2cfc4032050420050655265d3a@mail.gmail.com>
2005-04-20 14:29                     ` Linus Torvalds
2005-04-20 14:35                       ` C. Scott Ananian
2005-04-20 15:22               ` [PATCH] write-tree performance problems Chris Mason
2005-04-20 15:30                 ` C. Scott Ananian
2005-04-20 15:46                   ` Linus Torvalds
2005-04-20 15:52                     ` C. Scott Ananian
2005-04-20 16:21                       ` Linus Torvalds
2005-04-20 15:40                 ` Linus Torvalds
2005-04-20 16:10                   ` David Willmore
2005-04-20 16:33                   ` Linus Torvalds
2005-04-20 16:41                     ` Linus Torvalds
2005-04-20 16:37                   ` Chris Mason
2005-04-20 17:06                     ` Linus Torvalds
2005-04-20 17:23                       ` Chris Mason [this message]
2005-04-20 17:52                         ` Linus Torvalds
2005-04-20 19:04                           ` Chris Mason
2005-04-20 19:19                             ` Linus Torvalds
2005-04-20 19:47                               ` Linus Torvalds
2005-04-20 18:07                       ` David S. Miller
2005-04-19 22:09       ` David Lang
2005-04-19 22:21         ` Linus Torvalds
2005-04-19 23:00           ` David Lang
2005-04-19 23:09             ` Linus Torvalds
2005-04-19 23:42               ` David Lang
2005-04-19 23:59                 ` Linus Torvalds
2005-04-19 21:52                   ` Christopher Li
2005-04-19 18:51   ` Olivier Galibert
2005-04-19 22:47   ` C. Scott Ananian

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=200504201323.05447.mason@suse.com \
    --to=mason@suse.com \
    --cc=git@vger.kernel.org \
    --cc=torvalds@osdl.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).