[PATCH] Limit the number of requests outstanding in ssh-fetch.

[PATCH] Limit the number of requests outstanding in ssh-fetch.

[Daniel Barkalow]

This completes fetches if there are more than 100 outstanding requests
and there are more to prefetch.

Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
---

Lightly tested; I reduced the limit to 5, and pulled a small tree 
successfully with some of the requests being completed early. I didn't 
have the test case to verify that a limit of 100 is sufficiently low, but 
handwaving suggests that it should be.

 ssh-fetch.c |   28 +++++++++++++++++++++++++++-
 1 files changed, 27 insertions(+), 1 deletions(-)

528f784890df88a5c054913136bbc53b164ebb94
diff --git a/ssh-fetch.c b/ssh-fetch.c
--- a/ssh-fetch.c
+++ b/ssh-fetch.c
@@ -36,12 +36,26 @@ static ssize_t force_write(int fd, void 
 	return ret;
 }
 
+static int prefetches = 0;
+
+static struct object_list *in_transit = NULL;
+static struct object_list **end_of_transit = &in_transit;
+
 void prefetch(unsigned char *sha1)
 {
 	char type = 'o';
+	struct object_list *node;
+	if (prefetches > 100) {
+		fetch(in_transit->item->sha1);
+	}
+	node = xmalloc(sizeof(struct object_list));
+	node->next = NULL;
+	node->item = lookup_unknown_object(sha1);
+	*end_of_transit = node;
+	end_of_transit = &node->next;
 	force_write(fd_out, &type, 1);
 	force_write(fd_out, sha1, 20);
-	//memcpy(requested + 20 * prefetches++, sha1, 20);
+	prefetches++;
 }
 
 static char conn_buf[4096];
@@ -51,6 +65,18 @@ int fetch(unsigned char *sha1)
 {
 	int ret;
 	signed char remote;
+	struct object_list *temp;
+
+	if (memcmp(sha1, in_transit->item->sha1, 20)) {
+		// we must have already fetched it to clean the queue
+		return has_sha1_file(sha1) ? 0 : -1;
+	}
+	prefetches--;
+	temp = in_transit;
+	in_transit = in_transit->next;
+	if (!in_transit)
+		end_of_transit = &in_transit;
+	free(temp);
 
 	if (conn_buf_posn) {
 		remote = conn_buf[0];

Re: [PATCH] Limit the number of requests outstanding in ssh-fetch.

[Dan Aloni]

On Tue, Oct 04, 2005 at 12:24:55AM -0400, Daniel Barkalow wrote:
> This completes fetches if there are more than 100 outstanding requests
> and there are more to prefetch.
> 
> Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
> ---
> 
> Lightly tested; I reduced the limit to 5, and pulled a small tree 
> successfully with some of the requests being completed early. I didn't 
> have the test case to verify that a limit of 100 is sufficiently low, but 
> handwaving suggests that it should be.

Good, it seems that your patch works alright with the local Linux complete 
database fetch.

-- 
Dan Aloni
da-x@monatomic.org, da-x@colinux.org, da-x@gmx.net

Re: [PATCH] Limit the number of requests outstanding in ssh-fetch.

[Stefan Roese]

On Tuesday 04 October 2005 09:16, Dan Aloni wrote:
> > Lightly tested; I reduced the limit to 5, and pulled a small tree
> > successfully with some of the requests being completed early. I didn't
> > have the test case to verify that a limit of 100 is sufficiently low, but
> > handwaving suggests that it should be.
>
> Good, it seems that your patch works alright with the local Linux complete
> database fetch.

Works for me too (on a big ssh fetch, which stalled always without this 
patch).

Best regards,
Stefan