[PATCH] Introduce experimental remote object access mode

[PATCH] Introduce experimental remote object access mode

[Shawn Pearce]

Make it easy to experiment what remote access to objects would be
like if the network ran at say 1 ms round trip latency to obtain
any object not on the local repository.

  $ time git ls-tree -r HEAD
  real 0m0.059s

  $ time GIT_RTT=1 git ls-tree -r HEAD
  real 0m27.283s

Yes kids, slowing down loose object access by just 1ms if all
objects are remote can take a simple ls-tree from 59ms to more
than enough time to drink tea or coffee.

Why would you do this? Perhaps you need more time in your day
to consume tea or coffee. Set GIT_RTT and enjoy a beverage.

So-not-signed-off-by: this author or anyone else
---

  :-)

 sha1_file.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/sha1_file.c b/sha1_file.c
index 6e8c05d..9bdcbc3 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -38,6 +38,7 @@ const unsigned char null_sha1[20];

 static const char *no_log_pack_access = "no_log_pack_access";
 static const char *log_pack_access;
+static useconds_t rtt;

 /*
  * This is meant to hold a *small* number of objects that you would
@@ -436,9 +437,20 @@ void prepare_alt_odb(void)
  read_info_alternates(get_object_directory(), 0);
 }

+static void apply_rtt()
+{
+ if (!rtt) {
+ char *rtt_str = getenv("GIT_RTT");
+ rtt = rtt_str ? strtoul(rtt_str, NULL, 10) * 1000 : 1;
+ }
+ if (rtt > 1)
+ usleep(rtt);
+}
+
 static int has_loose_object_local(const unsigned char *sha1)
 {
  char *name = sha1_file_name(sha1);
+ apply_rtt();
  return !access(name, F_OK);
 }

@@ -1303,6 +1315,7 @@ void prepare_packed_git(void)

  if (prepare_packed_git_run_once)
  return;
+
  prepare_packed_git_one(get_object_directory(), 1);
  prepare_alt_odb();
  for (alt = alt_odb_list; alt; alt = alt->next) {
@@ -1439,6 +1452,7 @@ static int open_sha1_file(const unsigned char *sha1)
  struct alternate_object_database *alt;

  fd = git_open_noatime(name);
+ apply_rtt();
  if (fd >= 0)
  return fd;

-- 
1.9.0.rc1.175.g0b1dcb5

Re: [PATCH] Introduce experimental remote object access mode

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

> Why would you do this? Perhaps you need more time in your day
> to consume tea or coffee. Set GIT_RTT and enjoy a beverage.

So the conclusion is that it is not practical to do a lazy fetch if
it is done extremely naively at "we want this object --- wait a bit
and we'll give you" level?

I am wondering if we can do a bit better, like "we want this object
--- wait a bit, ah that's a commit, so it is likely that you may
want the trees and blobs associated with it, too, if not right now
but in a near future, let me push a pack that holds them to you"?

>
> So-not-signed-off-by: this author or anyone else
> ---
>
>   :-)
>
>  sha1_file.c | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
>
> diff --git a/sha1_file.c b/sha1_file.c
> index 6e8c05d..9bdcbc3 100644
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -38,6 +38,7 @@ const unsigned char null_sha1[20];
>
>  static const char *no_log_pack_access = "no_log_pack_access";
>  static const char *log_pack_access;
> +static useconds_t rtt;
>
>  /*
>   * This is meant to hold a *small* number of objects that you would
> @@ -436,9 +437,20 @@ void prepare_alt_odb(void)
>   read_info_alternates(get_object_directory(), 0);
>  }
>
> +static void apply_rtt()
> +{
> + if (!rtt) {
> + char *rtt_str = getenv("GIT_RTT");
> + rtt = rtt_str ? strtoul(rtt_str, NULL, 10) * 1000 : 1;
> + }
> + if (rtt > 1)
> + usleep(rtt);
> +}
> +
>  static int has_loose_object_local(const unsigned char *sha1)
>  {
>   char *name = sha1_file_name(sha1);
> + apply_rtt();
>   return !access(name, F_OK);
>  }
>
> @@ -1303,6 +1315,7 @@ void prepare_packed_git(void)
>
>   if (prepare_packed_git_run_once)
>   return;
> +
>   prepare_packed_git_one(get_object_directory(), 1);
>   prepare_alt_odb();
>   for (alt = alt_odb_list; alt; alt = alt->next) {
> @@ -1439,6 +1452,7 @@ static int open_sha1_file(const unsigned char *sha1)
>   struct alternate_object_database *alt;
>
>   fd = git_open_noatime(name);
> + apply_rtt();
>   if (fd >= 0)
>   return fd;

Re: [PATCH] Introduce experimental remote object access mode

[Shawn Pearce]

On Tue, Feb 11, 2014 at 11:29 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Shawn Pearce <spearce@spearce.org> writes:
>
>> Why would you do this? Perhaps you need more time in your day
>> to consume tea or coffee. Set GIT_RTT and enjoy a beverage.
>
> So the conclusion is that it is not practical to do a lazy fetch if
> it is done extremely naively at "we want this object --- wait a bit
> and we'll give you" level?

Yes, this is what I thought when someone proposed this hack in
sha1_file.c to me on Monday. So I ran a quick experiment to see if my
instinct was right.

> I am wondering if we can do a bit better, like "we want this object
> --- wait a bit, ah that's a commit, so it is likely that you may
> want the trees and blobs associated with it, too, if not right now
> but in a near future, let me push a pack that holds them to you"?

Ah, smart observation. That might work. However I doubt it.

I implemented a version of Git on top of Google Bigtable (and Apache
HBase and Apache Cassandra) multiple times using JGit. tl;dr: this
approach doesn't work in practice.


The naive implementation for these distributed NoSQL systems is to
store each object in its own row keyed by SHA-1, and lookup the object
when you want it. This is very slow and is more or less what this
stupid patch shows. Worse, none of them were able to even get close to
the 1ms latency I used in this example.

In another implementation (which I published into JGit as the "DHT"
backend) I stored a group of related commits together in a row. Row
target sizes were in the 1-2 MiB range when using pack style
compression for commits, so the average row held hundreds of commits.
Reading one commit would actually slurp back a number of related
commits. The idea was if we need commit A now we will need B, C, D, E
(its parents and ancestors) soon as the application walks the revision
history, like rev-list or pack-objects.

For a process like pack-objects this almost seems to work. If commits
are together we can slurp a group at a time to amortize the round trip
latency. Unfortunately the application can still go through hundreds
of commits faster than the real world RTT is. So I tried to fix this
by storing an extra metadata pointer in each row to identify the next
row, so next block of commits could start loading right away. Its
still slow, as the application can scan through data faster than the
RTT.

At least for pack generation the traversal code does commits and
builds up a list of all root trees. The root trees can be async loaded
in batches, but the depth first traversal is still a killer. There are
stalls while the application waits for the next subtree, even if you
cluster the trees also into groups using depth first traversal the way
the packer produces pack files today.


Junio's idea to cluster data by commit and its related trees and blobs
is just a different data organization. It may be necessary to make two
copies of the data, one clustered by commits and another by
commit+tree+blob to satisfy different access patterns. And in the
commit+tree+blob case you may need multiple redundant copies of blobs
near commits that use them if those commits are frequently accessed.
Its a lot of redundant disk space.

We always say disk is cheap, but disk is slow and not getting faster.
SSDs are helping, but SSDs are expensive and have size limitations
compared to spinning disk. Just making many copies of data isn't
necessarily a solution.