Integration branching

Integration branching

[Kai Hendry]

I've been asked to setup & host a git repo of an "integration branch"
of WebKit for our developers.

So I expected it to work like so:

git clone git://git.webkit.org/WebKit.git
git checkout -b test

<Make a few small changes and commit>

git-config remote.upload.url ssh://git.webvm.net/srv/git/webkit-test
x61:~/aplix/test/WebKit% git push --thin -v upload test
Pushing to ssh://git.webvm.net/srv/git/webkit-test
Counting objects: 371612, done.
Compressing objects: 100% (67851/67851), done.
Writing objects:  11% (43886/371612), 25.32 MiB | 88 KiB/

However when uploading the branch it seems to upload _everything_
which is far too big and impractical. Couldn't it just somehow
reference the remote branch and just publish the patch of the "small
changes"?

When doing a git-http-push instead of ssh, git resulted in a "Out of
memory" error:

   1. git clone git://git.webkit.org/WebKit.git # Check it out
   2. git checkout -b test # create a "test" branch and switch to it
   3. echo blah >> ChangeLog # change something
   4. git commit -a # commit that test change
   5. git-config remote.upload.url http://moto@moto.webvm.net/moto.git
# configure the upload url
   6. git push upload test # fatal: Out of memory, malloc failed

Was this the fault of the server(nox) or client(x61)?

x61:~% git --version
git version 1.5.4.5
nox:~% git --version
git version 1.5.4.4

So is there some better way of publishing a branch? Without uploading
the whole WebKit?



I did try something else, that was to create a webkit repo on the
server nox like so:

nox:/srv/git/webkit% git --bare init --shared
nox:/srv/git/webkit% git --bare fetch git://git.webkit.org/WebKit.git
master:master
git fetch -f git://git.webkit.org/WebKit.git master:master # again
forcing updates

And let developers branch from that. This proved a little cumbersome
as I need to manually force updates to webkit trunk when developers
asked me to do that.
However once a developer cloned ssh://git.webvm.net/srv/git/webkit.
Creating a test branch 'deleteme' and then `git push origin deleteme`
was extremely fast. So I guess this is the right way to achieve my
goals?


Any suggestions and tips would be greatly appreciated!

Re: Integration branching

[Junio C Hamano]

"Kai Hendry" <hendry@aplixcorp.com> writes:

> So I expected it to work like so:
>
> git clone git://git.webkit.org/WebKit.git
> git checkout -b test
>
> <Make a few small changes and commit>
>
> git-config remote.upload.url ssh://git.webvm.net/srv/git/webkit-test

A set of big questions is how that git.webvm.net/srv/git/webkit-test
repository was prepared, what its history look like and how it relates to
the history your repository has.  I presume that that is different from
the repository you are pushing from (which is "x61:aplix/test/WebKit"
repository), but other than that you did not give any details to guess
further.

> x61:~/aplix/test/WebKit% git push --thin -v upload test

By the way, that --thin is an implementation detail and not meant for the
end user consumption.  I do not think it would matter in this case,
though.

> Pushing to ssh://git.webvm.net/srv/git/webkit-test
> Counting objects: 371612, done.
> Compressing objects: 100% (67851/67851), done.
> Writing objects:  11% (43886/371612), 25.32 MiB | 88 KiB/
>
> However when uploading the branch it seems to upload _everything_
> which is far too big and impractical. Couldn't it just somehow
> reference the remote branch and just publish the patch of the "small
> changes"?

What you described is exactly what is supposed to be doing, but again as
we do not know how that git.webvm.net:/srv/gi/webkit-test/ repository was
prepared, we cannot tell.

But I have a theory:

    http://thread.gmane.org/gmane.comp.version-control.git/77643

and the latest code is still parked in 'pu' branch.

Re: Integration branching

[Kai Hendry]

On Mon, Apr 7, 2008 at 6:41 PM, Junio C Hamano <gitster@pobox.com> wrote:
>  A set of big questions is how that git.webvm.net/srv/git/webkit-test
>  repository was prepared, what its history look like and how it relates to
>  the history your repository has.  I presume that that is different from
>  the repository you are pushing from (which is "x61:aplix/test/WebKit"
>  repository), but other than that you did not give any details to guess
>  further.

webkit-test was just a `git --bare init --shared` and completely empty
with I tried pushing (publishing) the changes to WebKit trunk.

With thanks,

Re: Integration branching

[Avery Pennarun]

On Mon, Apr 7, 2008 at 2:18 PM, Kai Hendry <hendry@aplixcorp.com> wrote:
> On Mon, Apr 7, 2008 at 6:41 PM, Junio C Hamano <gitster@pobox.com> wrote:
>  >  A set of big questions is how that git.webvm.net/srv/git/webkit-test
>  >  repository was prepared, what its history look like and how it relates to
>  >  the history your repository has.  I presume that that is different from
>  >  the repository you are pushing from (which is "x61:aplix/test/WebKit"
>  >  repository), but other than that you did not give any details to guess
>  >  further.
>
>  webkit-test was just a `git --bare init --shared` and completely empty
>  with I tried pushing (publishing) the changes to WebKit trunk.
>
>  With thanks,

Basically every git repository needs to be entirely self-sufficient,
that is, it should be possible to check out a copy without referring
to any external repositories.  So your initial push to an empty
repository will always upload all the objects, even if you pulled from
some other.

The good news is that if your friends who checked out of the main
repository want to try out your branch, it *won't* accidentally
download all the extra objects over again.  git will notice that they
already have the same objects (retrieved from the main repo) and
download only the new ones.

So yes, some space is "wasted" on your extra repo, but disk space is
cheap anyway (and you now have a backup of the main repo for free :))

Have fun,

Avery

Re: Integration branching

[Kai Hendry]

[-- Attachment #1: Type: text/plain, Size: 880 bytes --]

Ok it makes sense that I need at least one checkout of all the
objects. But still it seems that when I branch and publish via http it
wants to upload zillions of objects.

For example see the attached webkit-http.txt log. I think there is
some bug in the http push transport as if the transport is ssh it's
much *much* faster:

x61:~/aplix/webkit-test/webkit% git-config remote.origin.url
ssh://git.webvm.net/srv/git/webkit
x61:~/aplix/webkit-test/webkit% git push origin deleteme
Counting objects: 5, done.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 289 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
To ssh://git.webvm.net/srv/git/webkit
 * [new branch]      deleteme -> deleteme
x61:~/aplix/webkit-test/webkit%

I need http:// to work as some of the engineers I work with are behind
a "corporate firewall" that doesn't allow ssh://. Crazy, I know.

[-- Attachment #2: webkit-http.txt --]
[-- Type: text/plain, Size: 2993 bytes --]

x61:~/aplix/webkit-test% git clone git://git.webvm.net/webkit
Initialized empty Git repository in /home/hendry/aplix/webkit-test/webkit/.git/
remote: Counting objects: 406041, done.
remote: Compressing objects: 100% (98138/98138), done.
remote: Total 406041 (delta 295642), reused 398766 (delta 288407)
Receiving objects: 100% (406041/406041), 440.93 MiB | 2484 KiB/s, done.
Resolving deltas: 100% (295642/295642), done.
Checking out files: 100% (44750/44750), done.
x61:~/aplix/webkit-test% ll
total 12K
drwxr-xr-x  3 hendry hendry 4.0K 2008-04-08 13:39 ./
drwxr-xr-x 20 hendry hendry 4.0K 2008-04-08 13:22 ../
drwxr-xr-x 15 hendry hendry 4.0K 2008-04-08 13:54 webkit/
x61:~/aplix/webkit-test% cd webkit
x61:~/aplix/webkit-test/webkit% vim .git/config
x61:~/aplix/webkit-test/webkit% git-config remote.origin.url http://moto@moto.webvm.net/webkit.git
x61:~/aplix/webkit-test/webkit% vim .git/config
x61:~/aplix/webkit-test/webkit% ll
total 220K
drwxr-xr-x 15 hendry hendry 4.0K 2008-04-08 13:54 ./
drwxr-xr-x  3 hendry hendry 4.0K 2008-04-08 13:39 ../
-rwxr-xr-x  1 hendry hendry 1.6K 2008-04-08 13:54 autogen.sh*
drwxr-xr-x 12 hendry hendry 4.0K 2008-04-08 13:54 BugsSite/
-rw-r--r--  1 hendry hendry 104K 2008-04-08 13:54 ChangeLog
-rw-r--r--  1 hendry hendry  19K 2008-04-08 13:54 configure.ac
drwxr-xr-x  8 hendry hendry 4.0K 2008-04-08 14:34 .git/
-rw-r--r--  1 hendry hendry 9.6K 2008-04-08 13:54 GNUmakefile.am
drwxr-xr-x 13 hendry hendry 4.0K 2008-04-08 13:54 JavaScriptCore/
drwxr-xr-x  7 hendry hendry 4.0K 2008-04-08 13:54 JavaScriptGlue/
drwxr-xr-x 18 hendry hendry 4.0K 2008-04-08 13:54 LayoutTests/
-rw-r--r--  1 hendry hendry 1.1K 2008-04-08 13:54 Makefile
-rw-r--r--  1 hendry hendry 1.1K 2008-04-08 13:54 Makefile.shared
drwxr-xr-x  3 hendry hendry 4.0K 2008-04-08 13:54 PageLoadTests/
drwxr-xr-x  5 hendry hendry 4.0K 2008-04-08 13:54 PlanetWebKit/
drwxr-xr-x  5 hendry hendry 4.0K 2008-04-08 13:54 SunSpider/
drwxr-xr-x 25 hendry hendry 4.0K 2008-04-08 13:54 WebCore/
drwxr-xr-x  9 hendry hendry 4.0K 2008-04-08 13:54 WebKit/
drwxr-xr-x  4 hendry hendry 4.0K 2008-04-08 13:54 WebKitLibraries/
-rw-r--r--  1 hendry hendry 3.3K 2008-04-08 13:54 WebKit.pri
-rw-r--r--  1 hendry hendry  474 2008-04-08 13:54 WebKit.pro
drwxr-xr-x 14 hendry hendry 4.0K 2008-04-08 13:54 WebKitSite/
drwxr-xr-x 19 hendry hendry 4.0K 2008-04-08 13:54 WebKitTools/
x61:~/aplix/webkit-test/webkit% git checkout -b deleteme
Switched to a new branch "deleteme"
x61:~/aplix/webkit-test/webkit% vim ChangeLog
x61:~/aplix/webkit-test/webkit% git ci
Created commit 4b56425: Test
 1 files changed, 1 insertions(+), 0 deletions(-)
x61:~/aplix/webkit-test/webkit% git push origin deleteme
Fetching remote heads...
  refs/
  refs/heads/
  refs/heads/origin/
  refs/remotes/
  refs/remotes/webkit/
  refs/remotes/origin/
  refs/remotes/debian/
  refs/tags/
updating 'refs/heads/deleteme'
  from 0000000000000000000000000000000000000000
  to   4b5642541a73f105cf6e99b0b0c4ee00f0b869c2
    sending 29257 objects

Re: Integration branching

[Junio C Hamano]

"Kai Hendry" <hendry@aplixcorp.com> writes:

> I need http:// to work as some of the engineers I work with are behind
> a "corporate firewall" that doesn't allow ssh://. Crazy, I know.

It is not "crazy", but simply is "unfortunate".  I am not involved in
http-push at all myself, but I recall there was somebody working on
improving it recently?

Re: Integration branching

[Mike Hommey]

On Tue, Apr 08, 2008 at 11:49:33AM -0700, Junio C Hamano wrote:
> "Kai Hendry" <hendry@aplixcorp.com> writes:
> 
> > I need http:// to work as some of the engineers I work with are behind
> > a "corporate firewall" that doesn't allow ssh://. Crazy, I know.
> 
> It is not "crazy", but simply is "unfortunate".  I am not involved in
> http-push at all myself, but I recall there was somebody working on
> improving it recently?

That was me, but it has been stalled by various debian work in progress.

I hope to be able to work on http-push again in a couple weeks.

Mike

Re: Integration branching

[Kai Hendry]

Hi Mike!

You're busy. :)

So do you acknowledge this as a bug or am I doing something wrong?

If you know where the problem lies, perhaps I can get a C ninja from
work to look at it.



Kind regards from the UK,

Re: Integration branching

[Mike Hommey]

On Tue, Apr 08, 2008 at 09:32:31PM +0100, Kai Hendry wrote:
> Hi Mike!
> 
> You're busy. :)
> 
> So do you acknowledge this as a bug or am I doing something wrong?
> 
> If you know where the problem lies, perhaps I can get a C ninja from
> work to look at it.

Could you provide a trace with GIT_CURL_VERBOSE set so that i can get a
better idea ?

Mike

Re: Integration branching

[Mike Hommey]

On Tue, Apr 08, 2008 at 10:43:19PM +0200, Mike Hommey wrote:
> On Tue, Apr 08, 2008 at 09:32:31PM +0100, Kai Hendry wrote:
> > Hi Mike!
> > 
> > You're busy. :)
> > 
> > So do you acknowledge this as a bug or am I doing something wrong?
> > 
> > If you know where the problem lies, perhaps I can get a C ninja from
> > work to look at it.
> 
> Could you provide a trace with GIT_CURL_VERBOSE set so that i can get a
> better idea ?

FWIW, it was made clear from off-list discussion with Kai that his
problem was a lack of running git update-server-info when required.

Mike

Re: Integration branching

[Junio C Hamano]

"Kai Hendry" <hendry@aplixcorp.com> writes:

> On Mon, Apr 7, 2008 at 6:41 PM, Junio C Hamano <gitster@pobox.com> wrote:
>>  A set of big questions is how that git.webvm.net/srv/git/webkit-test
>>  repository was prepared, what its history look like and how it relates to
>>  the history your repository has.  I presume that that is different from
>>  the repository you are pushing from (which is "x61:aplix/test/WebKit"
>>  repository), but other than that you did not give any details to guess
>>  further.
>
> webkit-test was just a `git --bare init --shared` and completely empty
> with I tried pushing (publishing) the changes to WebKit trunk.

That explains it without the "theory" in my previous message.  The
destination repository must be populated with full history somehow (and
one way would have been to start it by cloning from the upstream), but if
it was empty that is alright as well, because your push made it
up-to-date.  IOW, there is nothing to worry about and the system is
working as designed.

Further pushes will update it incrementally.