Darcs-Git: upgrading to Git 0.99

Darcs-Git: upgrading to Git 0.99

[Juliusz Chroboczek]

[CC'd to the Git mailling list; please CC any replies to Darcs-Devel]

David, Ian,

I'd like to upgrade the Git code used in Darcs to 0.99 (we're
currently using 0.6).  There are two good reasons for that, the first
of which is actually a showstopper:

 - the format of Git repositories has changed incompatibly, with a new
   kind of thing called the ``pack'' (a very neat performance hack, by
   the way); hence, Darcs-Git is unable to read recent Git repos,
   unless you use the Git tools to unpack them;

 - 0.99 actually exports usable interfaces, which will allow us to use
   pristine Git sources in Darcs.

Now I'm wondering how to do that.  Currently, I'm using a nasty hack
using the C preprocessor to include just the sources we need in
Darcs.  As 0.99 builds a ``libgit.a'', I'd like to use that instead.

There are three ways to do that:

  (1) require that the users put a suitable libgit.a in /usr/local/lib
      before building Darcs, and distribute a tarball of Git from
      darcs.net;

  (2) include just the sources needed for libgit.a in Darcs, and have
      the Darcs build build a local libgit

  (3) as (2), but include all of Git, including their
      ``user-friendly'' scripts.

Solution (2) will include 33 files totalling 167KB, while (3) is about
a megabyte of source.

My personal favourite is solution (2), as it is simple for both the
users and us.  I'm not very keen on (1), as it will cause problems
when the friendly Git folks change their interfaces, but have no
strong dislike towards it if it's what you think is right.  (3) is
definitely overkill.

                                        Juliusz

Re: [darcs-devel] Darcs-Git: upgrading to Git 0.99

[David Roundy]

On Sat, Jul 16, 2005 at 10:45:47PM +0200, Juliusz Chroboczek wrote:
> 
> I'd like to upgrade the Git code used in Darcs to 0.99 (we're
> currently using 0.6). [...]

Great!

> Now I'm wondering how to do that.  Currently, I'm using a nasty hack
> using the C preprocessor to include just the sources we need in
> Darcs.  As 0.99 builds a ``libgit.a'', I'd like to use that instead.
> 
> There are three ways to do that:
> 
>   (1) require that the users put a suitable libgit.a in /usr/local/lib
>       before building Darcs, and distribute a tarball of Git from
>       darcs.net;
> 
>   (2) include just the sources needed for libgit.a in Darcs, and have
>       the Darcs build build a local libgit
> 
>   (3) as (2), but include all of Git, including their
>       ``user-friendly'' scripts.
> 
> Solution (2) will include 33 files totalling 167KB, while (3) is about
> a megabyte of source.

I'd really prefer option (1), *if* the git folks can confirm that the API
is at least intended to be stable.  As an subtly different option, we could
include a script that would download and untar the git sources and then
build them.  But it'd be great to allow users to upgrade their libgit
without our intervention if a protocol or repository format change occurs
that doesn't affect the API.

I guess the real question is whether the API is more or less stable than
the protocols and disk formats.  If the API is more stable, we'd rather
link with an external libgit and be robust with respect to on-disk format
changes (such as pack files).  If the on-disk format is more stable, we'd
rather include a copy of the source code and be robust with respect to API
changes of libgit.

A fourth option would be to include git sources, but also include a
configure flag that could be used to link with an external libgit.  This is
probably the most robust solution, but also the most complex solution (and
thus probably not the best).
-- 
David Roundy
http://www.darcs.net

git, porcelain, darcs, and version 1.0

[Bryan Larsen]

Juliusz Chroboczek wrote:
> There are three ways to do that:
> 
>   (1) require that the users put a suitable libgit.a in /usr/local/lib
>       before building Darcs, and distribute a tarball of Git from
>       darcs.net;

I was under the impression that the stablest interface to git was the 
command line; we use spawnvp in stacked git.  I've been using it with 
repositories that make the Linux kernel look small.

I certainly don't think the lib interface is anywhere near stable: 
Linus accepted my change to index_fd far too easily.

 >
 >   (2) include just the sources needed for libgit.a in Darcs, and have
 >       the Darcs build build a local libgit
 >
 >   (3) as (2), but include all of Git, including their
 >       ``user-friendly'' scripts.

Ugh.  That's what they do in the commercial world.  We have it so much 
better here in Linux & BSD land: you just add a "depends libgit1" line 
to your package, and the right thing happens: minor updates happen 
automatically and changes that break the interface don't.  Of course, 
keeping it that easy for the user requires active effort from us, the 
developers, and sometimes the pain spills over, but the benefits are nice.

I see too major long-term alternatives:

1) talk the darcs guys into using spawnvp
2) talk the git people into exporting a stable lib interface

It's my opinion that #1 is a non-starter.  We want others to interact 
with us.  I'm going to use spawnvp for my porcelain, but we should be 
inclusive.

The current 'libgit' probably contains more than Linus and Junio are 
comfortable locking down as a stable interface, but I'm sure that 
there's a subset that they'd be comfortable with once a relative amount 
of stability is achieved, or it may be achievable via some other method.

I propose that Darcs includes all of git for now.  (I prefer this over 
partial inclusion; anybody should be able to take the darcs sources and 
easily drop in a later git).  However, the long term goal is for a 
library.  Darcs and git work together to determine the minimal amount 
that needs to go into libgit1.so.  It gets blessed by being documented, 
and doesn't change until libgit2.so.

I'd like to see this added to Junio's list of "1.0" goals.

A similar 1.0 goal would be to document porcelain's use of the .git 
directory.  For instance, stacked git uses .git/patches, 
.git/patchdescr.tmpl and .git/patchexport.tmpl.  If Linus does not 
accept a patch documenting this usage, stacked git should use .stgit 
instead.

Bryan

Re: git, porcelain, darcs, and version 1.0

[Junio C Hamano]

Bryan Larsen <bryan.larsen@gmail.com> writes:

> ... Darcs and git work together to determine the minimal amount
> that needs to go into libgit1.so.  It gets blessed by being
> documented, and doesn't change until libgit2.so.
>
> I'd like to see this added to Junio's list of "1.0" goals.

I should mention that I liked the libification work Brad Roberts
did around late April.  Unfortunately the things have been
always in great flux for the last couple of months, and it was
too hard to include that and maintain it back then.

I fully agree that supporting C-level linkage is worthy, and
should be one of our longer term goals.

> A similar 1.0 goal would be to document porcelain's use of the .git
> directory.  For instance, stacked git uses .git/patches,
> .git/patchdescr.tmpl and .git/patchexport.tmpl.  If Linus does not
> accept a patch documenting this usage, stacked git should use .stgit
> instead.

I agree that coordinating the namespace under $GIT_DIR among
Porcelains is something we need (it was what prompted me to
steal the branches/ convention from Cogito).  The job of the
core should be to help Porcelains avoid stepping on each other's
toes.

The documentation of the internals for $GIT_DIR/patches is
probably better left to StGIT documentation, though, at the
moment.  When other Porcelains start wishing to access the
"series of patches expressed as a set of commit chain" expressed
by StGIT there (e.g. show patch series in addition to regular
commit chain in gitk), the core should help the Porcelains to
work well with each other, to do things in a compatible way.
This may involve moving some common things to core side and
mention the convention for Porcelains to work well together in
the core documentation.

However, I am slightly negative about suggesting these two to be
part of the 1.0 goals.  Linus wanted to make 1.0 how many weeks
ago?  I personally think that a usable baseline, stable enough
to allow stripping out the core part currently shipped as part
of Cogito, would be a good place to stop and declare 1.0.  My
list was meant to enumuerate what might be missing from the
"usable baseline".

Re: git, porcelain, darcs, and version 1.0

[Juliusz Chroboczek]

> I certainly don't think the lib interface is anywhere near stable:
> Linus accepted my change to index_fd far too easily.

Noted, thanks for the info.

(This makes a lot of sense, Git is evolving very fast.  I haven't
looked at Git since mid-April, and I'm very much impressed at the
difference between 0.6 and 0.99.)

> Ugh.  That's what they do in the commercial world.  We have it so much
> better here in Linux & BSD land: you just add a "depends libgit1" line
> to your package, and the right thing happens: minor updates happen
> automatically and changes that break the interface don't.

This is, of course, only possible when there are stable interfaces,
which in turn make change problematic.  (Which, as far as I
understand, is the very reason why the Linux kernel tree contains
everyone's and his brother's driver rather than having a stable module
ABI, but that's besides the point.)

> Darcs and git work together to determine the minimal amount
> that needs to go into libgit1.so.

Hold on...  Nobody is speaking about *binary* compatibility, it's
source-level compatibility that we need.  There is absolutely no
reason to introduce the complexities of shared libraries into the
picture.

                                        Juliusz

Re: git, porcelain, darcs, and version 1.0

[Bryan Larsen]

Junio C Hamano wrote:
> 
> I fully agree that supporting C-level linkage is worthy, and
> should be one of our longer term goals.

Excellent.
> 
> 
>>A similar 1.0 goal would be to document porcelain's use of the .git
>>directory.  For instance, stacked git uses .git/patches,
>>.git/patchdescr.tmpl and .git/patchexport.tmpl.  If Linus does not
>>accept a patch documenting this usage, stacked git should use .stgit
>>instead.
> 
> 
> I agree that coordinating the namespace under $GIT_DIR among
> Porcelains is something we need (it was what prompted me to
> steal the branches/ convention from Cogito).  The job of the
> core should be to help Porcelains avoid stepping on each other's
> toes.
> 
> The documentation of the internals for $GIT_DIR/patches is
> probably better left to StGIT documentation, though, at the
> moment.  When other Porcelains start wishing to access the
> "series of patches expressed as a set of commit chain" expressed
> by StGIT there (e.g. show patch series in addition to regular
> commit chain in gitk), the core should help the Porcelains to
> work well with each other, to do things in a compatible way.
> This may involve moving some common things to core side and
> mention the convention for Porcelains to work well together in
> the core documentation.

I think we want the same thing, you're just expressing it explicitly: 
stgit's usage of the .git namespace should be mentioned in git 
documentation.  actual details belong in stgit, either explicitly in 
documentation or implicitly in the code.


> 
> However, I am slightly negative about suggesting these two to be
> part of the 1.0 goals.  Linus wanted to make 1.0 how many weeks
> ago?  I personally think that a usable baseline, stable enough
> to allow stripping out the core part currently shipped as part
> of Cogito, would be a good place to stop and declare 1.0.  My
> list was meant to enumuerate what might be missing from the
> "usable baseline".

All I'm looking for is a statement like "once we're at 1.0, darcs 
doesn't break until 2.0".  If we don't actually break out a blessed lib 
interface until 1.1,  that's fine with me.  To me, 1.0 implies core 
stability.

thanks,
Bryan

Re: git, porcelain, darcs, and version 1.0

[Bryan Larsen]

>>Darcs and git work together to determine the minimal amount
>>that needs to go into libgit1.so.
> 
> 
> Hold on...  Nobody is speaking about *binary* compatibility, it's
> source-level compatibility that we need.  There is absolutely no
> reason to introduce the complexities of shared libraries into the
> picture.
> 

Source level compatibility and stability is the big deal.  Compared to 
that, shared libraries are an implementation detail, in my opinion. 
Sometimes those details get "interesting", but they are soluble.

I could care less whether you use libgit.a or a libgit.so.  Just as long 
as distros or anybody else can update their darcs if a major data-loss 
bug is found in git.  A recompile is acceptable.  Dealing with the 
addition of a parameter to index_fd() is not.

Bryan