Core and Not-So Core

Core and Not-So Core

[Jon Seymour]

I have been experimenting with pure-Java implementation of GIT
concepts with a goal of eventually providing plugins to Eclipse to
allow the Eclipse GUI to interact with GIT repositories.

One thing I noticed when doing this is that the present index/cache
structure is rather arbitrary and the optimum index structure is
determined by the structure of the tools that use a GIT repository
rather than the structure of the GIT repository itself.

To give a concrete example: the cache currently contains most of the
posix stat structure primarily to allow quick change detection. In the
Java world, most of the posix stat structure is not directly
accessible via the pure-Java file system abstractions. However, for
most purposes detecting changes to files modification time and file
size would be enough. Given this is the case, a Java-GIT client
doesn't need to bother getting access to a posix stat structure and
could therefore get away with a simpler  index structure, provided it
doesn't need to interoperate with a 'C'-GIT client that shared the
same workspace. A Java-GIT client might also choose to represent an
index cache as a complex serialized Java object graph or (perhaps) an
XML document.

Another example: I can imagine a variant of the index file structure
that recorded all the parents which have been merged into the cache
and automatically include this information when performing the commit.

The point is that many different index file structures are possible
and will be determined in part by the tooling created in the porcelain
layer - there really is no one true index file format as there is a
one true repository format. Different tools can use different index
file formats and still interoperate at the repository level because
only the repository format needs to have a solid, unchanging
definition.

Currently the GIT stack is structured as follows:

cogito
git-core 

I think it would be worthwhile if care was taken to draw a distinction
between the repository and the cache aspects of the git core, perhaps
even going to the extreme of moving all knowledge of the  cache into
cogito itself. By clearly drawing this distinction, we will more
easily enable the creation of different kind of tools sets atop the
foundation of the GIT repository format.

e.g., either:

cogito
git-cache
git-respository

or:

cogito-tools
cogito-cache
git-repository

Anyway, I offer this as food for thought - chew or flame away as appropriate!

jon.
-- 
homepage: http://www.zeta.org.au/~jon/
blog: http://orwelliantremors.blogspot.com/

Re: Core and Not-So Core

[David Woodhouse]

On Wed, 2005-05-11 at 01:00 +1000, Jon Seymour wrote:
> I have been experimenting with pure-Java implementation of GIT
> concepts with a goal of eventually providing plugins to Eclipse to
> allow the Eclipse GUI to interact with GIT repositories.

It's not April 1st. Why would you want to reimplement it in Java instead
of just using the existing implementation? Is this a religious issue?

-- 
dwmw2

Re: Core and Not-So Core

[Eduardo Teixeira Dias]

He want's to make an eclipce plugin...

This must be done in Java.

> On Wed, 2005-05-11 at 01:00 +1000, Jon Seymour wrote:
>> I have been experimenting with pure-Java implementation of GIT
>> concepts with a goal of eventually providing plugins to Eclipse to
>> allow the Eclipse GUI to interact with GIT repositories.
>
> It's not April 1st. Why would you want to reimplement it in Java instead
> of just using the existing implementation? Is this a religious issue?
>
> --
> dwmw2
>
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
----------------------------
Eduardo Teixeira Dias
Tendencies Consultoria
Tel/Fax:   +55 11 3828-1281
http://www.tendencies.com.br

Re: Core and Not-So Core

[David Woodhouse]

On Tue, 2005-05-10 at 12:50 -0300, Eduardo Teixeira Dias wrote:
> He want's to make an eclipce plugin...

And there's no way for him to call execve()?

Hell, even the lisp nutters manage to make emacs invoke external
programs occasionally. What makes Java programmers so much less sane?

-- 
dwmw2

Re: Core and Not-So Core

[Eduardo Teixeira Dias]

> On Tue, 2005-05-10 at 12:50 -0300, Eduardo Teixeira Dias wrote:
>> He want's to make an eclipce plugin...
>
> And there's no way for him to call execve()?
>
> Hell, even the lisp nutters manage to make emacs invoke external
> programs occasionally. What makes Java programmers so much less sane?
>
> --
> dwmw2
Good point!

It can be done using execxxx().

But a Java version is better for this particular use (Eclipce Plugin):

  - Just a .jar download
  - Installation without external dependencies

Re: Core and Not-So Core

[Diego Calleja]

El Tue, 10 May 2005 13:19:49 -0300 (BRT),
"Eduardo Teixeira Dias" <eduardo@tendencies.com.br> escribió:

>   - Just a .jar download
>   - Installation without external dependencies


Someone who is going to hack the kernel can very well install more things.
And anyway, git is "the linux SCM tool" so all distros will package it. Also,
people who hacks the linux kernel usually runs it, so "git is not ported
to win32" is not a big problem.

...and I don't think people who use eclipse wants to have the fastest tool
on earth, so "java-invoking-C is slower than pure java" is not a great excuse
either...

Code reuse == good. Java programmers should know that. Anyway, people
is free to do whatever they want with their time :)

Re: Core and Not-So Core

[Eduardo Teixeira Dias]

I'm a Java programmer and I was thinking like one (Culture thing).

The reuse argument is very Strong.

The git-core are ready and it's a very good idea to reuse it. Any change
in the core or in the internal representation of git repository will
benefit the Git-Java implementation...

I think that a growing utilization of GCJ and ClassPath will contribute to
a Gradual Culture Change in the Java Land and promote a beter
reutilization among all GCC languages...

Write Once Run Everywhere that GCJ Runs (Sounds Good).

Eclipse will be compiled with GCJ in Fedora 4. Let's see how it works...

Cheers!

> El Tue, 10 May 2005 13:19:49 -0300 (BRT),
> "Eduardo Teixeira Dias" <eduardo@tendencies.com.br> escribió:
>
>>   - Just a .jar download
>>   - Installation without external dependencies
>
>
> Someone who is going to hack the kernel can very well install more
> things. And anyway, git is "the linux SCM tool" so all distros will
> package it. Also, people who hacks the linux kernel usually runs it, so
> "git is not ported to win32" is not a big problem.
>
> ...and I don't think people who use eclipse wants to have the fastest
> tool on earth, so "java-invoking-C is slower than pure java" is not a
> great excuse either...
>
> Code reuse == good. Java programmers should know that. Anyway, people is
> free to do whatever they want with their time :)

Re: Core and Not-So Core

[Diego Calleja]

[somewhat OT]

El Tue, 10 May 2005 19:33:14 -0300 (BRT),
"Eduardo Teixeira Dias" <eduardo@tendencies.com.br> escribió:

> Write Once Run Everywhere that GCJ Runs (Sounds Good).

GCJ in fact supports more architectures than Sun's java VM so with GCJ you _really_
can run your java code everywhere.

Re: Core and Not-So Core

[Horst von Brand]

Diego Calleja <diegocg@gmail.com> said:
> El Tue, 10 May 2005 19:33:14 -0300 (BRT),
> "Eduardo Teixeira Dias" <eduardo@tendencies.com.br> escribió:
> > Write Once Run Everywhere that GCJ Runs (Sounds Good).

> GCJ in fact supports more architectures than Sun's java VM so with GCJ
> you _really_ can run your java code everywhere.

But then again, a half decent C implementation is even more ubicuous, so...
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: Core and Not-So Core

[Eduardo Teixeira Dias]

I'm a Java programmer and I was thinking like one (Culture thing).

The reuse argument is very Strong.

The git-core are ready and it's a very good idea to reuse it. Any change
in the core or in the internal representation of git repository will
benefit the Git-Java implementation...

I think that a growing utilization of GCJ and ClassPath will contribute to
a Gradual Culture Change in the Java Land and promote a beter
reutilization among all GCC languages...

Write Once Run Everywhere that GCJ Runs (Sounds Good).

Eclipse will be compiled with GCJ in Fedora 4. Let's see how it works...

Cheers!

> El Tue, 10 May 2005 13:19:49 -0300 (BRT),
> "Eduardo Teixeira Dias" <eduardo@tendencies.com.br> escribió:
>
>>   - Just a .jar download
>>   - Installation without external dependencies
>
>
> Someone who is going to hack the kernel can very well install more
> things. And anyway, git is "the linux SCM tool" so all distros will
> package it. Also, people who hacks the linux kernel usually runs it, so
> "git is not ported to win32" is not a big problem.
>
> ...and I don't think people who use eclipse wants to have the fastest
> tool on earth, so "java-invoking-C is slower than pure java" is not a
> great excuse either...
>
> Code reuse == good. Java programmers should know that. Anyway, people is
> free to do whatever they want with their time :)

Re: Core and Not-So Core

[Eduardo Teixeira Dias]

I'm a Java programmer and I was thinking like one (Culture thing).

The reuse argument is very Strong.

The git-core are ready and it's a very good idea to reuse it. Any change
in the core or in the internal representation of git repository will
benefit the Git-Java implementation...

I think that a growing utilization of GCJ and ClassPath will contribute to
a Gradual Culture Change in the Java Land and promote a beter
reutilization among all GCC languages...

Write Once Run Everywhere that GCJ Runs (Sounds Good).

Eclipse will be compiled with GCJ in Fedora 4. Let's see how it works...

Cheers!

> El Tue, 10 May 2005 13:19:49 -0300 (BRT),
> "Eduardo Teixeira Dias" <eduardo@tendencies.com.br> escribió:
>
>>   - Just a .jar download
>>   - Installation without external dependencies
>
>
> Someone who is going to hack the kernel can very well install more
> things. And anyway, git is "the linux SCM tool" so all distros will
> package it. Also, people who hacks the linux kernel usually runs it, so
> "git is not ported to win32" is not a big problem.
>
> ...and I don't think people who use eclipse wants to have the fastest
> tool on earth, so "java-invoking-C is slower than pure java" is not a
> great excuse either...
>
> Code reuse == good. Java programmers should know that. Anyway, people is
> free to do whatever they want with their time :)

Re: Core and Not-So Core

[Petr Baudis]

Dear diary, on Tue, May 10, 2005 at 11:45:01PM CEST, I got a letter
where Diego Calleja <diegocg@gmail.com> told me that...
> Someone who is going to hack the kernel can very well install more things.
> And anyway, git is "the linux SCM tool" so all distros will package it. Also,
> people who hacks the linux kernel usually runs it, so "git is not ported
> to win32" is not a big problem.

It's not like everything git is ever going to be used for is kernel and
only the kernel.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor

Re: Core and Not-So Core

[Andreas Gal]

Btw, I rewrote about 90% of git in python, just out of curiousity. 
Performance is pretty competitive, and I can optimize certain operations 
by keeping the index in memory temporarily. It also saves a lot of output 
parsing, simplifies error-handling (if the frontend is written in python 
too, which bit now is), and the best of all: its about 600 lines of python 
code. I am all for a Java implementation. After its done we will see 
whether it works and how well it works. We can still throw it away if it 
sucks. 

Andreas

On Wed, 11 May 2005, Petr Baudis wrote:

> Dear diary, on Tue, May 10, 2005 at 11:45:01PM CEST, I got a letter
> where Diego Calleja <diegocg@gmail.com> told me that...
> > Someone who is going to hack the kernel can very well install more things.
> > And anyway, git is "the linux SCM tool" so all distros will package it. Also,
> > people who hacks the linux kernel usually runs it, so "git is not ported
> > to win32" is not a big problem.
> 
> It's not like everything git is ever going to be used for is kernel and
> only the kernel.
> 
> -- 
> 				Petr "Pasky" Baudis
> Stuff: http://pasky.or.cz/
> C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Core and Not-So Core

[James Purser]

On Wed, 2005-05-11 at 08:44, Petr Baudis wrote:
> Dear diary, on Tue, May 10, 2005 at 11:45:01PM CEST, I got a letter
> where Diego Calleja <diegocg@gmail.com> told me that...
> > Someone who is going to hack the kernel can very well install more things.
> > And anyway, git is "the linux SCM tool" so all distros will package it. Also,
> > people who hacks the linux kernel usually runs it, so "git is not ported
> > to win32" is not a big problem.
> 
> It's not like everything git is ever going to be used for is kernel and
> only the kernel.
So does this mean I can keep working on my wxWidgets implementation? :)
-- 
James Purser
http://ksit.dynalias.com

Re: Core and Not-So Core

[Christoph Hellwig]

On Tue, May 10, 2005 at 11:45:01PM +0200, Diego Calleja wrote:
> ...and I don't think people who use eclipse wants to have the fastest tool
> on earth, so "java-invoking-C is slower than pure java" is not a great excuse
> either...

Note that it's also not true.  At least if you use the gcj-specific CNI
mechanisms instead of Sun's braindead JNI.

Re: Core and Not-So Core

[Jon Seymour]

On 5/11/05, Christoph Hellwig <hch@infradead.org> wrote:
> On Tue, May 10, 2005 at 11:45:01PM +0200, Diego Calleja wrote:
> > ...and I don't think people who use eclipse wants to have the fastest tool
> > on earth, so "java-invoking-C is slower than pure java" is not a great excuse
> > either...
> 
> Note that it's also not true.  At least if you use the gcj-specific CNI
> mechanisms instead of Sun's braindead JNI.
> 
> 

What I actually claimed (contrary to Diego's misquote) is that
"Java-invokes-C-executable" would be slower than pure-Java. I didn't
make a claim one way or the other about the performance of a Java
implementation invoking some form of native interface.

jon.
-- 
homepage: http://www.zeta.org.au/~jon/
blog: http://orwelliantremors.blogspot.com/

Re: Core and Not-So Core

[Jon Seymour]

On 5/11/05, David Woodhouse <dwmw2@infradead.org> wrote:
> On Wed, 2005-05-11 at 01:00 +1000, Jon Seymour wrote:
> > I have been experimenting with pure-Java implementation of GIT
> > concepts with a goal of eventually providing plugins to Eclipse to
> > allow the Eclipse GUI to interact with GIT repositories.
> 
> It's not April 1st. Why would you want to reimplement it in Java instead
> of just using the existing implementation? Is this a religious issue?
> 

Not really - in the Java world, things are simpler if you don't have
to carry around a JNI library - that way, you can just run it wherever
a Java interpreter exists and not worry about
having someone recompiling the JNI part. 

A pure-Java implementation will perform better than a
Java-invokes-C-executable approach [ though not, of course, a C-only
approach ].

Another benefit of playing around with a Java abstraction is that I
can more easily experiment with abstractions than I can in C since the
abstraction-facilities of Java more directly support such playing than
does C. So, for example, I can easily create a virtual repository
which layers a local repository over a remote repository and
transparently populates the local repository from the remote
repository "on-demand". Of course, this sort of thing can be done in
C, but it requires much more "work" to set up the abstractions.

You could perhaps argue that the existing 'C' tools in some way
encapsulate access  to the repository format. Maybe, but the fact is
the rapid adoption of GIT has already effectively fixed the GIT
repository format in practice so any change in repository format will
require considerable planning anyway - planning that will allow
sufficient time for implementations in other languages to catch up.

So, no, it's not a religious issue. If anything, it is being dogmatic
to insist that the sacred GIT repository structure only be manipulated
by 'C' tools blessed by the hands of Linus.

The concepts in GIT are bigger than the programming language its
toolsets are implemented in.

jon.
-- 
homepage: http://www.zeta.org.au/~jon/
blog: http://orwelliantremors.blogspot.com/

Re: Core and Not-So Core

[David Woodhouse]

On Wed, 2005-05-11 at 02:22 +1000, Jon Seymour wrote:
> So, no, it's not a religious issue. If anything, it is being dogmatic
> to insist that the sacred GIT repository structure only be manipulated
> by 'C' tools blessed by the hands of Linus.

Given the volatility of the structure -- at least the details if not the
fundamentals -- it seems bizarre to want to reimplement it rather than
just using the existing tools.

This is the same mentality which gives Eclipse a half-arsed SSH
reimplementation which doesn't behave like normal SSH is configured to
behave either, right?

-- 
dwmw2

Core and Not-So Core

[Jon Seymour]

On 5/11/05, David Woodhouse <dwmw2@infradead.org> wrote:
> On Wed, 2005-05-11 at 02:22 +1000, Jon Seymour wrote:
> > So, no, it's not a religious issue. If anything, it is being dogmatic
> > to insist that the sacred GIT repository structure only be manipulated
> > by 'C' tools blessed by the hands of Linus.
>
> Given the volatility of the structure -- at least the details if not the
> fundamentals -- it seems bizarre to want to reimplement it rather than
> just using the existing tools.

I did consider wrapping it - I really did. But after thinking about it
for a couple of weeks
I eventually came to the conclusion it would be a sub-optimal solution.

And I don't agree that the structure is volatile. The actual structure
of GIT repository has been rock-solid since Linus reversed the
compression-signature order. Yes, there has been the excellent delta
work that Nicholas has been working on, but a Java version doesn't
need to use that anymore than Linus himself does [ and last time I
checked (the mailing list), Linus was being rather conservative about
that ].

What _has_ been changing at a great rate recently is the behaviour of
the tooling layer.

So, if I want a stable foundation to build my stuff on, basing it on
the output of the C tools would be a huge mistake. No, I think it
would be far safer for me to build my tooling using assumptions that
have proven to be rock-solid over the last few weeks - the structure
of the GIT repository format itself.

>
> This is the same mentality which gives Eclipse a half-arsed SSH
> reimplementation which doesn't behave like normal SSH is configured to
> behave either, right?
>

David, I have nothing whatsoever to do with the Eclipse implementation
of SSH, so what exactly is your point? All Java programmers are
fundamentally brain-damaged? Grow up, please.

jon.
-- 
homepage: http://www.zeta.org.au/~jon/
blog: http://orwelliantremors.blogspot.com/

Re: Core and Not-So Core

[David Woodhouse]

On Wed, 2005-05-11 at 03:15 +1000, Jon Seymour wrote:
> David, I have nothing whatsoever to do with the Eclipse implementation
> of SSH, so what exactly is your point? 

That all programmers of _any_ persuasion who want to reimplement the
world in their own language instead of using existing tools are
fundamentally brain-damaged. 

I'd say the same if the KDevelop people wanted to rewrite git and ssh in
C++, too. 

I don't think that's such a childish observation, surely? Piecing
together tools which do one thing, and do it well, is part of the UNIX
philosophy.

-- 
dwmw2

Re: Core and Not-So Core

[Jon Seymour]

On 5/11/05, David Woodhouse <dwmw2@infradead.org> wrote:
> On Wed, 2005-05-11 at 03:15 +1000, Jon Seymour wrote:
> > David, I have nothing whatsoever to do with the Eclipse implementation
> > of SSH, so what exactly is your point?
> 
> That all programmers of _any_ persuasion who want to reimplement the
> world in their own language instead of using existing tools are
> fundamentally brain-damaged.
> 
> I'd say the same if the KDevelop people wanted to rewrite git and ssh in
> C++, too.
> 
> I don't think that's such a childish observation, surely? Piecing
> together tools which do one thing, and do it well, is part of the UNIX
> philosophy.

This might surprise you, but everything in the world is not a unix
command line. Using tools designed for use on a unix command line in a
highly interactive environment like Eclipse would produce an
abominably clunky result.

Presumably you think this is the Unix way.

I have already explained all the pragmatic reasons for doing a GIT
implementation in Java but you are prepared to ignore all of those
reasons. You have ignored all these reasons rather than lift a finger
to compose a well-reasoned rebuttal.

Instead you cling to a dogmatic insistence that GIT respositories only
be accessed via the C toolsets.

David, you are being extremely childish and extremely dogmatic. Why bother?

jon
-- 
homepage: http://www.zeta.org.au/~jon/
blog: http://orwelliantremors.blogspot.com/

Re: Core and Not-So Core

[Christoph Hellwig]

On Wed, May 11, 2005 at 03:36:29AM +1000, Jon Seymour wrote:
> This might surprise you, but everything in the world is not a unix
> command line. Using tools designed for use on a unix command line in a
> highly interactive environment like Eclipse would produce an
> abominably clunky result.


The unix commandline is an highly interactive enviroment aswell,
and supports scripting in addition :)

> I have already explained all the pragmatic reasons for doing a GIT
> implementation in Java but you are prepared to ignore all of those
> reasons. You have ignored all these reasons rather than lift a finger
> to compose a well-reasoned rebuttal.

You tried to argue for re-inventing the wheel.  Fortunately you are
allowed to reinvent the wheel here (which isn't given anymore these
days).  Just don't expect any support from people who have been burnt
by that before.  And the Java world is re-inventing the wheel far to
often - I suspect that'll be cured when the community gets more mature
in a few years..

> Instead you cling to a dogmatic insistence that GIT respositories only
> be accessed via the C toolsets.

The C toolchain is a well-defined interface.  One that makes a lot
of sense.

Core and Not-So Core

[Jon Seymour]

On 5/11/05, Christoph Hellwig <hch@infradead.org> wrote:
> On Wed, May 11, 2005 at 03:36:29AM +1000, Jon Seymour wrote:
> > I have already explained all the pragmatic reasons for doing a GIT
> > implementation in Java but you are prepared to ignore all of those
> > reasons. You have ignored all these reasons rather than lift a finger
> > to compose a well-reasoned rebuttal.
>
> You tried to argue for re-inventing the wheel.  Fortunately you are
> allowed to reinvent the wheel here (which isn't given anymore these
> days).  Just don't expect any support from people who have been burnt
> by that before.  And the Java world is re-inventing the wheel far to
> often - I suspect that'll be cured when the community gets more mature
> in a few years..
>

I am _not_ re-inventing any wheel. Merely building a road through
different territory upon which the GIT wheel may roll more freely.

Don't get me wrong. I appreciate the UNIX mindset of building small
tools and composing them freely. But the UNIX way isn't the only way
to interact with the world and it is being hopelessly dogmatic to
insist otherwise.

jon.
--
homepage: http://www.zeta.org.au/~jon/
blog: http://orwelliantremors.blogspot.com/

Re: Core and Not-So Core

[Davide Libenzi]

On Wed, 11 May 2005, Jon Seymour wrote:

> On 5/11/05, Christoph Hellwig <hch@infradead.org> wrote:
>> On Wed, May 11, 2005 at 03:36:29AM +1000, Jon Seymour wrote:
>>> I have already explained all the pragmatic reasons for doing a GIT
>>> implementation in Java but you are prepared to ignore all of those
>>> reasons. You have ignored all these reasons rather than lift a finger
>>> to compose a well-reasoned rebuttal.
>>
>> You tried to argue for re-inventing the wheel.  Fortunately you are
>> allowed to reinvent the wheel here (which isn't given anymore these
>> days).  Just don't expect any support from people who have been burnt
>> by that before.  And the Java world is re-inventing the wheel far to
>> often - I suspect that'll be cured when the community gets more mature
>> in a few years..
>>
>
> I am _not_ re-inventing any wheel. Merely building a road through
> different territory upon which the GIT wheel may roll more freely.

Would you mind fixing your mailer before, so that your messages won't 
appear as new Subject: every time?


- Davide

Re: Core and Not-So Core

[Rik van Riel]

On Wed, 11 May 2005, Jon Seymour wrote:

> I did consider wrapping it - I really did. But after thinking about it 
> for a couple of weeks I eventually came to the conclusion it would be a 
> sub-optimal solution.

> So, if I want a stable foundation to build my stuff on, basing it on
> the output of the C tools would be a huge mistake.

Can Java use a library that's implemented in C, like Python
and Perl can?  If that is the case, the C implementation of
git simply needs the ability to be called as a library and
you can implement Java bindings for it.

That should also make it easier to create front-ends in
other languages...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: Core and Not-So Core

[Jon Seymour]

On 5/11/05, Rik van Riel <riel@redhat.com> wrote:
> On Wed, 11 May 2005, Jon Seymour wrote:
> 
> > I did consider wrapping it - I really did. But after thinking about it
> > for a couple of weeks I eventually came to the conclusion it would be a
> > sub-optimal solution.
> 
> > So, if I want a stable foundation to build my stuff on, basing it on
> > the output of the C tools would be a huge mistake.
> 
> Can Java use a library that's implemented in C, like Python
> and Perl can?  If that is the case, the C implementation of
> git simply needs the ability to be called as a library and
> you can implement Java bindings for it.
> 

It can in principle, yes. This is the so-called Java-Native-Interface (JNI).

I think in the longer term it would make sense to write a JNI layer
but the GIT source code is probably a little too unstable for that
now. What needs to happen first, I think, is that a solid and stable C
API (e.g. a libgit) be proposed and developed.

One outcome of the work I am doing (a long time away, probably will
never happen) might eventually be to propose an C API for GIT based,
in part, on the prototyping work I do in Java. However, I suspect its
Java-tainted heritage would mean instant death to it for purely
dogmatic reasons so I think it I'll leave it to others to propose a
C-API and then write a JNI wrapper for that when/if it ever happens.

> That should also make it easier to create front-ends in
> other languages...
> 

Agreed.

jon.

Re: Core and Not-So Core

[Petr Baudis]

Dear diary, on Wed, May 11, 2005 at 04:09:49AM CEST, I got a letter
where Jon Seymour <jon.seymour@gmail.com> told me that...
> On 5/11/05, Rik van Riel <riel@redhat.com> wrote:
> > On Wed, 11 May 2005, Jon Seymour wrote:
> > 
> > > I did consider wrapping it - I really did. But after thinking about it
> > > for a couple of weeks I eventually came to the conclusion it would be a
> > > sub-optimal solution.
> > 
> > > So, if I want a stable foundation to build my stuff on, basing it on
> > > the output of the C tools would be a huge mistake.
> > 
> > Can Java use a library that's implemented in C, like Python
> > and Perl can?  If that is the case, the C implementation of
> > git simply needs the ability to be called as a library and
> > you can implement Java bindings for it.
> > 
> 
> It can in principle, yes. This is the so-called Java-Native-Interface (JNI).
> 
> I think in the longer term it would make sense to write a JNI layer
> but the GIT source code is probably a little too unstable for that
> now. What needs to happen first, I think, is that a solid and stable C
> API (e.g. a libgit) be proposed and developed.

FWIW, this is actually happening now (see Brad Roberts' posts in nearby
thread).

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor

Re: Core and Not-So Core

[Daniel Barkalow]

On Wed, 11 May 2005, Jon Seymour wrote:

> I have been experimenting with pure-Java implementation of GIT
> concepts with a goal of eventually providing plugins to Eclipse to
> allow the Eclipse GUI to interact with GIT repositories.
> 
> One thing I noticed when doing this is that the present index/cache
> structure is rather arbitrary and the optimum index structure is
> determined by the structure of the tools that use a GIT repository
> rather than the structure of the GIT repository itself.

I'd guess that an Eclipse plugin wouldn't have a on-disk cache at all,
since it expects to keep the same address space through a whole day's
work.

> To give a concrete example: the cache currently contains most of the
> posix stat structure primarily to allow quick change detection. In the
> Java world, most of the posix stat structure is not directly
> accessible via the pure-Java file system abstractions. However, for
> most purposes detecting changes to files modification time and file
> size would be enough. Given this is the case, a Java-GIT client
> doesn't need to bother getting access to a posix stat structure and
> could therefore get away with a simpler  index structure, provided it
> doesn't need to interoperate with a 'C'-GIT client that shared the
> same workspace. A Java-GIT client might also choose to represent an
> index cache as a complex serialized Java object graph or (perhaps) an
> XML document.

The two clients should be able to interoperate trivially; the C version
would just treat the Java one as a text editor whose modifications happen
to match commits going into the same head; as soon as the C user does
either read-tree or update-cache, everything will be seen to be consistent
again.

> Currently the GIT stack is structured as follows:
> 
> cogito
> git-core 
> 
> I think it would be worthwhile if care was taken to draw a distinction
> between the repository and the cache aspects of the git core, perhaps
> even going to the extreme of moving all knowledge of the  cache into
> cogito itself. By clearly drawing this distinction, we will more
> easily enable the creation of different kind of tools sets atop the
> foundation of the GIT repository format.

I think this is nonsensical. The cache format is tied to the way in which
the repository accessing code is written, so a git-core separate from the
cache wouldn't have a useful set of code.

It might be worthwhile to produce a separate document describing the
repository format, such that it could be accessed by different code,
however. Of course, the index file wouldn't be part of this documentation,
for the same reason that the public git repositories don't include index
files.

	-Daniel
*This .sig left intentionally blank*

Re: Core and Not-So Core

[Jon Seymour]

>
> > I think it would be worthwhile if care was taken to draw a distinction
> > between the repository and the cache aspects of the git core, perhaps
> > even going to the extreme of moving all knowledge of the  cache into
> > cogito itself. By clearly drawing this distinction, we will more
> > easily enable the creation of different kind of tools sets atop the
> > foundation of the GIT repository format.
>
> I think this is nonsensical. The cache format is tied to the way in which
> the repository accessing code is written, so a git-core separate from the
> cache wouldn't have a useful set of code.
>

I guess I agree it is somewhat nonsensical if one considers the
current git toolset as a collection of programs - the exercise now
might simply reduce to classifying git tools as
index-using/non-index-using and nothing more. However, it might be
worth keeping in mind when/if the "libification" of git happens so
that there is a clean separation of layers in the API between the
repository API and index/cache/workspace API.

> It might be worthwhile to produce a separate document describing the
> repository format, such that it could be accessed by different code,
> however. Of course, the index file wouldn't be part of this documentation,
> for the same reason that the public git repositories don't include index
> files.
>

And thanks for your constructive criticism.

jon.
--
homepage: http://www.zeta.org.au/~jon/
blog: http://orwelliantremors.blogspot.com/

Re: Core and Not-So Core

[Petr Baudis]

Dear diary, on Wed, May 11, 2005 at 01:05:55AM CEST, I got a letter
where Jon Seymour <jon.seymour@gmail.com> told me that...
> >
> > > I think it would be worthwhile if care was taken to draw a distinction
> > > between the repository and the cache aspects of the git core, perhaps
> > > even going to the extreme of moving all knowledge of the  cache into
> > > cogito itself. By clearly drawing this distinction, we will more
> > > easily enable the creation of different kind of tools sets atop the
> > > foundation of the GIT repository format.
> >
> > I think this is nonsensical. The cache format is tied to the way in which
> > the repository accessing code is written, so a git-core separate from the
> > cache wouldn't have a useful set of code.
> >
> 
> I guess I agree it is somewhat nonsensical if one considers the
> current git toolset as a collection of programs - the exercise now
> might simply reduce to classifying git tools as
> index-using/non-index-using and nothing more. However, it might be
> worth keeping in mind when/if the "libification" of git happens so
> that there is a clean separation of layers in the API between the
> repository API and index/cache/workspace API.

In that case the repository API's input/output is data structure
equivalent to the index, and workspace API's input/output is the index
too. That is what it really is - and it is kept on the disk only since
the commands are invoked separately so it needs to keep the state around
somewhere.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor

Re: Core and Not-So Core

[Jon Seymour]

On 5/11/05, Petr Baudis <pasky@ucw.cz> wrote:
> Dear diary, on Wed, May 11, 2005 at 01:05:55AM CEST, I got a letter
> where Jon Seymour <jon.seymour@gmail.com> told me that...
> >
> > I guess I agree it is somewhat nonsensical if one considers the
> > current git toolset as a collection of programs - the exercise now
> > might simply reduce to classifying git tools as
> > index-using/non-index-using and nothing more. However, it might be
> > worth keeping in mind when/if the "libification" of git happens so
> > that there is a clean separation of layers in the API between the
> > repository API and index/cache/workspace API.
> 
> In that case the repository API's input/output is data structure
> equivalent to the index, and workspace API's input/output is the index
> too. That is what it really is - and it is kept on the disk only since
> the commands are invoked separately so it needs to keep the state around
> somewhere.
> 

Not sure I agree...

The repository API would contain functionality equivalent to cat-file,
ls-tree, most of fsck-cache, rev-list, rev-tree, diff-tree, most of
the transport code - things that don't involve use of the index.

The workspace API would contain read-tree, write-tree, commit-tree,
etc - things that do involve use of the the index.

jon.

Re: Core and Not-So Core

[Daniel Barkalow]

On Wed, 11 May 2005, Jon Seymour wrote:

> The repository API would contain functionality equivalent to cat-file,
> ls-tree, most of fsck-cache, rev-list, rev-tree, diff-tree, most of
> the transport code - things that don't involve use of the index.
> 
> The workspace API would contain read-tree, write-tree, commit-tree,
> etc - things that do involve use of the the index.

Unfortunately for this idea, you can't actually check files into or out of
the repository using the git tools without the index (in memory at least,
if not on disk). This is a bit like having a libc with all the system
calls except read and write. Sure, there are a number of programs that
would be fine that way, but it makes the API unintuitive, and most serious
programs need the extension anyway.

	-Daniel
*This .sig left intentionally blank*

Re: Core and Not-So Core

[Jon Seymour]

On 5/12/05, Daniel Barkalow <barkalow@iabervon.org> wrote:
> On Wed, 11 May 2005, Jon Seymour wrote:
>
> > The repository API would contain functionality equivalent to cat-file,
> > ls-tree, most of fsck-cache, rev-list, rev-tree, diff-tree, most of
> > the transport code - things that don't involve use of the index.
> >
> > The workspace API would contain read-tree, write-tree, commit-tree,
> > etc - things that do involve use of the the index.
>
> Unfortunately for this idea, you can't actually check files into or out of
> the repository using the git tools without the index (in memory at least,
> if not on disk). This is a bit like having a libc with all the system
> calls except read and write. Sure, there are a number of programs that
> would be fine that way, but it makes the API unintuitive, and most serious
> programs need the extension anyway.
>

Daniel,

Checking in/checking out is definitely a workspace API function that
involves an index (almost certainly) an on-disk index, but reading and
writing from and to the repository is purely a repository API
function.

There is no reason why the workspace API can't make use of
"lower-level" functions in the repository API. I guess I would argue
that the repository API really shouldn't need much, if any knowledge,
of the workspace API.

The repository API knows about the on-disk structure of repository
objects - it knows how to write writing compressed, SHA1 signed
objects, it also knows how to pack and unpack the various object types
from their on-disk representation into structures manipulated by the
workspace API.

On the otherhand the workspace API knows about managing index
(workspace) state. It knows how to determine whether the
pre-conditions for a commit have been reached. It knows how to read
the workspace structures and coordinate read and writes from and to
the repository.

I really don't think this makes the API unintuitive, but then I don't
really understand why you think this is the case.

jon.
--
homepage: http://www.zeta.org.au/~jon/
blog: http://orwelliantremors.blogspot.com

Re: Core and Not-So Core

[Petr Baudis]

Dear diary, on Tue, May 10, 2005 at 05:00:33PM CEST, I got a letter
where Jon Seymour <jon.seymour@gmail.com> told me that...
> One thing I noticed when doing this is that the present index/cache
> structure is rather arbitrary and the optimum index structure is
> determined by the structure of the tools that use a GIT repository
> rather than the structure of the GIT repository itself.

Yes. And that's how it should be - the directory cache is just that - a
_cache_. It does not hold any permanent information, merely serves to
record git tools' working state relative to the given working tree. So
unlike the objects database which has well-defined format and is
supposed to be "public", you should view the directory cache as internal
git tools' structure. If you want to mess with it too, either use the
proper level of abstraction and call the git tools, or don't mess with
it at all. And you need to care about it only if you want the git tools
working on the same tree properly too - so in that case use the git
tools too.

 From your arguments, it's not clear to me what really is the big
problem with the git tools. They are _designed_ for automatic use
instead of human interaction - you can perceive them just as methods
with funny (but actually friendly to your programs) calling convention.

Kind regards,

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor

Re: Core and Not-So Core

[Jon Seymour]

On 5/11/05, Petr Baudis <pasky@ucw.cz> wrote:
> Dear diary, on Tue, May 10, 2005 at 05:00:33PM CEST, I got a letter
> where Jon Seymour <jon.seymour@gmail.com> told me that...
> 
> Yes. And that's how it should be - the directory cache is just that - a
> _cache_. 

No argument there.

>  So unlike the objects database which has well-defined format and is
> supposed to be "public", you should view the directory cache as internal
> git tools' structure. If you want to mess with it too, either use the
> proper level of abstraction and call the git tools, or don't mess with
> it at all. And you need to care about it only if you want the git tools
> working on the same tree properly too - so in that case use the git
> tools too.

I agree in principle, though I'd like users to be able to easily
switch between the Eclipse and git tools view of the workspace if they
want to - who am I to say how a user should work? Eclipse does this
kind of thing quite well with CVS precisely because it shares the
workspace structures with the CVS command line tools rather than
"re-inventing" the wheel. Yes, separation of concerns has been lost
(two implementations of a CVS client), but the big win is that the
tools behave like the user wants them to behave.

The trickiest case here is when the user switches between toolsets
mid-merge. I guess what I can do is this:

user switch from eclipse -> to git-tools:
    1. blow away existing git tools index
    2. use git-read-tree to repeat the merge executed in eclipse (my
workspace will track parents)
    3. use git-update-cache --add/--remove to reflect merge actions
that have occurred since the workspace deviated from the HEAD.

Alternatively, I can just make Eclipse reflect cache changing actions
out onto the git-tools, via an exec of those tools, as and when they
occur.

Making use of the git tools index going the other way isn't so easy to
achieve because the git tools workspace as it stands doesn't track the
merges that have occurred (i.e. which parents were used to form the
current cache). However, that's not necessarily a big problem. I just
rebuild my "cache" from scratch based on the merges I know about and
treat every other difference from the HEAD as a user edit to the
workspace.
 
> 
> From your arguments, it's not clear to me what really is the big
> problem with the git tools. They are _designed_ for automatic use
> instead of human interaction - you can perceive them just as methods
> with funny (but actually friendly to your programs) calling convention.
> 

I am not really arguing that there is a big problem with the existing
git tools. However, what I am arguing is that the existing workspace
tools are just one way to manage the workspace (Eclipse might be
another, as an example) and it would be helpful to keep this in mind,
particularly when/if libification ever happens.

jon.

Re: Core and Not-So Core

[Peter Williams]

Jon Seymour wrote:
> On 5/11/05, Petr Baudis <pasky@ucw.cz> wrote:
> 
>>Dear diary, on Tue, May 10, 2005 at 05:00:33PM CEST, I got a letter
>>where Jon Seymour <jon.seymour@gmail.com> told me that...
>>
>>Yes. And that's how it should be - the directory cache is just that - a
>>_cache_. 
> 
> 
> No argument there.
> 
> 
>> So unlike the objects database which has well-defined format and is
>>supposed to be "public", you should view the directory cache as internal
>>git tools' structure. If you want to mess with it too, either use the
>>proper level of abstraction and call the git tools, or don't mess with
>>it at all. And you need to care about it only if you want the git tools
>>working on the same tree properly too - so in that case use the git
>>tools too.
> 
> 
> I agree in principle, though I'd like users to be able to easily
> switch between the Eclipse and git tools view of the workspace if they
> want to - who am I to say how a user should work? Eclipse does this
> kind of thing quite well with CVS precisely because it shares the
> workspace structures with the CVS command line tools rather than
> "re-inventing" the wheel.

I disagree here.  My experience is that Eclipse does not work well with 
command line use of CVS within one of its workspaces.  This is because 
Eclipse keeps too much "state" information internally instead of relying 
on CVS for all information about the state of the CVS playground.  This 
is one of the reasons that you can't attach Eclipse to an already 
checked out CVS playground.  The problem should be easily avoidable.

In spite of the above, Eclipse's SCM interface is the best that I've 
stumbled across and is the main reason that I use Eclipse.  I 
particularly like the ability to compare a file in the workspace with 
those in different versions, branches, etc.

So I'm looking forward to a git plug-in.

Peter
-- 
Peter Williams                                   pwil3058@bigpond.net.au

"Learning, n. The kind of ignorance distinguishing the studious."
  -- Ambrose Bierce

Re: Core and Not-So Core

[Nicolas Pitre]

On Wed, 11 May 2005, Jon Seymour wrote:

> On 5/11/05, Petr Baudis <pasky@ucw.cz> wrote:
> > So unlike the objects database which has well-defined format and is
> > supposed to be "public", you should view the directory cache as internal
> > git tools' structure. If you want to mess with it too, either use the
> > proper level of abstraction and call the git tools, or don't mess with
> > it at all. And you need to care about it only if you want the git tools
> > working on the same tree properly too - so in that case use the git
> > tools too.
> 
> I agree in principle, though I'd like users to be able to easily
> switch between the Eclipse and git tools view of the workspace if they
> want to - who am I to say how a user should work? Eclipse does this
> kind of thing quite well with CVS precisely because it shares the
> workspace structures with the CVS command line tools rather than
> "re-inventing" the wheel.

In this case you just need to manage to use the current index format 
somehow.  

Honestly I don't think Linus might be interested into changing the index 
format just to make a Java implementation happier.  So your best bet is 
probably to use it as is or ignore it entirely.


Nicolas

Re: Core and Not-So Core

[Jon Seymour]

On 5/11/05, Nicolas Pitre <nico@cam.org> wrote:
> On Wed, 11 May 2005, Jon Seymour wrote:
> > I agree in principle, though I'd like users to be able to easily
> > switch between the Eclipse and git tools view of the workspace if they
> > want to - who am I to say how a user should work? Eclipse does this
> > kind of thing quite well with CVS precisely because it shares the
> > workspace structures with the CVS command line tools rather than
> > "re-inventing" the wheel.
> 
> In this case you just need to manage to use the current index format
> somehow.
> 
> Honestly I don't think Linus might be interested into changing the index
> format just to make a Java implementation happier.  So your best bet is
> probably to use it as is or ignore it entirely.
> 

Agreed - I am not proposing a change to the index format, merely
highlighting that the index format is closely bound to the
workspace-oriented toolset and that the workspace-oriented toolset is
distinct from the repository-oriented toolset; different
workspace-oriented toolsets will have different requirements for their
index files and so their index files might quite rightly differ in
both form and substance.

That said, their is no harm making my set of tools interoperable with
the git tools, where that is possible so that the user can decide to
use which ever toolset fits the job immediately at hand. With this in
mind, a published specification of the index format would help, but I
am not going insist on that. I am certainly not going to ask for
changes to it! Life is too short :-)

jon.

Re: Core and Not-So Core

[Noel Grandin]

[-- Attachment #1: Type: text/plain, Size: 3510 bytes --]

Hi

Note that eclipse in particular has a fairly complicated repository 
provider interface.
The subversion plugin developers (the subclipse project) took quite a 
while to implement their stuff.
Basing your stuff off of their code would be a good idea.

Also, they worked in 2 stages - in the first stage, they created 
something called the JavaHL interface, behind which they talked to the 
subversion C libraries using JNI.
Then they created a pure-java implemenation of the subversion C 
libraries which also implemented the JavaHL interface, allowing them to 
compare and contrast the behaviour.

Regards,
   Noel Grandin


Jon Seymour wrote:

>I have been experimenting with pure-Java implementation of GIT
>concepts with a goal of eventually providing plugins to Eclipse to
>allow the Eclipse GUI to interact with GIT repositories.
>
>One thing I noticed when doing this is that the present index/cache
>structure is rather arbitrary and the optimum index structure is
>determined by the structure of the tools that use a GIT repository
>rather than the structure of the GIT repository itself.
>
>To give a concrete example: the cache currently contains most of the
>posix stat structure primarily to allow quick change detection. In the
>Java world, most of the posix stat structure is not directly
>accessible via the pure-Java file system abstractions. However, for
>most purposes detecting changes to files modification time and file
>size would be enough. Given this is the case, a Java-GIT client
>doesn't need to bother getting access to a posix stat structure and
>could therefore get away with a simpler  index structure, provided it
>doesn't need to interoperate with a 'C'-GIT client that shared the
>same workspace. A Java-GIT client might also choose to represent an
>index cache as a complex serialized Java object graph or (perhaps) an
>XML document.
>
>Another example: I can imagine a variant of the index file structure
>that recorded all the parents which have been merged into the cache
>and automatically include this information when performing the commit.
>
>The point is that many different index file structures are possible
>and will be determined in part by the tooling created in the porcelain
>layer - there really is no one true index file format as there is a
>one true repository format. Different tools can use different index
>file formats and still interoperate at the repository level because
>only the repository format needs to have a solid, unchanging
>definition.
>
>Currently the GIT stack is structured as follows:
>
>cogito
>git-core 
>
>I think it would be worthwhile if care was taken to draw a distinction
>between the repository and the cache aspects of the git core, perhaps
>even going to the extreme of moving all knowledge of the  cache into
>cogito itself. By clearly drawing this distinction, we will more
>easily enable the creation of different kind of tools sets atop the
>foundation of the GIT repository format.
>
>e.g., either:
>
>cogito
>git-cache
>git-respository
>
>or:
>
>cogito-tools
>cogito-cache
>git-repository
>
>Anyway, I offer this as food for thought - chew or flame away as appropriate!
>
>jon.
>  
>


NOTICE: Please note that this email, and the contents thereof, 
are subject to the standard Peralex email disclaimer, which may 
be found at: http://www.peralex.com/disclaimer.html

If you cannot access the disclaimer through the URL attached 
 and you wish to receive a copy thereof please send 
 an email to email@peralex.com

Re: Core and Not-So Core

[Jon Seymour]

On 5/11/05, Noel Grandin <noel@peralex.com> wrote:
> Hi
> 
> Note that eclipse in particular has a fairly complicated repository
> provider interface.
> The subversion plugin developers (the subclipse project) took quite a
> while to implement their stuff.
> Basing your stuff off of their code would be a good idea.
> 

Noel,

Thanks for the info. I'll certainly have a look at what the subclipse
folks did as I am sure it will be helpful to understand the strategy. 
I do think we are in a slightly different situation here as we don't
quite have a stable library interface to git yet - Brad Roberts work
notwithstanding.

The repository API in pure Java is almost a no-brainer since Linus has
done such a good job in keeping the repository specification simple
and unambiguous.

A Java workspace API can take advantage of the abstraction and GUI
facilities that Java and Eclipse afford so will naturally be different
in form to the existing command line tools for manipulating the git
index. Certainly, there will be similarities - aspects of the 3-way
merge, for example, but there will be differences too - workspace
change detection will be somewhat assisted by the change notification
framework in Eclipse and won't require as much manual intervention.

> Also, they worked in 2 stages - in the first stage, they created
> something called the JavaHL interface, behind which they talked to the
> subversion C libraries using JNI.
> Then they created a pure-java implemenation of the subversion C
> libraries which also implemented the JavaHL interface, allowing them to
> compare and contrast the behaviour.
> 

At this stage, my thoughts are to implement a listener pattern to keep
the git index up to date, and I may well implement that by calling out
the the C git-update-cache executable. This can run in a background
thread so it needn't be a huge drag on interactive user performance.

Anyway, thanks for your input.

jon.

Re: Core and Not-So Core

[Juliusz Chroboczek]

> To give a concrete example: the cache currently contains most of the
> posix stat structure primarily to allow quick change detection. In the
> Java world, most of the posix stat structure is not directly
> accessible via the pure-Java file system abstractions. However, for
> most purposes detecting changes to files modification time and file
> size would be enough.

I've got exactly this problem in Darcs-git; and I ignore all of the
cached data except the file size, mtime and sha1.  I don't currently
ever write to the cache.

> I think it would be worthwhile if care was taken to draw a distinction
> between the repository and the cache aspects of the git core, perhaps
> even going to the extreme of moving all knowledge of the  cache into
> cogito itself.

There's nothing that prevents you from ignoring the Git cache and
using your own cache instead.

                                        Juliusz