Efficient retrieval of commit log info

Efficient retrieval of commit log info

[Eirik Bjørsnøs]

Hi,

I'm developing a piece of software that grabs logs from various types
of SCMs and presents (hopefully) useful information about the history.

My current approach with Git is do a "git clone --n" and then parse
the output of "git log". To check for updates I do a "git pull"
followed by a new "git log".

This approach works fine, but cloning the whole repository just to get
the change log seems like a somewhat inefficient use of bandwidth and
storage.

What I would like to do is to fetch just the change log information
from the remote repository.

(Using the "CVS done right" tool I can do this with: "svn log --xml -v
-r<last, HEAD> http://svn.example.com/")

I haven't found a way to do this using the Git command line tools.

Is it at all possible using the git network protocols to fetch just
the commit log info, without transferring the contents?

I'd be happy to hear from anyone who has some insight into this.

Thanks,

Eirik.

Re: Efficient retrieval of commit log info

[Johannes Schindelin]

Hi,

On Wed, 12 Dec 2007, Eirik Bj?rsn?s wrote:

> I'm developing a piece of software that grabs logs from various types of 
> SCMs and presents (hopefully) useful information about the history.
> 
> My current approach with Git is do a "git clone --n" and then parse the 
> output of "git log". To check for updates I do a "git pull" followed by 
> a new "git log".
> 
> This approach works fine, but cloning the whole repository just to get 
> the change log seems like a somewhat inefficient use of bandwidth and 
> storage.
> 
> What I would like to do is to fetch just the change log information
> from the remote repository.
> 
> (Using the "CVS done right" tool I can do this with: "svn log --xml -v
> -r<last, HEAD> http://svn.example.com/")
> 
> I haven't found a way to do this using the Git command line tools.

It is not possible to get just the metadata.  Remember, svn can do it only 
since the repository is purely remote.  And git is a SCM (source code 
management system), not a CMV (commit metadata viewer).

You might be able to cobble up something that works accessing gitweb, but 
it might be even more inefficient.

Ciao,
Dscho

Re: Efficient retrieval of commit log info

[Nicolas Pitre]

On Wed, 12 Dec 2007, Johannes Schindelin wrote:

> Hi,
> 
> On Wed, 12 Dec 2007, Eirik Bj?rsn?s wrote:
> 
> > I'm developing a piece of software that grabs logs from various types of 
> > SCMs and presents (hopefully) useful information about the history.
> > 
> > My current approach with Git is do a "git clone --n" and then parse the 
> > output of "git log". To check for updates I do a "git pull" followed by 
> > a new "git log".
> > 
> > This approach works fine, but cloning the whole repository just to get 
> > the change log seems like a somewhat inefficient use of bandwidth and 
> > storage.
> > 
> > What I would like to do is to fetch just the change log information
> > from the remote repository.
> > 
> > (Using the "CVS done right" tool I can do this with: "svn log --xml -v
> > -r<last, HEAD> http://svn.example.com/")
> > 
> > I haven't found a way to do this using the Git command line tools.
> 
> It is not possible to get just the metadata.  Remember, svn can do it only 
> since the repository is purely remote.  And git is a SCM (source code 
> management system), not a CMV (commit metadata viewer).
> 
> You might be able to cobble up something that works accessing gitweb, but 
> it might be even more inefficient.

On the other hand, this is pretty trivial to extend the protocol so only 
commit objects are transferred.  Using 'git log' on the client side 
would work, as long as you don't use any of the content walking options.


Nicolas

Re: Efficient retrieval of commit log info

[Eirik Bjørsnøs]

> On the other hand, this is pretty trivial to extend the protocol so only
> commit objects are transferred.  Using 'git log' on the client side
> would work, as long as you don't use any of the content walking options.

Nicolas,

So what you're suggesting is that:

1) git-clone could be changed such that the user could ask not to
fetch content, only commit objects.
2) git-daemon (or the protocol layer, again I'm not familiar with the
implementation) could be changed such that it only sends commit
objects on request.
3) Changes to git-log is not necessary.

Anybody have a clue how much work this would require and if this is
something that would be generally useful for the community?

Eirik.

Re: Efficient retrieval of commit log info

[Santi Béjar]

On Dec 12, 2007 3:55 PM, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
> On Wed, 12 Dec 2007, Eirik Bj?rsn?s wrote:
>
[...]
> > What I would like to do is to fetch just the change log information
> > from the remote repository.
> >
[ ... ]
>
> It is not possible to get just the metadata.  Remember, svn can do it only
> since the repository is purely remote.  And git is a SCM (source code
> management system), not a CMV (commit metadata viewer).
>
> You might be able to cobble up something that works accessing gitweb, but
> it might be even more inefficient.

You could see how it is done in the git-browser:

http://repo.or.cz/w/git-browser.git

It is a gitk-like browser written in javascript. But I suspect that it
should be enabled in the web server.

Santi

Re: Efficient retrieval of commit log info

[Jon Smirl]

On 12/12/07, Eirik Bjørsnøs <eirbjo@gmail.com> wrote:
> > On the other hand, this is pretty trivial to extend the protocol so only
> > commit objects are transferred.  Using 'git log' on the client side
> > would work, as long as you don't use any of the content walking options.
>
> Nicolas,
>
> So what you're suggesting is that:

Write a script the runs the command locally on the remote sever and
emails you the info. Or ssh to the remote server and run the commands
locally.

>
> 1) git-clone could be changed such that the user could ask not to
> fetch content, only commit objects.
> 2) git-daemon (or the protocol layer, again I'm not familiar with the
> implementation) could be changed such that it only sends commit
> objects on request.
> 3) Changes to git-log is not necessary.
>
> Anybody have a clue how much work this would require and if this is
> something that would be generally useful for the community?
>
> Eirik.
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 
Jon Smirl
jonsmirl@gmail.com

Re: Efficient retrieval of commit log info

[Eirik Bjørsnøs]

> Write a script the runs the command locally on the remote sever and
> emails you the info. Or ssh to the remote server and run the commands
> locally.

Hi Jon,

Unfortunately that won't work for my use case since I'm fetching
change metadata from remote repositories that I don't have any kind of
control over.

Eirik.

Re: Efficient retrieval of commit log info

[Linus Torvalds]

On Wed, 12 Dec 2007, Eirik Bj?rsn?s wrote:
> 
> Is it at all possible using the git network protocols to fetch just
> the commit log info, without transferring the contents?

>From a _protocol_ angle that's trivial (the git network protocol doesn't 
really care what it transfers, it just transfers objects), but no, we 
don't actually expose any capability like that. If you were to just want 
this privately, the quick hack is to just disable traversal of trees in 
builtin-pack-objects on the server side, but it sounds like you want to do 
this in general (not just with git and not just with a server you 
control), and if so, you're kind of screwed.

You could hack around it (very inefficiently) by parsing gitweb output, of 
course, but I bet that you'd probably get better performance just cloning 
the whole thing (which allows streaming, rather than trying to click 
through commits on gitweb)

		Linus