.git/info/refs

.git/info/refs

[H. Peter Anvin]

Would it be an incompatible change to add the commit date (and perhaps 
the author date) to .git/info/refs?  I believe that would make it 
possible to dramatically (orders of magnitude) speed up the generation 
of the gitweb index page, which is easily the most expensive gitweb page 
to generate.

	-=hpa

Re: .git/info/refs

[Jakub Narebski]

H. Peter Anvin wrote:

> Would it be an incompatible change to add the commit date (and perhaps 
> the author date) to .git/info/refs?  I believe that would make it 
> possible to dramatically (orders of magnitude) speed up the generation 
> of the gitweb index page, which is easily the most expensive gitweb page 
> to generate.

With new gitweb and new git it is not that expensive. It is now one call
to git-for-each-ref per repository.

Besides, we can't rely that .git/info/refs is up to date, or even exists.
It is for dumb protocols, not for gitweb.
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: .git/info/refs

[H. Peter Anvin]

Jakub Narebski wrote:
> H. Peter Anvin wrote:
> 
>> Would it be an incompatible change to add the commit date (and perhaps 
>> the author date) to .git/info/refs?  I believe that would make it 
>> possible to dramatically (orders of magnitude) speed up the generation 
>> of the gitweb index page, which is easily the most expensive gitweb page 
>> to generate.
> 
> With new gitweb and new git it is not that expensive. It is now one call
> to git-for-each-ref per repository.

That IS hugely expensive.  On kernel.org, that is 24175 calls to git.

> Besides, we can't rely that .git/info/refs is up to date, or even exists.
> It is for dumb protocols, not for gitweb.

Well, SOMETHING needs to be done for this page, since it can take 15 
minutes or more to generate.  Caching doesn't help one iota, since it's 
stale before being generated.

	-hpa

Re: .git/info/refs

[Johannes Schindelin]

Hi,

On Wed, 24 Jan 2007, H. Peter Anvin wrote:

> Jakub Narebski wrote:
> > H. Peter Anvin wrote:
> > 
> > > Would it be an incompatible change to add the commit date (and perhaps the
> > > author date) to .git/info/refs?  I believe that would make it possible to
> > > dramatically (orders of magnitude) speed up the generation of the gitweb
> > > index page, which is easily the most expensive gitweb page to generate.
> > 
> > With new gitweb and new git it is not that expensive. It is now one call
> > to git-for-each-ref per repository.
> 
> That IS hugely expensive.  On kernel.org, that is 24175 calls to git.
> 
> > Besides, we can't rely that .git/info/refs is up to date, or even exists.
> > It is for dumb protocols, not for gitweb.
> 
> Well, SOMETHING needs to be done for this page, since it can take 15 
> minutes or more to generate.  Caching doesn't help one iota, since it's 
> stale before being generated.

To me, it seems like all boils down to caching parsed data structures. 
I.e. parse the config, then serialize the parsed data to a file. Don't 
reparse the config unless it is 1 hour older than the config.

Likewise, run for-each-ref, and serialize the parsed data into a file. 
Don't rerun for-each-ref if that file is younger than 15 minutes.

Maybe the same for the first 200 commits of each branch.

(I made those times up, but you get the idea.)

Ciao,
Dscho

Re: .git/info/refs

[H. Peter Anvin]

Johannes Schindelin wrote:
> 
> To me, it seems like all boils down to caching parsed data structures. 
> I.e. parse the config, then serialize the parsed data to a file. Don't 
> reparse the config unless it is 1 hour older than the config.
> 
> Likewise, run for-each-ref, and serialize the parsed data into a file. 
> Don't rerun for-each-ref if that file is younger than 15 minutes.
> 
> Maybe the same for the first 200 commits of each branch.
> 
> (I made those times up, but you get the idea.)
> 

A much better idea is to have that data structure updated on repository 
updates, which is the whole point behind .git/info/refs.  On kernel.org, 
at least, if you don't keep .git/info/refs up to date you need to get 
your fingers whacked anyway, since it damages usability for one 
particular class of users.

	-hpa

Re: .git/info/refs

[Johannes Schindelin]

Hi,

On Wed, 24 Jan 2007, H. Peter Anvin wrote:

> Johannes Schindelin wrote:
> > 
> > To me, it seems like all boils down to caching parsed data structures. I.e.
> > parse the config, then serialize the parsed data to a file. Don't reparse
> > the config unless it is 1 hour older than the config.
> > 
> > Likewise, run for-each-ref, and serialize the parsed data into a file. Don't
> > rerun for-each-ref if that file is younger than 15 minutes.
> > 
> > Maybe the same for the first 200 commits of each branch.
> > 
> > (I made those times up, but you get the idea.)
> > 
> 
> A much better idea is to have that data structure updated on repository 
> updates, which is the whole point behind .git/info/refs.  On kernel.org, 
> at least, if you don't keep .git/info/refs up to date you need to get 
> your fingers whacked anyway, since it damages usability for one 
> particular class of users.

Granted, for some things this might work. However, I would not wreak havoc 
by changing the format of .git/info/refs, rather put the details you 
wanted into .git/info/refs-details.

However, for other things (like showing a certain number of commits), it 
_might_ make sense to cache them (e.g. when literally thousands of people 
look at the 100 last commits of linux-2.6.git), but not for others (e.g. 
the 100th last to the 200th last commit of git-tools.git).

Having said that, it should be relatively easy to store the (parsed, or at 
least easily parseable) 500 last commits of a branch into 
.git/info/commits-<branch>.

This would put the burden of publishing a branch higher, easening the 
overall load on the server.

Jakub?

Ciao,
Dscho

Re: .git/info/refs

[H. Peter Anvin]

Johannes Schindelin wrote:
> 
> Granted, for some things this might work. However, I would not wreak havoc 
> by changing the format of .git/info/refs, rather put the details you 
> wanted into .git/info/refs-details.
> 

It's not clear to me if it would be wrecking havoc.  After all, if a 
format can't be expanded *at all*, there is something wrong, and adding 
things to the end of a line is a common structured way of expansion. 
Hence the original query

> However, for other things (like showing a certain number of commits), it 
> _might_ make sense to cache them (e.g. when literally thousands of people 
> look at the 100 last commits of linux-2.6.git), but not for others (e.g. 
> the 100th last to the 200th last commit of git-tools.git).

Any query that's within a repository is fairly easily cachable 
post-generation.  The front page (and its RSS variant) is a bit of an 
exception, because it involves all repositories at once.

Doesn't mean we couldn't do better, but...

	-hpa

Re: .git/info/refs

[Johannes Schindelin]

Hi,

On Wed, 24 Jan 2007, H. Peter Anvin wrote:

> Johannes Schindelin wrote:
> > 
> > Granted, for some things this might work. However, I would not wreak 
> > havoc by changing the format of .git/info/refs, rather put the details 
> > you wanted into .git/info/refs-details.
> > 
> 
> It's not clear to me if it would be wrecking havoc.  After all, if a 
> format can't be expanded *at all*, there is something wrong, and adding 
> things to the end of a line is a common structured way of expansion. 
> Hence the original query

The idea of .git/info/refs is to enable dumb transports to fetch something 
akin to intelligently. They don't need that information, and frankly, I 
don't think they should need to understand it.

I also expect that they interpret everything after the sha1 as refname, 
what with our having become quite liberal with refnames (they can contain 
spaces, tabs, and even a small amount of special K). So I don't see a way 
to upgrade the file format.

But as should be clear by now, I'd prefer additional information -- that 
is of no interest to dumb transports anyway -- to be put in an own file.

That also opens the possibility of, say .git/info/perl/, which contains 
_only_ serialized perl objects! I imagine this could be a performance 
booster.

> > However, for other things (like showing a certain number of commits), 
> > it _might_ make sense to cache them (e.g. when literally thousands of 
> > people look at the 100 last commits of linux-2.6.git), but not for 
> > others (e.g. the 100th last to the 200th last commit of 
> > git-tools.git).
> 
> Any query that's within a repository is fairly easily cachable 
> post-generation.  The front page (and its RSS variant) is a bit of an 
> exception, because it involves all repositories at once.

... and here we have a problem, right? No single update hook can update 
the _whole_ information.

Ciao,
Dscho

Re: .git/info/refs

[H. Peter Anvin]

Johannes Schindelin wrote:
> Hi,
> 
> On Wed, 24 Jan 2007, H. Peter Anvin wrote:
> 
>> Johannes Schindelin wrote:
>>> Granted, for some things this might work. However, I would not wreak 
>>> havoc by changing the format of .git/info/refs, rather put the details 
>>> you wanted into .git/info/refs-details.
>>>
>> It's not clear to me if it would be wrecking havoc.  After all, if a 
>> format can't be expanded *at all*, there is something wrong, and adding 
>> things to the end of a line is a common structured way of expansion. 
>> Hence the original query
> 
> The idea of .git/info/refs is to enable dumb transports to fetch something 
> akin to intelligently. They don't need that information, and frankly, I 
> don't think they should need to understand it.

I don't think adding 10 digits to each line is going to be a sizable 
impact on anything.

> I also expect that they interpret everything after the sha1 as refname, 
> what with our having become quite liberal with refnames (they can contain 
> spaces, tabs, and even a small amount of special K). So I don't see a way 
> to upgrade the file format.

They can also contain newlines, probably, so escaping is obligatory anyway.

> But as should be clear by now, I'd prefer additional information -- that 
> is of no interest to dumb transports anyway -- to be put in an own file.

Yes, but the argument seems to be philosophical.

> That also opens the possibility of, say .git/info/perl/, which contains 
> _only_ serialized perl objects! I imagine this could be a performance 
> booster.

For certain things, I'm sure.

>>> However, for other things (like showing a certain number of commits), 
>>> it _might_ make sense to cache them (e.g. when literally thousands of 
>>> people look at the 100 last commits of linux-2.6.git), but not for 
>>> others (e.g. the 100th last to the 200th last commit of 
>>> git-tools.git).
>> Any query that's within a repository is fairly easily cachable 
>> post-generation.  The front page (and its RSS variant) is a bit of an 
>> exception, because it involves all repositories at once.
> 
> ... and here we have a problem, right? No single update hook can update 
> the _whole_ information.

I don't see a problem.

	-hpa

Re: .git/info/refs

[Jakub Narebski]

H. Peter Anvin wrote:

> Johannes Schindelin wrote:
>> 
>> Granted, for some things this might work. However, I would not wreak havoc 
>> by changing the format of .git/info/refs, rather put the details you 
>> wanted into .git/info/refs-details.
> 
> It's not clear to me if it would be wrecking havoc.  After all, if a 
> format can't be expanded *at all*, there is something wrong, and adding 
> things to the end of a line is a common structured way of expansion. 
> Hence the original query

I don't think it can be easily expanded. .git/info/refs is meant for
http-fetch, and it mimics git-ls-remote / git-peek-remote output.

BTW. putting the info of git-for-each-ref into .git/info/refs-details
would mean that instead of "24175 calls to git" one would need to
read 24175 files. Perhaps the whole info needed to generate projects
index page should be pre-generated on push (update), instead of per
project (per repository) .git/info/refs-details

>> However, for other things (like showing a certain number of commits), it 
>> _might_ make sense to cache them (e.g. when literally thousands of people 
>> look at the 100 last commits of linux-2.6.git), but not for others (e.g. 
>> the 100th last to the 200th last commit of git-tools.git).
> 
> Any query that's within a repository is fairly easily cachable 
> post-generation.  The front page (and its RSS variant) is a bit of an 
> exception, because it involves all repositories at once.

Actually "RSS", or to be more exact OPML variant of front page in its
current invocation is equivalent of project_index page, and it can be
generated once (well, once per adding / removing / renaming a repository).

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: .git/info/refs

[Johannes Schindelin]

Hi,

On Wed, 24 Jan 2007, Jakub Narebski wrote:

> H. Peter Anvin wrote:
> 
> > Johannes Schindelin wrote:
> >> 
> >> Granted, for some things this might work. However, I would not wreak havoc 
> >> by changing the format of .git/info/refs, rather put the details you 
> >> wanted into .git/info/refs-details.
> > 
> > It's not clear to me if it would be wrecking havoc.  After all, if a 
> > format can't be expanded *at all*, there is something wrong, and adding 
> > things to the end of a line is a common structured way of expansion. 
> > Hence the original query
> 
> I don't think it can be easily expanded. .git/info/refs is meant for
> http-fetch, and it mimics git-ls-remote / git-peek-remote output.

Exactly.

> BTW. putting the info of git-for-each-ref into .git/info/refs-details
> would mean that instead of "24175 calls to git" one would need to
> read 24175 files. Perhaps the whole info needed to generate projects
> index page should be pre-generated on push (update), instead of per
> project (per repository) .git/info/refs-details

You completely lost me there. A push (update) is done as a specific user, 
who should not be able to write to a _global_ file!

Nevertheless, "24175 calls to git" is sure as hell more expensive than 
"reading 24175 files".

Plus, if we integrate the functionality to write .git/info/refs-details 
into update-server-info, you can reduce that further: it is no longer 
per-branch, but per-repo.

Ciao,
Dscho

Re: .git/info/refs

[Jakub Narebski]

H. Peter Anvin wrote:

>> I also expect that they interpret everything after the sha1 as refname, 
>> what with our having become quite liberal with refnames (they can contain 
>> spaces, tabs, and even a small amount of special K). So I don't see a way 
>> to upgrade the file format.
> 
> They can also contain newlines, probably, so escaping is obligatory
> anyway. 

No, refnames can not contain newlines.

And .git/info/refs mimics git-ls-remote / git-peek-remote output,
and is meant for dumb transports.
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: .git/info/refs

[Jakub Narebski]

H. Peter Anvin wrote:
> Jakub Narebski wrote:

>> Besides, we can't rely that .git/info/refs is up to date, or even exists.
>> It is for dumb protocols, not for gitweb.
> 
> Well, SOMETHING needs to be done for this page, since it can take 15 
> minutes or more to generate.  Caching doesn't help one iota, since it's 
> stale before being generated.

The simple and fast solution would be to make post-update hook contain
the git-for-each-ref with parameters like in git_get_last_activity,
saving e.g. to .git/info/last-committer, and in gitweb read this file
if it exist, run git-for-each-ref otherwise (similar to what we used to
do with .git/info/refs and git-peek-remote in gitweb).

-- 
Jakub Narebski
Poland

Re: .git/info/refs

[hpa]

> H. Peter Anvin wrote:
>
>>> Besides, we can't rely that .git/info/refs is up to date, or even
>>> exists.
>>> It is for dumb protocols, not for gitweb.
>>
>> Well, SOMETHING needs to be done for this page, since it can take 15
>> minutes or more to generate.  Caching doesn't help one iota, since it's
>> stale before being generated.
>
> The simple and fast solution would be to make post-update hook contain
> the git-for-each-ref with parameters like in git_get_last_activity,
> saving e.g. to .git/info/last-committer, and in gitweb read this file
> if it exist, run git-for-each-ref otherwise (similar to what we used to
> do with .git/info/refs and git-peek-remote in gitweb).
>

Right, this is basically what I'm saying; the question is only whether or
not this fits into .git/info/refs or should be a separate file.

Either way, I think git-update-server-info should generate all these files.

    -hpa

Re: .git/info/refs

[hpa]

> H. Peter Anvin wrote:
>
>>> Besides, we can't rely that .git/info/refs is up to date, or even
>>> exists.
>>> It is for dumb protocols, not for gitweb.
>>
>> Well, SOMETHING needs to be done for this page, since it can take 15
>> minutes or more to generate.  Caching doesn't help one iota, since it's
>> stale before being generated.
>
> The simple and fast solution would be to make post-update hook contain
> the git-for-each-ref with parameters like in git_get_last_activity,
> saving e.g. to .git/info/last-committer, and in gitweb read this file
> if it exist, run git-for-each-ref otherwise (similar to what we used to
> do with .git/info/refs and git-peek-remote in gitweb).
>

Right, this is basically what I'm saying; the question is only whether or
not this fits into .git/info/refs or should be a separate file.

Either way, I think git-update-server-info should generate all these files.

    -hpa

Re: .git/info/refs

[Johannes Schindelin]

Hi,

On Wed, 24 Jan 2007, hpa@zytor.com wrote:

> > H. Peter Anvin wrote:
> >
> >>> Besides, we can't rely that .git/info/refs is up to date, or even
> >>> exists.
> >>> It is for dumb protocols, not for gitweb.
> >>
> >> Well, SOMETHING needs to be done for this page, since it can take 15
> >> minutes or more to generate.  Caching doesn't help one iota, since it's
> >> stale before being generated.
> >
> > The simple and fast solution would be to make post-update hook contain
> > the git-for-each-ref with parameters like in git_get_last_activity,
> > saving e.g. to .git/info/last-committer, and in gitweb read this file
> > if it exist, run git-for-each-ref otherwise (similar to what we used to
> > do with .git/info/refs and git-peek-remote in gitweb).
> >
> 
> Right, this is basically what I'm saying; the question is only whether or
> not this fits into .git/info/refs or should be a separate file.
> 
> Either way, I think git-update-server-info should generate all these files.

Well, no. At least not per default. What you want is _very_ special to 
gitweb. It is _only_ needed by gitweb. And .git/info/refs is for _dumb 
transports_, _not_ for gitweb.

That said, I think it makes sense _in your setup_ to trigger updating 
_another_ file for use in gitweb.

Remember, this is all very, very special for gitweb. So let's separate it 
cleanly from all which is not special for gitweb.

I hope I have made it clear why (at least IMHO) it would be wrong, wrong, 
wrong to change the format of .git/info/refs _only_ for gitweb, which it 
is not meant for to begin with.

So let's introduce another file in .git/info/ especially dedicated to 
gitweb.

Then we are free to introduce real cool performance hacks, like using 
Storable to store the parsed data structures (I was alluding to this in an 
earlier reply, as "serializing"). Then you just retrieve the file -- if it 
exists -- or call for-each-ref (like Jakub said).

By separating this gitweb-special thing cleanly, maybe into a hook, we can 
have a perl script which writes this file. We can write a simple hash, 
which may or may not contain keys, thus being of "extensible format".

By having this perl script, you can -- as root -- run it as the 
appropriate user for each repository where it does not exist yet.

Remains the problem: how do we _force_ this hook enabled site-wide, i.e. 
in _all_ repos?

But that is too easy: just edit the existing template, and then replace 
the update hooks in all repos (possibly verifying that the existing update 
hook indeed matches the old template).

So what problems remain with this approach?

Ciao,
Dscho

Re: .git/info/refs

[H. Peter Anvin]

Johannes Schindelin wrote:
>>
>> Either way, I think git-update-server-info should generate all these files.
> 
> Well, no. At least not per default. What you want is _very_ special to 
> gitweb. It is _only_ needed by gitweb. And .git/info/refs is for _dumb 
> transports_, _not_ for gitweb.
> 
> That said, I think it makes sense _in your setup_ to trigger updating 
> _another_ file for use in gitweb.
> 
> Remember, this is all very, very special for gitweb. So let's separate it 
> cleanly from all which is not special for gitweb.
> 
> I hope I have made it clear why (at least IMHO) it would be wrong, wrong, 
> wrong to change the format of .git/info/refs _only_ for gitweb, which it 
> is not meant for to begin with.

No, you keep using circular reasoning.

	-hpa

Re: .git/info/refs

[Johannes Schindelin]

Hi,

On Thu, 25 Jan 2007, H. Peter Anvin wrote:

> > I hope I have made it clear why (at least IMHO) it would be wrong, 
> > wrong, wrong to change the format of .git/info/refs _only_ for gitweb, 
> > which it is not meant for to begin with.
> 
> No, you keep using circular reasoning.

No. Once again, .git/info/refs is _not_ for gitweb. But I will stop 
arguing about that topic, because I don't have enough time for that.

Ciao,
Dscho

Re: .git/info/refs

[H. Peter Anvin]

Jakub Narebski wrote:
> 
> I don't think it can be easily expanded. .git/info/refs is meant for
> http-fetch, and it mimics git-ls-remote / git-peek-remote output.

For heaven's sake, in computer science we can *NEVER* use the same 
feature for *MORE THAN ONE THING*.  If it doesn't work format-wise 
that's fine, but "it's only supposed to be used by dumb transports" is 
ridiculous.

> BTW. putting the info of git-for-each-ref into .git/info/refs-details
> would mean that instead of "24175 calls to git" one would need to
> read 24175 files. Perhaps the whole info needed to generate projects
> index page should be pre-generated on push (update), instead of per
> project (per repository) .git/info/refs-details

No, it should be one file per repository, not one file per ref.  Why? 
Obviously we don't want 24175 files to be accessed.  However, a push can 
only affect files for which the repository owner has permission and 
which resides in the repository filespace, so it should stay inside that 
  space.

On kernel.org, this would reduce the load from 24175 calls to git to 
reading 250 files.  Although the latter is still expensive (and will 
probably need post-generation caching) the files should be small and 
cacheable by the kernel, and the resulting I/O load should be quite small.

Anyway, as far as git-update-server-index is concerned, I'm *very* 
concerned that there be a single command that updates all the cached 
information across the repository.  Telling everyone to update their 
hooks every time we want to add cached information is silly.  Right now, 
git-update-server-index is the command to update cached information, and 
for usability reasons there should be a single entry point.

	-hpa

Re: .git/info/refs

[Junio C Hamano]

"H. Peter Anvin" <hpa@zytor.com> writes:

> Jakub Narebski wrote:
>> H. Peter Anvin wrote:
>>
>>> Would it be an incompatible change to add the commit date (and
>>> perhaps the author date) to .git/info/refs?  I believe that would
>>> make it possible to dramatically (orders of magnitude) speed up the
>>> generation of the gitweb index page, which is easily the most
>>> expensive gitweb page to generate.
>>
>> With new gitweb and new git it is not that expensive. It is now one call
>> to git-for-each-ref per repository.
>
> That IS hugely expensive.  On kernel.org, that is 24175 calls to git.

Do you mean you have 24k _REPOSITORIES_ served by gitweb on
kernel.org?

Re: .git/info/refs

[H. Peter Anvin]

Junio C Hamano wrote:
>>>
>>>> Would it be an incompatible change to add the commit date (and
>>>> perhaps the author date) to .git/info/refs?  I believe that would
>>>> make it possible to dramatically (orders of magnitude) speed up the
>>>> generation of the gitweb index page, which is easily the most
>>>> expensive gitweb page to generate.
>>> With new gitweb and new git it is not that expensive. It is now one call
>>> to git-for-each-ref per repository.
>> That IS hugely expensive.  On kernel.org, that is 24175 calls to git.
> 
> Do you mean you have 24k _REPOSITORIES_ served by gitweb on
> kernel.org?
> 

No, we currently have 250 repositories with a total of 24175 refs.

	-hpa

Re: .git/info/refs

[Junio C Hamano]

"H. Peter Anvin" <hpa@zytor.com> writes:

>>>> With new gitweb and new git it is not that expensive. It is now one call
>>>> to git-for-each-ref per repository.
>>> That IS hugely expensive.  On kernel.org, that is 24175 calls to git.
>>
>> Do you mean you have 24k _REPOSITORIES_ served by gitweb on
>> kernel.org?
>
> No, we currently have 250 repositories with a total of 24175 refs.

Then that would mean 250 calls to git-for-each-ref, wouldn't it?

Re: .git/info/refs

[H. Peter Anvin]

Junio C Hamano wrote:
> "H. Peter Anvin" <hpa@zytor.com> writes:
> 
>>>>> With new gitweb and new git it is not that expensive. It is now one call
>>>>> to git-for-each-ref per repository.
>>>> That IS hugely expensive.  On kernel.org, that is 24175 calls to git.
>>> Do you mean you have 24k _REPOSITORIES_ served by gitweb on
>>> kernel.org?
>> No, we currently have 250 repositories with a total of 24175 refs.
> 
> Then that would mean 250 calls to git-for-each-ref, wouldn't it?
> 

Well, I think it was Johannes that said once for each ref.  But either 
which way, it's a totally unacceptable load with resulting unacceptable 
latency.

	-hpa

Re: .git/info/refs

[Johannes Schindelin]

Hi,

On Thu, 25 Jan 2007, H. Peter Anvin wrote:

> Junio C Hamano wrote:
>
> > Then that would mean 250 calls to git-for-each-ref, wouldn't it?
> 
> Well, I think it was Johannes that said once for each ref.  But either 
> which way, it's a totally unacceptable load with resulting unacceptable 
> latency.

No. I would never say that you have to run for-each-ref for each ref. 
That's plain stupid.

BTW I take some satisfaction in that you finally agreed (in another email) 
that some post-creation caching is necessary.

I would be even more satisfied if you finally agreed that it is a good 
practice to separate conceptually different things, and not continued ad 
infinitum (and ad nauseam) arguing that .git/info/refs should serve dumb 
transports, and gitweb, and eventually bring peace to everybody on this 
planet.

Ciao,
Dscho

Re: .git/info/refs

[Jakub Narebski]

H. Peter Anvin <hpa@zytor.com> wrote:
> Jakub Narebski wrote:
>> 
>> I don't think it can be easily expanded. .git/info/refs is meant for
>> http-fetch, and it mimics git-ls-remote / git-peek-remote output.
> 
> For heaven's sake, in computer science we can *NEVER* use the same 
> feature for *MORE THAN ONE THING*.  If it doesn't work format-wise 
> that's fine, but "it's only supposed to be used by dumb transports" is 
> ridiculous.

.git/info/refs is for dumb transports, so if we follow "do not use
the same feature for more than one thing" principle we should not
change its format for gitweb.

.git/info/refs is one of auxiliary info files to help dumb servers,
(servers that does not do on-the-fly pack generation), to help
clients discover what references server has.  The second auxiliary
info file is .git/objects/info/packs.  Both are generated by
git-update-server-info command, usually run from post-update hook.

Because .git/info/refs format is the same as git-ls-remote output
(AFAIK smart servers use git-ls-remote or git-peek-remote; dumb
servers use .git/info/refs) we used and can use it as ''cached''
"git ls-remote ." / "git peek-remote ." / "git show-ref --dereference"
output. For bare repositories where new data arrives only via
'update' (via push or fetch) and always trigger post-update hook,
and not for example via git-commit which does not invoke post-update
hook, the information in .git/info/refs is always fresh.

What I propose as quick solution is to add new (perhaps local)
git-update-gitweb-info command which is to be used in post-update
(and perhaps post-commit for non-bare repos) hook, and which results
we would use in gitweb.  See patch at the bottom.

>> BTW. putting the info of git-for-each-ref into .git/info/refs-details
>> would mean that instead of "24175 calls to git" one would need to
>> read 24175 files. Perhaps the whole info needed to generate projects
>> index page should be pre-generated on push (update), instead of per
>> project (per repository) .git/info/refs-details
> 
> No, it should be one file per repository, not one file per ref.  Why? 
> Obviously we don't want 24175 files to be accessed.  However, a push can 
> only affect files for which the repository owner has permission and 
> which resides in the repository filespace, so it should stay inside that 
> space.

Gitweb _newer_ did one call to git _per ref_, but always one call to git
_per repository_!  Old git always used HEAD ref to get "Last Change" info
and used one call to git-rev-list (if I remember correctly), new git
checks all refs to get "Last Change" info but uses _one_ call to 
git-for-each-ref.  Because we did not want to affect gitweb performance
badly we waited for changing "Last Change" to check all refs and not
only HEAD to have git-for-each-ref to use one call to git command for that.
Historically it was first use of git-for-each-ref in gitweb.

Sidenote: I planned to add new %feature to gitweb to allow to chose
if to use all refs for "Last Change" info, HEAD ref, or some given ref
(for example "master").  But that would perhaps wait for .git/config
parser in Perl.

> On kernel.org, this would reduce the load from 24175 calls to git to 
> reading 250 files.  Although the latter is still expensive (and will 
> probably need post-generation caching) the files should be small and 
> cacheable by the kernel, and the resulting I/O load should be quite small.

Oh, so there are around 250 projects, and around 24175 references
together in those projects on kernel.org?  I thought it were 24175
_projects_ (repositories)...

Currently, it is 250 calls to git, reading 24175 files (unless refs
are packed, then it would be reading 250 files) to get refs (heads)
info, and reading around 2*250 files (packs + index) to get last
change info.  Not "24175 calls to git".

> Anyway, as far as git-update-server-info is concerned, I'm *very* 
> concerned that there be a single command that updates all the cached 
> information across the repository.  Telling everyone to update their 
> hooks every time we want to add cached information is silly.  Right now, 
> git-update-server-info is the command to update cached information, and 
> for usability reasons there should be a single entry point.

git-update-server-info is to "update auxiliary info file to help dumb
servers". I propose to use (new) git-update-gitweb-info to help gitweb.
One command for one feature.  This would mean unfortunately adding
"exec git-update-gitweb-info" line (if it does not exist) to existing
projects post-update hooks; for new projects it would be I think enough
to modify post-update template (templates/hooks--post-update or
/usr/share/git-core/templates/hooks/post-update).


Below the patches of how it can be done.  Does not include corrections
to Makefile to install git-update-gitweb-info.  NOT TESTED!

BTW final version of git-update-gitweb-info probably should be a built-in
command, like git-update-server-info, not a script.


diff --git a/git-update-gitweb-info.sh b/git-update-gitweb-info.sh
new file mode 100755
index 0000000..5bb44df
--- /dev/null
+++ b/git-update-gitweb-info.sh
@@ -0,0 +1,7 @@
+#!/bin/sh
+
+. git-sh-setup
+test -w "$GIT_DIR/info/last-changed" &&
+git-for-each-ref \
+	--format='%(committer)' --sort=-committerdate --count=1 refs/heads \
+	> "$GIT_DIR/info/last-changed"
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 88af2e6..e7874a6 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -1150,12 +1150,16 @@ sub git_get_last_activity {
 	my ($path) = @_;
 	my $fd;
 
-	$git_dir = "$projectroot/$path";
-	open($fd, "-|", git_cmd(), 'for-each-ref',
-	     '--format=%(committer)',
-	     '--sort=-committerdate',
-	     '--count=1',
-	     'refs/heads') or return;
+	if (-r "$projectroot/$path/info/last-changed") {
+		open $fd, "$projectroot/$path/info/last-changed";
+	} else {
+		$git_dir = "$projectroot/$path";
+		open($fd, "-|", git_cmd(), 'for-each-ref',
+		     '--format=%(committer)',
+		     '--sort=-committerdate',
+		     '--count=1',
+		     'refs/heads') or return;
+	}
 	my $most_recent = <$fd>;
 	close $fd or return;
 	if ($most_recent =~ / (\d+) [-+][01]\d\d\d$/) {
diff --git a/templates/hooks--post-update b/templates/hooks--post-update
old mode 100644
new mode 100755
index bcba893..b119224
--- a/templates/hooks--post-update
+++ b/templates/hooks--post-update
@@ -6,3 +6,4 @@
 # To enable this hook, make this file executable by "chmod +x post-update".
 
 exec git-update-server-info
+exec git-update-gitweb-info

-- 
Jakub Narebski
Poland

Re: .git/info/refs

[Junio C Hamano]

"H. Peter Anvin" <hpa@zytor.com> writes:

> For heaven's sake, in computer science we can *NEVER* use the same
> feature for *MORE THAN ONE THING*.  If it doesn't work format-wise
> that's fine, but "it's only supposed to be used by dumb transports" is
> ridiculous.

Hmmmm... I am lost here....

> Right
> now, git-update-server-index is the command to update cached
> information, and for usability reasons there should be a single entry
> point.

Modulo s/-index/-info/, I agree that would be a very sensible
position, as long as the cost to generate additional cached
information necessary to help gitweb is reasonably small, I am
not opposed to have it generate another file [*1*].


[*1*]

I've been looking for backward-compatible holes in ls-remote and
its users, hoping we somehow could shoehorn this information in
info/refs, as I do not think its file format is sacred, nor the
file is there _only_ to help dumb transports.  As long as the
published way to access that information stays consistent, the
underlying file format is a fair game.  However, I do not think
the ls-remote command implementations in the wild has such a
hole I can exploit.

Re: .git/info/refs

[H. Peter Anvin]

Junio C Hamano wrote:
> "H. Peter Anvin" <hpa@zytor.com> writes:
> 
>> For heaven's sake, in computer science we can *NEVER* use the same
>> feature for *MORE THAN ONE THING*.  If it doesn't work format-wise
>> that's fine, but "it's only supposed to be used by dumb transports" is
>> ridiculous.
> 
> Hmmmm... I am lost here....
> 

Jakub and Johannes seems to have been arguing that "info/refs is for 
dumb transports, therefore it cannot be used for any other purpose."  I 
find this argument utterly bizarre, since in general, in computer 
science, you try to be multipurpose whenever practical.

	-hpa

Re: .git/info/refs

[Jakub Narebski]

H. Peter Anvin wrote:
> Junio C Hamano wrote:
>> "H. Peter Anvin" <hpa@zytor.com> writes:
>> 
>>> For heaven's sake, in computer science we can *NEVER* use the same
>>> feature for *MORE THAN ONE THING*.  If it doesn't work format-wise
>>> that's fine, but "it's only supposed to be used by dumb transports" is
>>> ridiculous.

Please, for the future, mark irony if it might be mistaken...
 
>> Hmmmm... I am lost here....
>
> Jakub and Johannes seems to have been arguing that "info/refs is for 
> dumb transports, therefore it cannot be used for any other purpose."  I 
> find this argument utterly bizarre, since in general, in computer 
> science, you try to be multipurpose whenever practical.

First, changing info/refs format _might_ break fetch related scripts,
which rely on git-peek-remote / git-ls-remote / info/refs format.

Second, it is a bit impractical because info/refs contain (and must
contain) also _tags_ information (which is not needed for gitweb
"Last Change" field in projects list) and referenced object for
those tags. Tags need not to point to commits, nor dereference
to commits: for example in git.git tags v1.0rc1 to v1.0rc6 points
to other tags, and junio-gpg-pub point to out-of-tree blob (which
does not have any "commit time" associated). So what to write there
in the "commit time" field? What to write in "commit time" for tags?
-- 
Jakub Narebski
Poland

Re: .git/info/refs

[Johannes Schindelin]

Hi,

On Fri, 26 Jan 2007, H. Peter Anvin wrote:

> Junio C Hamano wrote:
> > "H. Peter Anvin" <hpa@zytor.com> writes:
> > 
> > > For heaven's sake, in computer science we can *NEVER* use the same
> > > feature for *MORE THAN ONE THING*.  If it doesn't work format-wise
> > > that's fine, but "it's only supposed to be used by dumb transports" is
> > > ridiculous.
> > 
> > Hmmmm... I am lost here....
> > 
> 
> Jakub and Johannes seems to have been arguing that "info/refs is for dumb
> transports, therefore it cannot be used for any other purpose."  I find this
> argument utterly bizarre, since in general, in computer science, you try to be
> multipurpose whenever practical.

You keep on harping on that issue. I really get tired of it.

You seem to propose that we should stuff things into .git/info/refs, just 
because it is already there.

You seem to suggest that computer science is all about breaking things, to 
muddle waters by mixing things which are clearly different kinds of 
kettle, to "just" add a small thing here and there, all under the guise of 
multi-purposity or whatever.

You seem to reason that practicality is more important than good style.

You know, this reasoning brought to us that big crap sh*tpile called 
Windows. They also thought: it's not a big deal, let's introduce just a 
little thing here and there, and a direct call from this component to that 
component cannot hurt, cannit?

I am really, really getting tired of that kind of reasoning.

If you don't see how UNELEGANT it is to force dumb transports to download 
things MEANT FOR GITWEB, and how much NICER it would be to have a file 
WHICH IS ONLY MEANT FOR GITWEB TO BEGIN WITH, and which is easily 
EXTENSIBLE, since we don't have to CARE about DUMB TRANSPORTS, because 
gitweb data is a PURELY LOCAL thing, while dumb transports are NOT, I will 
just start to ignore you.

Ciao,
Dscho

Re: .git/info/refs

[H. Peter Anvin]

Johannes Schindelin wrote:
[... stuff ...]

I really could care less, as long as a single invocation is used to 
update the cache information.

I disagree with your aesthetic argument, but it doesn't matter much to 
me either way.

	-hpa

Re: .git/info/refs

[H. Peter Anvin]

By the way, let me be the first to apologize for the emotional 
escalation.  What matters to me is that the information is cached and 
updated by a common cached information entry point (the existing 
git-update-server-index would be preferred, obviously), not where the 
information ends up.

	-hpa

Re: .git/info/refs

[H. Peter Anvin]

Johannes Schindelin wrote:
> No. I would never say that you have to run for-each-ref for each ref. 
> That's plain stupid.

I went back and looked at the thread, and I had indeed misread the 
original message, which was from Jakub, not you.  I think I got in the 
"this is surreal" mode as a result of that (invoking for-each-ref 250 
times is bad enough, obviously.)

> BTW I take some satisfaction in that you finally agreed (in another email) 
> that some post-creation caching is necessary.

I don't believe I have ever disputed that (in fact, I have pushed very 
hard for gitweb to do post-creation caching.)

> I would be even more satisfied if you finally agreed that it is a good 
> practice to separate conceptually different things, and not continued ad 
> infinitum (and ad nauseam) arguing that .git/info/refs should serve dumb 
> transports, and gitweb, and eventually bring peace to everybody on this 
> planet.

I've already said I think it's an aesthetic argument, but I don't really 
care either way, as long as there is only one hook that updates all the 
caches.  I don't want the user to have to juggle an arbitrary and 
increasing number of hooks.

Fair?

	-hpa

Re: .git/info/refs

[Santi Béjar]

On 1/27/07, H. Peter Anvin <hpa@zytor.com> wrote:
> Johannes Schindelin wrote:
> > No. I would never say that you have to run for-each-ref for each ref.
> > That's plain stupid.
>
> I went back and looked at the thread, and I had indeed misread the
> original message, which was from Jakub, not you.  I think I got in the
> "this is surreal" mode as a result of that (invoking for-each-ref 250
> times is bad enough, obviously.)
>

Normally I'm not interested in the "Last Change" column, I just want
to go to the project summary page, and normally I'm not interested in
the last 16 tags (the last three are just enough). For me they should
be show only when explicitly asked.

Santi

Re: .git/info/refs

[Johannes Schindelin]

Hi,

I just had another idea: why not generate the content of the "cover page" 
in a cron job, every minute or so, and save it into a static index.html? 
This should take quite a load from the server, since not even Perl has to 
be started to serve that page.

Ciao,
Dscho

Re: .git/info/refs

[H. Peter Anvin]

Johannes Schindelin wrote:
> Hi,
> 
> I just had another idea: why not generate the content of the "cover page" 
> in a cron job, every minute or so, and save it into a static index.html? 
> This should take quite a load from the server, since not even Perl has to 
> be started to serve that page.
> 

Ehm... because it often takes longer than that to generate the page?

We can pre-generate the page before the first hit, but that's not a 
replacement for update-time caching.

	-hpa

Re: .git/info/refs

[Johannes Schindelin]

Hi,

On Thu, 1 Feb 2007, H. Peter Anvin wrote:

> Johannes Schindelin wrote:
> > Hi,
> > 
> > I just had another idea: why not generate the content of the "cover page" in
> > a cron job, every minute or so, and save it into a static index.html? This
> > should take quite a load from the server, since not even Perl has to be
> > started to serve that page.
> > 
> 
> Ehm... because it often takes longer than that to generate the page?

Sorry, I should have been clearer. Plan:

1. echo "Generating" > /htdocs/git/index.html
2. edit crontab to do this every minute:
2.1 gitweb is called directly_, to generate /htdocs/git/index.html.new
2.2 /htdocs/git/index.html.new is _moved_ into /htdocs/git/index.html, 
    overwriting the existing one.

Yes, there could be two instances of this task concurrently. No, it does 
not matter.

> We can pre-generate the page before the first hit, but that's not a 
> replacement for update-time caching.

It was only meant as a quick fix for the horrible workload.

Just a thought, feel free to ignore me,
Dscho

Re: .git/info/refs

[H. Peter Anvin]

Johannes Schindelin wrote:
> 
> Sorry, I should have been clearer. Plan:
> 
> 1. echo "Generating" > /htdocs/git/index.html
> 2. edit crontab to do this every minute:
> 2.1 gitweb is called directly_, to generate /htdocs/git/index.html.new
> 2.2 /htdocs/git/index.html.new is _moved_ into /htdocs/git/index.html, 
>     overwriting the existing one.
> 
> Yes, there could be two instances of this task concurrently. No, it does 
> not matter.
> 

Yes, it does matter, because it drives the load up further.  If you 
start having this going on in overlapping instances, then you're soon on 
the downhill slope of a cascading failure.

>> We can pre-generate the page before the first hit, but that's not a 
>> replacement for update-time caching.
> 
> It was only meant as a quick fix for the horrible workload.

And we have already experimented with it.  It unfortunately doesn't help 
much, it only makes matters worse.

	-hpa

Re: .git/info/refs

[Matthias Lederhofer]

H. Peter Anvin <hpa@zytor.com> wrote:
> Yes, it does matter, because it drives the load up further.  If you 
> start having this going on in overlapping instances, then you're soon on 
> the downhill slope of a cascading failure.
Add some other locking mechanism.

> And we have already experimented with it.  It unfortunately doesn't help 
> much, it only makes matters worse.
The gitweb overview page has less than one hit per minute?  Otherwise
this should help.

Re: .git/info/refs

[H. Peter Anvin]

Matthias Lederhofer wrote:
> H. Peter Anvin <hpa@zytor.com> wrote:
>> Yes, it does matter, because it drives the load up further.  If you 
>> start having this going on in overlapping instances, then you're soon on 
>> the downhill slope of a cascading failure.
> Add some other locking mechanism.
> 
>> And we have already experimented with it.  It unfortunately doesn't help 
>> much, it only makes matters worse.
> The gitweb overview page has less than one hit per minute?  Otherwise
> this should help.

We already cache it with a forced duration of some 15 minutes.  The end 
result is exactly the same.

	-hpa