[RFC] Reporting index properties

[RFC] Reporting index properties

[Alex Vandiver]

Heya,

As part of gathering some automated performance statistics of git, it
would be useful to be able to extract some vital properties of the
index.  While `git update-index` has ways to _set_ the index version,
and enable/disable various extensions, AFAICT there is no method by
which one can ask for reporting about the current state of them --
short of re-implementing all of the index-parsing logic external to
Git, which is not terribly appealing.

I hesitate to propose adding a flag to `git update-index` which reads
but does no updating, as that seems to make a misnomer of the
command.  But this also doesn't seem worthy of a new top-level command.

Do folks have feelings about surfacing this information, and where such
logic should live?

 - Alex

Re: [RFC] Reporting index properties

[Junio C Hamano]

Alex Vandiver <alexmv@dropbox.com> writes:

> Do folks have feelings about surfacing this information, and where such
> logic should live?

Probably one of the t/helper/test-dump-*.c programs, if we already
do not have one.

Re: [RFC] Reporting index properties

[Alex Vandiver]

On Fri, 6 Oct 2017, Junio C Hamano wrote:
> > Do folks have feelings about surfacing this information, and where such
> > logic should live?
> 
> Probably one of the t/helper/test-dump-*.c programs, if we already
> do not have one.

The goal would be to be able to extract this information from
repositories, with a standard git install.  That directory only
contains developer tools, which aren't part of the install, no?

 - Alex

Re: [RFC] Reporting index properties

[Junio C Hamano]

Alex Vandiver <alexmv@dropbox.com> writes:

> On Fri, 6 Oct 2017, Junio C Hamano wrote:
>> > Do folks have feelings about surfacing this information, and where such
>> > logic should live?
>> 
>> Probably one of the t/helper/test-dump-*.c programs, if we already
>> do not have one.
>
> The goal would be to be able to extract this information from
> repositories, with a standard git install.  That directory only
> contains developer tools, which aren't part of the install, no?

Yes, and I am saying that such logic should not live in standard
install outside developer tools ;-)

Re: [RFC] Reporting index properties

[Alex Vandiver]

On Fri, 6 Oct 2017, Junio C Hamano wrote:
> Yes, and I am saying that such logic should not live in standard
> install outside developer tools ;-)

Fair enough.  It seems a little odd to me that git provides logic for
_altering_ those bits, but not to report on the state that can be so
altered.

 - Alex

Re: [RFC] Reporting index properties

[Jonathan Nieder]

Alex Vandiver wrote:

> As part of gathering some automated performance statistics of git, it
> would be useful to be able to extract some vital properties of the
> index.  While `git update-index` has ways to _set_ the index version,
> and enable/disable various extensions, AFAICT there is no method by
> which one can ask for reporting about the current state of them --
> short of re-implementing all of the index-parsing logic external to
> Git, which is not terribly appealing.

Aside: git-update-index(1) says

	Version 4 is relatively young (first released in 1.8.0 in
	October 2012).

My first reaction is to wonder if it is time to introduce a config
option to use index format version 4 by default, since after 5 years
it is not relatively young any more.

My second reaction is to notice that the index.version configuration
already exists.  Probably git-update-index(1) ought to point to it.

JGit still doesn't support index format 4, so 4 is still not a good
default for `index.version` for a user that hasn't explicitly
configured it, but the moment JGit gains that support I think it will
be. :)

> I hesitate to propose adding a flag to `git update-index` which reads
> but does no updating, as that seems to make a misnomer of the
> command.  But this also doesn't seem worthy of a new top-level command.

The existing scripting command that inspects the index is
"git ls-files".

It doesn't go into this kind of low-level detail about the index
format, though.  In the same spirit, I don't think we have an existing
command in this spirit for analyzing packfiles, either.

So I agree with Junio that this would be a good debugging aid to put
in t/helper/ for now, and then once we've had experience with it we'd
end up in a better position to come up with a stable, public-facing
interface, if one is needed.

Thanks and hope that helps,
Jonathan

Re: [RFC] Reporting index properties

[Junio C Hamano]

Jonathan Nieder <jrnieder@gmail.com> writes:

> Aside: git-update-index(1) says
>
> 	Version 4 is relatively young (first released in 1.8.0 in
> 	October 2012).
>
> My first reaction is to wonder if it is time to introduce a config
> option to use index format version 4 by default, since after 5 years
> it is not relatively young any more.
>
> My second reaction is to notice that the index.version configuration
> already exists.  Probably git-update-index(1) ought to point to it.

Both good points.

> JGit still doesn't support index format 4, so 4 is still not a good
> default for `index.version` for a user that hasn't explicitly
> configured it, but the moment JGit gains that support I think it will
> be. :)

After reading ce_write_entry(), I tend to agree.  v4 is the first
one that loses the alignment padding, which made sense back when we
mmap(2)ed the index file and read from it directly without the
ondisk-vs-incore conversion, but that stopped to be the case several
years ago.

I find it interesting to find that the recently proposed design for
reftable uses the same "common prefix compression from the previous
name" trick, even though index v4 does not have any resync point
like reftable does.