How to add custom metadata to Git commit object

How to add custom metadata to Git commit object

[Kirill Likhodedov]

Is it possible to add custom metadata to Git commit object? 
Such metadata should be ignored by Git commands, but could be used by a 3-party tool which knows the format and knows where to look. 

I assume that this should be possible, given that Git objects are actually patches, and patches can contain additional details. But can this be done with the help of Git commands? 

----

The reason why I am asking this - is to create a tool which could overcome false rename detection. 

As all of you know, if I make significant changes to the code together with rename, Git won’t detect this rename and will treat this change as added + deleted. And sometimes there are false rename detections as well. It would be useful to record the fact of rename and use it afterwards.

If a user is developing with our IDE (IntelliJ IDEA), we could remember the fact that he renamed a file, then write this information to the commit object, and when the commit information is requested (e.g. from the git log graphical view), the IDE could read the additional information of the commit and display the file as renamed, not as added + deleted. The IDE could also use this information to follow rename through the file history.

As a real example, in our project we are converting a lot of files from Java to Kotlin, and such conversion always looses history unless the developer remembers to prepare a separate rename-commit first, which is tedious. 

There are git-notes, which could be used for the purpose, but they are visible to the user via standard Git command, and could be used by the user for other purposes, so they are not very suitable for the task.

Thanks a lot!
-- Kirill

Re: How to add custom metadata to Git commit object

[Konstantin Khomoutov]

On Mon, 30 May 2016 20:58:08 +0300
Kirill Likhodedov <kirill.likhodedov@jetbrains.com> wrote:

> Is it possible to add custom metadata to Git commit object? 
> Such metadata should be ignored by Git commands, but could be used by
> a 3-party tool which knows the format and knows where to look. 
> 
> I assume that this should be possible, given that Git objects are
> actually patches, and patches can contain additional details. But can
> this be done with the help of Git commands? 
[...]
> There are git-notes, which could be used for the purpose, but they
> are visible to the user via standard Git command, and could be used
> by the user for other purposes, so they are not very suitable for the
> task.

AFAIK, within your restrictions, it's not possible because there are
only two ways to add meta information for a Git commit:

* Store it externally and somehow correlate it with the commit.

  This is what git-notes does.

* Encode it directly into a commit object.

  Since you can't use your own headers in commit objects,
  you have to encode this information into the commit message in some
  form parsable by a machine.   This is what, say, git-svn does to
  make it possible to correlate the commits it creates with their source
  Subversion revisions.

In both cases the information can be viewed by the user.

What I can't really understand is what is so bad about the user being
able to peer at that data.

Re: How to add custom metadata to Git commit object

[Jeff King]

On Mon, May 30, 2016 at 08:58:08PM +0300, Kirill Likhodedov wrote:

> Is it possible to add custom metadata to Git commit object?  Such
> metadata should be ignored by Git commands, but could be used by a
> 3-party tool which knows the format and knows where to look.

Yes. The recommended place to stick this is in a "trailer" at the bottom
of the commit message, like:

  Rename-detection-hint: foo.java -> bar.java

or whatever scheme is useful to your tool.

> I assume that this should be possible, given that Git objects are
> actually patches, and patches can contain additional details. But can
> this be done with the help of Git commands?

Git objects aren't actually patches. Try:

  git cat-file commit HEAD

to see what an actual commit object looks like. You will see that there
are user-invisible headers before the commit message, too, but we do not
usually recommend people to add new headers here, as their semantics
would be unclear to git. For example, when rebasing such a commit,
should the header be preserved or not? It depends on its meaning.
Whereas commit messages are always preserved.

> There are git-notes, which could be used for the purpose, but they are
> visible to the user via standard Git command, and could be used by the
> user for other purposes, so they are not very suitable for the task.

Notes would work for this, too, but their main advantage is that they
can be created _after_ a commit has already been made (whereas anything
in the commit object itself will influence its sha1 id).

-Peff

Re: How to add custom metadata to Git commit object

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Mon, May 30, 2016 at 08:58:08PM +0300, Kirill Likhodedov wrote:
> ...
>> There are git-notes, which could be used for the purpose, but they are
>> visible to the user via standard Git command, and could be used by the
>> user for other purposes, so they are not very suitable for the task.
>
> Notes would work for this, too, but their main advantage is that they
> can be created _after_ a commit has already been made (whereas anything
> in the commit object itself will influence its sha1 id).

I would have said the same but with s/but/and/.  If the "rename
hint" or whatever other "custom metadata" Kirill gives to a commit
is found to be wrong, it can be corrected later.

And "the user can use notes for other purposes" is not a good reason
to reject them.  The whole point of allowing custom notes ref is so
that Kirill is not restricted to use the usual notes/commits ref to
store this custom notes in its dedicated notes/kirills-metadata ref.

Re: How to add custom metadata to Git commit object

[Kirill Likhodedov]

Thanks a lot for your suggestions!

> And "the user can use notes for other purposes" is not a good reason
> to reject them.  The whole point of allowing custom notes ref is so
> that Kirill is not restricted to use the usual notes/commits ref to
> store this custom notes in its dedicated notes/kirills-metadata ref.

I think the suggestion from Junio will suit best.
 
Somehow I missed the fact that notes can reside in several namespaces. 
Now I see that the notes are very powerful having namespaces
and several configuration and environment variables.

Jeff,

> there are user-invisible headers before the commit message, too, but we do not
> usually recommend people to add new headers here, as their semantics
> would be unclear to git. For example, when rebasing such a commit,
> should the header be preserved or not? It depends on its meaning.
> Whereas commit messages are always preserved.


Good point about rebase. 
Just out of curiosity, is it possible to add a custom invisible header 
to commit object with some Git command? 
git commit or commit-tree don’t have an option for this.


> On 30 May 2016, at 21:52 , Junio C Hamano <gitster@pobox.com> wrote:
> 
> Jeff King <peff@peff.net> writes:
> 
>> On Mon, May 30, 2016 at 08:58:08PM +0300, Kirill Likhodedov wrote:
>> ...
>>> There are git-notes, which could be used for the purpose, but they are
>>> visible to the user via standard Git command, and could be used by the
>>> user for other purposes, so they are not very suitable for the task.
>> 
>> Notes would work for this, too, but their main advantage is that they
>> can be created _after_ a commit has already been made (whereas anything
>> in the commit object itself will influence its sha1 id).
> 
> I would have said the same but with s/but/and/.  If the "rename
> hint" or whatever other "custom metadata" Kirill gives to a commit
> is found to be wrong, it can be corrected later.
> 
> And "the user can use notes for other purposes" is not a good reason
> to reject them.  The whole point of allowing custom notes ref is so
> that Kirill is not restricted to use the usual notes/commits ref to
> store this custom notes in its dedicated notes/kirills-metadata ref.

Re: How to add custom metadata to Git commit object

[Junio C Hamano]

Kirill Likhodedov <kirill.likhodedov@jetbrains.com> writes:

> Just out of curiosity, is it possible to add a custom invisible header 
> to commit object with some Git command? 
> git commit or commit-tree don’t have an option for this.

There isn't, and that is very much deliberate. We do not want to
make it easy for people to even experiment with such "custom"
headers; we want to encourage people to find solutions to whatever
problem they are trying to solve that does not involve touching the
object header.