Linking topic merges to mailing list threads

Linking topic merges to mailing list threads

[Emily Shaffer]

Hi all,

We've been wanting to gather metrics on Git's code review process -
how long it takes from first contact on list to merge, how many
iterations are needed, time between iterations, etc. One missing link
is the actual merge time in `next` and `master` - a human can infer
the link between the patch and the mailing list thread, but it's more
challenging for a script to do it.

Would it be possible to modify the maintainer workflow to include a
link to the cover letter as merged in the merge commit message (or the
link to the latest iteration of the patch if it's a single-patch
change)? What issues could come up with that workflow?

I guess one is that we could move from lore.kernel.org to something
else, like we saw the migration from public-inbox.org some years ago.
But the Message-ID was preserved between the two archives, so maybe
it's enough to include the Message-ID in the merge commit? Another is,
of course, the added burden on the maintainer. But maybe there is some
script that is already used that we can modify to make the extra load
negligible?

(Or, even better, if anybody else is already successfully measuring
these kinds of metrics without such a reference, could you let me know
how you're doing it? :) )

Thanks,
 - Emily

Re: Linking topic merges to mailing list threads

[Konstantin Ryabitsev]

On Mon, Sep 30, 2024 at 09:21:11AM GMT, Emily Shaffer wrote:
> Hi all,
> 
> We've been wanting to gather metrics on Git's code review process -
> how long it takes from first contact on list to merge, how many
> iterations are needed, time between iterations, etc. One missing link
> is the actual merge time in `next` and `master` - a human can infer
> the link between the patch and the mailing list thread, but it's more
> challenging for a script to do it.
> 
> Would it be possible to modify the maintainer workflow to include a
> link to the cover letter as merged in the merge commit message (or the
> link to the latest iteration of the patch if it's a single-patch
> change)? What issues could come up with that workflow?

One of the goals of b4 on the kernel side of things was to promote the use of
cover letters as merge commit templates, but this requires buy-in from
maintainers. It also doesn't really work for single-patch series.

For example, applying a series with "b4 shazam -M" will:

- fetch the series into FETCH_HEAD
- use the cover letter as the basis for the merge commit message
- insert the links to the source of the series
- open up the editor, allowing the maintainer to edit the merge commit message

Here's an example of such merge:

https://git.kernel.org/pub/scm/utils/b4/b4.git/commit/?id=b6b73918d94985bb2a017784fc14e013b36b38d0

> I guess one is that we could move from lore.kernel.org to something
> else, like we saw the migration from public-inbox.org some years ago.
> But the Message-ID was preserved between the two archives, so maybe
> it's enough to include the Message-ID in the merge commit?

This should be sufficient, yes, because you should still be able to find the
origin thread even if lore.kernel.org is defunct at some point.

> Another is, of course, the added burden on the maintainer. But maybe there
> is some script that is already used that we can modify to make the extra
> load negligible?

There is. :)

> (Or, even better, if anybody else is already successfully measuring
> these kinds of metrics without such a reference, could you let me know
> how you're doing it? :) )

On the kernel side, any time the topic of metrics comes up, it gets
immediately bogged down in "how much tracking is okay and how much is spying"
kinds of discussions that have never resulted in anything, really.

-K

Re: Linking topic merges to mailing list threads

[Junio C Hamano]

Emily Shaffer <nasamuffin@google.com> writes:

> We've been wanting to gather metrics on Git's code review process -
> how long it takes from first contact on list to merge, how many
> iterations are needed, time between iterations, etc. One missing link
> is the actual merge time in `next` and `master` - a human can infer
> the link between the patch and the mailing list thread, but it's more
> challenging for a script to do it.
>
> Would it be possible to modify the maintainer workflow ...

I suspect that there is no need for any workflow change, as all the
necessary information should be available from public sources.

The first-parent chain from 'next' (or 'master' for that matter)
already record when they got merged.  From there, C^1..C^2 are
the commit objects that were merged.  notes/amlog knows where
they came from (i.e. their Message-Id).  From lore/public-inbox
you can find out how the iterations of topics went, as long as
the topics are threaded properly (and if not, that would not be
fixable with any maintainer workflow changes), just like how b4 can
figure all of that out.

Ahh, nothing officially documents amlog and that is what you are
missing.  It would be very nice if somebody, preferrably somebody
other than I, after trying the "maintainer workflow" by pretending
to be a maintainer for a day or two with the new info revealed here,
updates the Documentation/howto/maintain-git.txt file with the
information below.

The script post-appplypatch found in the todo branch is made
available as .git/hooks/post-applypatch so that "git am" knows to
run it after creating a commit out of an e-mailed patch.  It
populates a mapping from commit object name to "Message-Id" of
individual patch.

"git rebase" knows how to propagate this across rebases because
I have

    [notes] rewriteref = refs/notes/amlog

in the .git/config (which means I have to use rebase not cherry-pick
even when I am touching a single patch, as cherry-pick does not
preserve notes by design).

Now I think you should have everything, together with what is
already in Documentation/howto/maintain-git.txt, piece them together
to illustrate the life of a patch series.

As I do not publish reflog for 'seen', you cannot do "when was the
topic got picked up to 'seen'?", but as far as I am concerned, it is
by design.  Being in 'seen' does not mean anything other than I
happened to have seen it, or saw that somebody indicate interest in
it.

Thanks.

[PATCH] Documentation: mention the amlog in howto/maintain-git.txt

[Taylor Blau]

Part of the maintainer's job is to keep up-to-date and publish the
'amlog' which stores a mapping between a patch's 'Message-Id' e-mail
header and the commit generated by applying said patch.

But our Documentation/howto/maintain-git.txt does not mention the amlog,
or the scripts which exist to help the maintainer keep the amlog
up-to-date.

(This bit me during the first integration round I did as interim
maintainer[1] involved a lot of manual clean-up. More recently it has
come up as part of a research effort to better understand a patch's
lifecycle on the list[2].)

Address this gap by briefly documenting the existence and purpose of the
'post-applypatch' hook in maintaining the amlog entries.

[1]: https://lore.kernel.org/git/Y19dnb2M+yObnftj@nand.local/
[2]: https://lore.kernel.org/git/CAJoAoZ=4ARuH3aHGe5yC_Xcnou_c396q_ZienYPY7YnEzZcyEg@mail.gmail.com/

Suggested-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 Documentation/howto/maintain-git.txt | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/Documentation/howto/maintain-git.txt b/Documentation/howto/maintain-git.txt
index da31332f11..fd1560327c 100644
--- a/Documentation/howto/maintain-git.txt
+++ b/Documentation/howto/maintain-git.txt
@@ -165,6 +165,22 @@ by doing the following:
    In practice, almost no patch directly goes to 'master' or
    'maint'.
 
+   The maintainer is expected to update refs/notes/amlog with a
+   mapping between the applied commit and the 'Message-Id'
+   corresponding to the e-mail which carried the patch.
+
+   This mapping is created with the aid of the "post-applypatch" hook
+   found in the 'todo' branch. That hook should be installed before
+   applying patches. It is also helpful to carry forward any relevant
+   amlog entries when rebasing, so the following config may be useful:
+
+      [notes]
+	rewriteref = refs/notes/amlog
+
+   Finally, take care that the amlog entries are pushed out during
+   integration cycles since external tools and contributors (in
+   addition to internal scripts) may rely on them.
+
  - Review the last issue of "What's cooking" message, review the
    topics ready for merging (topic->master and topic->maint).  Use
    "Meta/cook -w" script (where Meta/ contains a checkout of the

base-commit: 3857aae53f3633b7de63ad640737c657387ae0c6
-- 
2.46.2.633.gf09c3c1769.dirty

Re: [PATCH] Documentation: mention the amlog in howto/maintain-git.txt

[Kristoffer Haugsbakk]

On Mon, Sep 30, 2024, at 22:45, Taylor Blau wrote:
> Part of the maintainer's job is to keep up-to-date and publish the
> 'amlog' which stores a mapping between a patch's 'Message-Id' e-mail
> header and the commit generated by applying said patch.
>
> But our Documentation/howto/maintain-git.txt does not mention the amlog,
> or the scripts which exist to help the maintainer keep the amlog
> up-to-date.
>
> (This bit me during the first integration round I did as interim
> maintainer[1] involved a lot of manual clean-up. More recently it has
> come up as part of a research effort to better understand a patch's
> lifecycle on the list[2].)
>
> Address this gap by briefly documenting the existence and purpose of the
> 'post-applypatch' hook in maintaining the amlog entries.
>
> [1]: https://lore.kernel.org/git/Y19dnb2M+yObnftj@nand.local/
> [2]:
> https://lore.kernel.org/git/CAJoAoZ=4ARuH3aHGe5yC_Xcnou_c396q_ZienYPY7YnEzZcyEg@mail.gmail.com/
>
> Suggested-by: Junio C Hamano <gitster@pobox.com>
> Signed-off-by: Taylor Blau <me@ttaylorr.com>
> ---
>  Documentation/howto/maintain-git.txt | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
>
> diff --git a/Documentation/howto/maintain-git.txt
> b/Documentation/howto/maintain-git.txt
> index da31332f11..fd1560327c 100644
> --- a/Documentation/howto/maintain-git.txt
> +++ b/Documentation/howto/maintain-git.txt
> @@ -165,6 +165,22 @@ by doing the following:
>     In practice, almost no patch directly goes to 'master' or
>     'maint'.
>
> +   The maintainer is expected to update refs/notes/amlog with a
> +   mapping between the applied commit and the 'Message-Id'
> +   corresponding to the e-mail which carried the patch.
> +
> +   This mapping is created with the aid of the "post-applypatch" hook
> +   found in the 'todo' branch. That hook should be installed before
> +   applying patches. It is also helpful to carry forward any relevant
> +   amlog entries when rebasing, so the following config may be useful:
> +
> +      [notes]
> +	rewriteref = refs/notes/amlog

Nit: `[notes]` is indented with spaces while the next line is indented
with a tab.  I guess it’s supposed to just be spaces in this context?

> +
> +   Finally, take care that the amlog entries are pushed out during
> +   integration cycles since external tools and contributors (in
> +   addition to internal scripts) may rely on them.
> +
>   - Review the last issue of "What's cooking" message, review the
>     topics ready for merging (topic->master and topic->maint).  Use
>     "Meta/cook -w" script (where Meta/ contains a checkout of the
>
> base-commit: 3857aae53f3633b7de63ad640737c657387ae0c6
> --
> 2.46.2.633.gf09c3c1769.dirty

It might be worth explicitly mentioning the git-cherry-pick(1) footgun
that Junio talked about in his email: you have to restrict yourself to
git-rebase(1) and `git commit --amend`.  Since git-cherry-pick(1)
doesn’t care about (respect?) this configuration.

Right now it’s implied of course (“when rebasing”).

-- 
Kristoffer Haugsbakk

Re: [PATCH] Documentation: mention the amlog in howto/maintain-git.txt

[Taylor Blau]

On Mon, Sep 30, 2024 at 11:06:03PM +0200, Kristoffer Haugsbakk wrote:
> > @@ -165,6 +165,22 @@ by doing the following:
> >     In practice, almost no patch directly goes to 'master' or
> >     'maint'.
> >
> > +   The maintainer is expected to update refs/notes/amlog with a
> > +   mapping between the applied commit and the 'Message-Id'
> > +   corresponding to the e-mail which carried the patch.
> > +
> > +   This mapping is created with the aid of the "post-applypatch" hook
> > +   found in the 'todo' branch. That hook should be installed before
> > +   applying patches. It is also helpful to carry forward any relevant
> > +   amlog entries when rebasing, so the following config may be useful:
> > +
> > +      [notes]
> > +	rewriteref = refs/notes/amlog
>
> Nit: `[notes]` is indented with spaces while the next line is indented
> with a tab.  I guess it’s supposed to just be spaces in this context?

Oops, good catch, thanks.

> It might be worth explicitly mentioning the git-cherry-pick(1) footgun
> that Junio talked about in his email: you have to restrict yourself to
> git-rebase(1) and `git commit --amend`.  Since git-cherry-pick(1)
> doesn’t care about (respect?) this configuration.

I think that's worth mentioning, and I added a small tidbit in the
latest round mentioning it, thanks.

Thanks,
Taylor

Re: [PATCH] Documentation: mention the amlog in howto/maintain-git.txt

[Junio C Hamano]

Taylor Blau <me@ttaylorr.com> writes:

> Part of the maintainer's job is to keep up-to-date and publish the
> 'amlog' which stores a mapping between a patch's 'Message-Id' e-mail
> header and the commit generated by applying said patch.
>
> But our Documentation/howto/maintain-git.txt does not mention the amlog,
> or the scripts which exist to help the maintainer keep the amlog
> up-to-date.
>
> (This bit me during the first integration round I did as interim
> maintainer[1] involved a lot of manual clean-up. More recently it has
> come up as part of a research effort to better understand a patch's
> lifecycle on the list[2].)
>
> Address this gap by briefly documenting the existence and purpose of the
> 'post-applypatch' hook in maintaining the amlog entries.
>
> [1]: https://lore.kernel.org/git/Y19dnb2M+yObnftj@nand.local/
> [2]: https://lore.kernel.org/git/CAJoAoZ=4ARuH3aHGe5yC_Xcnou_c396q_ZienYPY7YnEzZcyEg@mail.gmail.com/
>
> Suggested-by: Junio C Hamano <gitster@pobox.com>
> Signed-off-by: Taylor Blau <me@ttaylorr.com>
> ---
>  Documentation/howto/maintain-git.txt | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)

This addition to the procedure part of the documentation reads good.

We'd need a matching addition to "The Policy" part, to describe the
motivation separately.  The procedure exists only to realize what
the policy gives, and we need something to back up the expectation
"to update refs/notes/amlog" (i.e. because there is this policy).

Existig "policy" entries are only about how integration branches are
maintained and used, but notes/amlog are solely about the individual
patches, so we'd need an entirely new section there, I guess.

While at it, I notice that there is no mention on where these notes
are published (the configuration you added to the text is solely
about the local repository the maintainer uses).

I just added this change

 [remote "github2"]
         url = https://github.com/git/git
         fetch = +refs/heads/*:refs/remotes/github2/*
         pushurl = github.com:git/git.git
         push = refs/heads/maint:refs/heads/maint
         push = refs/heads/master:refs/heads/master
         push = refs/heads/next:refs/heads/next
         push = +refs/heads/seen:refs/heads/seen
+        push = +refs/notes/amlog

to github.com/git/git/ and other publishing repositories.  My
broken-out repository github.com/gitster/git/ have been pushed
with the mirror mode, so there needs no change, but others like
k.org repositories will start seeing this additional ref when I push
out today's integration results.

The "policy" part of the change may read like the following.

Thanks.

 Documentation/howto/maintain-git.txt | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git c/Documentation/howto/maintain-git.txt w/Documentation/howto/maintain-git.txt
index da31332f11..9b72d435e6 100644
--- c/Documentation/howto/maintain-git.txt
+++ w/Documentation/howto/maintain-git.txt
@@ -35,6 +35,14 @@ The maintainer's Git time is spent on three activities.
 The Policy
 ----------
 
+Because most of the lines of code in Git are written by individual
+contributors, and contributions come in the form of e-mailed patches
+published on the mailing list, the project maintains a mapping from
+individual commits to the Message-Id of the e-mail that resulted in
+the commit, to help tracking the origin of the changes.  The notes
+in "refs/notes/amlog" are used for this purpose, and are published
+along with the broken-out branches to the maintainer's repository.
+
 The policy on Integration is informally mentioned in "A Note
 from the maintainer" message, which is periodically posted to
 the mailing list after each feature release is made:

Re: [PATCH] Documentation: mention the amlog in howto/maintain-git.txt

[Taylor Blau]

On Mon, Sep 30, 2024 at 02:54:23PM -0700, Junio C Hamano wrote:
> The "policy" part of the change may read like the following.
>
> Thanks.
>
>  Documentation/howto/maintain-git.txt | 8 ++++++++
>  1 file changed, 8 insertions(+)
>
> diff --git c/Documentation/howto/maintain-git.txt w/Documentation/howto/maintain-git.txt
> index da31332f11..9b72d435e6 100644
> --- c/Documentation/howto/maintain-git.txt
> +++ w/Documentation/howto/maintain-git.txt
> @@ -35,6 +35,14 @@ The maintainer's Git time is spent on three activities.
>  The Policy
>  ----------
>
> +Because most of the lines of code in Git are written by individual
> +contributors, and contributions come in the form of e-mailed patches
> +published on the mailing list, the project maintains a mapping from
> +individual commits to the Message-Id of the e-mail that resulted in
> +the commit, to help tracking the origin of the changes.  The notes
> +in "refs/notes/amlog" are used for this purpose, and are published
> +along with the broken-out branches to the maintainer's repository.
> +
>  The policy on Integration is informally mentioned in "A Note
>  from the maintainer" message, which is periodically posted to
>  the mailing list after each feature release is made:

Thanks, this looks good to me, and I added it in the latest version of
this patch, with your Helped-by.

I moved this section to the end of the this section, not the beginning,
since it seems more important to first discuss the mechanics of topic
branches, next, seen, master, etc., before getting to the nuts and bolts
of the amlog ;-).

Thanks,
Taylor

[PATCH v2] Documentation: mention the amlog in howto/maintain-git.txt

[Taylor Blau]

Part of the maintainer's job is to keep up-to-date and publish the
'amlog' which stores a mapping between a patch's 'Message-Id' e-mail
header and the commit generated by applying said patch.

But our Documentation/howto/maintain-git.txt does not mention the amlog,
or the scripts which exist to help the maintainer keep the amlog
up-to-date.

(This bit me during the first integration round I did as interim
maintainer[1] involved a lot of manual clean-up. More recently it has
come up as part of a research effort to better understand a patch's
lifecycle on the list[2].)

Address this gap by briefly documenting the existence and purpose of the
'post-applypatch' hook in maintaining the amlog entries.

[1]: https://lore.kernel.org/git/Y19dnb2M+yObnftj@nand.local/
[2]: https://lore.kernel.org/git/CAJoAoZ=4ARuH3aHGe5yC_Xcnou_c396q_ZienYPY7YnEzZcyEg@mail.gmail.com/

Suggested-by: Junio C Hamano <gitster@pobox.com>
Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
Similar to v1, but with:

- an added change to "The Policy" section written by Junio

- a tab/space fix in the notes.rewriteRef example

- and a mention of the fact that the notes.rewriteRef configuration is
  not read by 'cherry-pick'.

 Documentation/howto/maintain-git.txt | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/Documentation/howto/maintain-git.txt b/Documentation/howto/maintain-git.txt
index da31332f11..76d6688d4c 100644
--- a/Documentation/howto/maintain-git.txt
+++ b/Documentation/howto/maintain-git.txt
@@ -122,6 +122,13 @@ Note that before v1.9.0 release, the version numbers used to be
 structured slightly differently.  vX.Y.Z were feature releases while
 vX.Y.Z.W were maintenance releases for vX.Y.Z.

+Because most of the lines of code in Git are written by individual
+contributors, and contributions come in the form of e-mailed patches
+published on the mailing list, the project maintains a mapping from
+individual commits to the Message-Id of the e-mail that resulted in
+the commit, to help tracking the origin of the changes. The notes
+in "refs/notes/amlog" are used for this purpose, and are published
+along with the broken-out branches to the maintainer's repository.

 A Typical Git Day
 -----------------
@@ -165,6 +172,24 @@ by doing the following:
    In practice, almost no patch directly goes to 'master' or
    'maint'.

+   The maintainer is expected to update refs/notes/amlog with a
+   mapping between the applied commit and the 'Message-Id'
+   corresponding to the e-mail which carried the patch.
+
+   This mapping is created with the aid of the "post-applypatch" hook
+   found in the 'todo' branch. That hook should be installed before
+   applying patches. It is also helpful to carry forward any relevant
+   amlog entries when rebasing, so the following config may be useful:
+
+      [notes]
+        rewriteRef = refs/notes/amlog
+
+   (note that this configuration is not read by 'cherry-pick').
+
+   Finally, take care that the amlog entries are pushed out during
+   integration cycles since external tools and contributors (in
+   addition to internal scripts) may rely on them.
+
  - Review the last issue of "What's cooking" message, review the
    topics ready for merging (topic->master and topic->maint).  Use
    "Meta/cook -w" script (where Meta/ contains a checkout of the

Range-diff against v1:
1:  a4b1da93e1 ! 1:  5cc8e2bcb8 Documentation: mention the amlog in howto/maintain-git.txt
    @@ Commit message
         [2]: https://lore.kernel.org/git/CAJoAoZ=4ARuH3aHGe5yC_Xcnou_c396q_ZienYPY7YnEzZcyEg@mail.gmail.com/

         Suggested-by: Junio C Hamano <gitster@pobox.com>
    +    Helped-by: Junio C Hamano <gitster@pobox.com>
         Signed-off-by: Taylor Blau <me@ttaylorr.com>

      ## Documentation/howto/maintain-git.txt ##
    +@@ Documentation/howto/maintain-git.txt: Note that before v1.9.0 release, the version numbers used to be
    + structured slightly differently.  vX.Y.Z were feature releases while
    + vX.Y.Z.W were maintenance releases for vX.Y.Z.
    +
    ++Because most of the lines of code in Git are written by individual
    ++contributors, and contributions come in the form of e-mailed patches
    ++published on the mailing list, the project maintains a mapping from
    ++individual commits to the Message-Id of the e-mail that resulted in
    ++the commit, to help tracking the origin of the changes. The notes
    ++in "refs/notes/amlog" are used for this purpose, and are published
    ++along with the broken-out branches to the maintainer's repository.
    +
    + A Typical Git Day
    + -----------------
     @@ Documentation/howto/maintain-git.txt: by doing the following:
         In practice, almost no patch directly goes to 'master' or
         'maint'.
    @@ Documentation/howto/maintain-git.txt: by doing the following:
     +   amlog entries when rebasing, so the following config may be useful:
     +
     +      [notes]
    -+	rewriteref = refs/notes/amlog
    ++        rewriteRef = refs/notes/amlog
    ++
    ++   (note that this configuration is not read by 'cherry-pick').
     +
     +   Finally, take care that the amlog entries are pushed out during
     +   integration cycles since external tools and contributors (in

base-commit: 3857aae53f3633b7de63ad640737c657387ae0c6
--
2.47.0.rc0.4.gd73fb26592.dirty

Re: [PATCH v2] Documentation: mention the amlog in howto/maintain-git.txt

[Junio C Hamano]

Taylor Blau <me@ttaylorr.com> writes:

Now the policy explains what is done (i.e. amlog gives a mapping)
and for what purpose (i.e. we want to be able to go back to the
origin), "... is expected to" in the actual procedure is redundant.
In other words, the procedure section can focus on what is done
without repeating why.

> +   The maintainer is expected to update refs/notes/amlog with a
> +   mapping between the applied commit and the 'Message-Id'
> +   corresponding to the e-mail which carried the patch.

I'd replace the above with something like:

      Applying the e-mailed patches using "git am" automatically
      records the mappings from Message-Id to resulting commit in
      the "amlog" note.  Periodically check that this is working
      with "git show -s --notes=amlog $commit".

> +   This mapping is created with the aid of the "post-applypatch" hook

"created" -> "maintained".

> +   found in the 'todo' branch. That hook should be installed before
> +   applying patches. It is also helpful to carry forward any relevant
> +   amlog entries when rebasing, so the following config may be useful:
> +
> +      [notes]
> +        rewriteRef = refs/notes/amlog
> +
> +   (note that this configuration is not read by 'cherry-pick').

"(note ...)" ->

      Avoid "cherry-pick", as it does not propagate notes by design.
      Use either "git commit --amend" or "git rebase" to make
      corrections to an existing commit, even for a single-patch
      topic.

> +   Finally, take care that the amlog entries are pushed out during
> +   integration cycles since external tools and contributors (in
> +   addition to internal scripts) may rely on them.

The purpose of pushing amlog out does not need to be repeated here
in the procedure section.
	
      Make sure that push refspec for refs/notes/amlog is in the
      remote configuration for publishing repositories.  A few
      sample .git/config entries may look like this:

        [remote "github2"]
                url = https://github.com/git/git
                fetch = +refs/heads/*:refs/remotes/github2/*
                pushurl = github.com:git/git.git
                push = refs/heads/maint:refs/heads/maint
                push = refs/heads/master:refs/heads/master
                push = refs/heads/next:refs/heads/next
                push = +refs/heads/seen:refs/heads/seen
                push = +refs/notes/amlog

        [remote "github"]
                url = https://github.com/gitster/git
                pushurl = github.com:gitster/git.git
                mirror

Other than that, nicely done.

Thanks for filling the gap in the documentation.

Re: [PATCH v2] Documentation: mention the amlog in howto/maintain-git.txt

[Taylor Blau]

On Wed, Oct 02, 2024 at 12:47:29PM -0700, Junio C Hamano wrote:
> Other than that, nicely done.

All very fair suggestions. A v3 is on its way...

Thanks,
Taylor

[PATCH v3] Documentation: mention the amlog in howto/maintain-git.txt

[Taylor Blau]

Part of the maintainer's job is to keep up-to-date and publish the
'amlog' which stores a mapping between a patch's 'Message-Id' e-mail
header and the commit generated by applying said patch.

But our Documentation/howto/maintain-git.txt does not mention the amlog,
or the scripts which exist to help the maintainer keep the amlog
up-to-date.

(This bit me during the first integration round I did as interim
maintainer[1] involved a lot of manual clean-up. More recently it has
come up as part of a research effort to better understand a patch's
lifecycle on the list[2].)

Address this gap by briefly documenting the existence and purpose of the
'post-applypatch' hook in maintaining the amlog entries.

[1]: https://lore.kernel.org/git/Y19dnb2M+yObnftj@nand.local/
[2]: https://lore.kernel.org/git/CAJoAoZ=4ARuH3aHGe5yC_Xcnou_c396q_ZienYPY7YnEzZcyEg@mail.gmail.com/

Suggested-by: Junio C Hamano <gitster@pobox.com>
Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 Documentation/howto/maintain-git.txt | 44 ++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)

diff --git a/Documentation/howto/maintain-git.txt b/Documentation/howto/maintain-git.txt
index da31332f113..f52f32eda93 100644
--- a/Documentation/howto/maintain-git.txt
+++ b/Documentation/howto/maintain-git.txt
@@ -122,6 +122,13 @@ Note that before v1.9.0 release, the version numbers used to be
 structured slightly differently.  vX.Y.Z were feature releases while
 vX.Y.Z.W were maintenance releases for vX.Y.Z.
 
+Because most of the lines of code in Git are written by individual
+contributors, and contributions come in the form of e-mailed patches
+published on the mailing list, the project maintains a mapping from
+individual commits to the Message-Id of the e-mail that resulted in
+the commit, to help tracking the origin of the changes. The notes
+in "refs/notes/amlog" are used for this purpose, and are published
+along with the broken-out branches to the maintainer's repository.
 
 A Typical Git Day
 -----------------
@@ -165,6 +172,43 @@ by doing the following:
    In practice, almost no patch directly goes to 'master' or
    'maint'.
 
+   Applying the e-mailed patches using "git am" automatically records
+   the mappings from 'Message-Id' to the applied commit in the "amlog"
+   notes. Periodically check that this is working with "git show -s
+   --notes=amlog $commit".
+
+   This mapping is maintained with the aid of the "post-applypatch"
+   hook found in the 'todo' branch. That hook should be installed
+   before applying patches. It is also helpful to carry forward any
+   relevant amlog entries when rebasing, so the following config may
+   be useful:
+
+      [notes]
+        rewriteRef = refs/notes/amlog
+
+   Avoid "cherry-pick", as it does not propagate notes by design. Use
+   either "git commit --amend" or "git rebase" to make corrections to
+   an existing commit, even for a single-patch topic.
+
+   Make sure that a push refspec for 'refs/notes/amlog' is in the
+   remote configuration for publishing repositories. A few sample
+   configurations look like the following:
+
+      [remote "github"]
+        url = https://github.com/gitster/git
+        pushurl = github.com:gitster/git.git
+        mirror
+
+      [remote "github2"]
+        url = https://github.com/git/git
+        fetch = +refs/heads/*:refs/remotes/github2/*
+        pushurl = github.com:git/git.git
+        push = refs/heads/maint:refs/heads/maint
+        push = refs/heads/master:refs/heads/master
+        push = refs/notes/next:refs/notes/next
+        push = +refs/heads/seen:refs/heads/seen
+        push = +refs/notes/amlog
+
  - Review the last issue of "What's cooking" message, review the
    topics ready for merging (topic->master and topic->maint).  Use
    "Meta/cook -w" script (where Meta/ contains a checkout of the

Range-diff against v2:
1:  5cc8e2bcb88 ! 1:  88a13b9f2b6 Documentation: mention the amlog in howto/maintain-git.txt
    @@ Documentation/howto/maintain-git.txt: by doing the following:
         In practice, almost no patch directly goes to 'master' or
         'maint'.
      
    -+   The maintainer is expected to update refs/notes/amlog with a
    -+   mapping between the applied commit and the 'Message-Id'
    -+   corresponding to the e-mail which carried the patch.
    ++   Applying the e-mailed patches using "git am" automatically records
    ++   the mappings from 'Message-Id' to the applied commit in the "amlog"
    ++   notes. Periodically check that this is working with "git show -s
    ++   --notes=amlog $commit".
     +
    -+   This mapping is created with the aid of the "post-applypatch" hook
    -+   found in the 'todo' branch. That hook should be installed before
    -+   applying patches. It is also helpful to carry forward any relevant
    -+   amlog entries when rebasing, so the following config may be useful:
    ++   This mapping is maintained with the aid of the "post-applypatch"
    ++   hook found in the 'todo' branch. That hook should be installed
    ++   before applying patches. It is also helpful to carry forward any
    ++   relevant amlog entries when rebasing, so the following config may
    ++   be useful:
     +
     +      [notes]
     +        rewriteRef = refs/notes/amlog
     +
    -+   (note that this configuration is not read by 'cherry-pick').
    ++   Avoid "cherry-pick", as it does not propagate notes by design. Use
    ++   either "git commit --amend" or "git rebase" to make corrections to
    ++   an existing commit, even for a single-patch topic.
     +
    -+   Finally, take care that the amlog entries are pushed out during
    -+   integration cycles since external tools and contributors (in
    -+   addition to internal scripts) may rely on them.
    ++   Make sure that a push refspec for 'refs/notes/amlog' is in the
    ++   remote configuration for publishing repositories. A few sample
    ++   configurations look like the following:
    ++
    ++      [remote "github"]
    ++        url = https://github.com/gitster/git
    ++        pushurl = github.com:gitster/git.git
    ++        mirror
    ++
    ++      [remote "github2"]
    ++        url = https://github.com/git/git
    ++        fetch = +refs/heads/*:refs/remotes/github2/*
    ++        pushurl = github.com:git/git.git
    ++        push = refs/heads/maint:refs/heads/maint
    ++        push = refs/heads/master:refs/heads/master
    ++        push = refs/notes/next:refs/notes/next
    ++        push = +refs/heads/seen:refs/heads/seen
    ++        push = +refs/notes/amlog
     +
       - Review the last issue of "What's cooking" message, review the
         topics ready for merging (topic->master and topic->maint).  Use

base-commit: 3857aae53f3633b7de63ad640737c657387ae0c6
-- 
2.47.0.rc0.8.g9a975e77790

Re: [PATCH v3] Documentation: mention the amlog in howto/maintain-git.txt

[Junio C Hamano]

Taylor Blau <me@ttaylorr.com> writes:

> (This bit me during the first integration round I did as interim
> maintainer[1] involved a lot of manual clean-up. More recently it has
> come up as part of a research effort to better understand a patch's
> lifecycle on the list[2].)

Thanks.

Re: [PATCH v3] Documentation: mention the amlog in howto/maintain-git.txt

[Ramsay Jones]

On 03/10/2024 02:09, Taylor Blau wrote:
> Part of the maintainer's job is to keep up-to-date and publish the
> 'amlog' which stores a mapping between a patch's 'Message-Id' e-mail
> header and the commit generated by applying said patch.
> 
> But our Documentation/howto/maintain-git.txt does not mention the amlog,
> or the scripts which exist to help the maintainer keep the amlog
> up-to-date.
> 
> (This bit me during the first integration round I did as interim
> maintainer[1] involved a lot of manual clean-up. More recently it has
> come up as part of a research effort to better understand a patch's
> lifecycle on the list[2].)
> 
> Address this gap by briefly documenting the existence and purpose of the
> 'post-applypatch' hook in maintaining the amlog entries.
> 
> [1]: https://lore.kernel.org/git/Y19dnb2M+yObnftj@nand.local/
> [2]: https://lore.kernel.org/git/CAJoAoZ=4ARuH3aHGe5yC_Xcnou_c396q_ZienYPY7YnEzZcyEg@mail.gmail.com/
> 
> Suggested-by: Junio C Hamano <gitster@pobox.com>
> Helped-by: Junio C Hamano <gitster@pobox.com>
> Signed-off-by: Taylor Blau <me@ttaylorr.com>
> ---
>  Documentation/howto/maintain-git.txt | 44 ++++++++++++++++++++++++++++
>  1 file changed, 44 insertions(+)
> 
> diff --git a/Documentation/howto/maintain-git.txt b/Documentation/howto/maintain-git.txt
> index da31332f113..f52f32eda93 100644
> --- a/Documentation/howto/maintain-git.txt
> +++ b/Documentation/howto/maintain-git.txt
> @@ -122,6 +122,13 @@ Note that before v1.9.0 release, the version numbers used to be
>  structured slightly differently.  vX.Y.Z were feature releases while
>  vX.Y.Z.W were maintenance releases for vX.Y.Z.
>  
> +Because most of the lines of code in Git are written by individual
> +contributors, and contributions come in the form of e-mailed patches
> +published on the mailing list, the project maintains a mapping from
> +individual commits to the Message-Id of the e-mail that resulted in
> +the commit, to help tracking the origin of the changes. The notes
> +in "refs/notes/amlog" are used for this purpose, and are published
> +along with the broken-out branches to the maintainer's repository.
>  
>  A Typical Git Day
>  -----------------
> @@ -165,6 +172,43 @@ by doing the following:
>     In practice, almost no patch directly goes to 'master' or
>     'maint'.
>  
> +   Applying the e-mailed patches using "git am" automatically records
> +   the mappings from 'Message-Id' to the applied commit in the "amlog"
> +   notes. Periodically check that this is working with "git show -s
> +   --notes=amlog $commit".
> +
> +   This mapping is maintained with the aid of the "post-applypatch"
> +   hook found in the 'todo' branch. That hook should be installed
> +   before applying patches. It is also helpful to carry forward any
> +   relevant amlog entries when rebasing, so the following config may
> +   be useful:
> +
> +      [notes]
> +        rewriteRef = refs/notes/amlog
> +
> +   Avoid "cherry-pick", as it does not propagate notes by design. Use
> +   either "git commit --amend" or "git rebase" to make corrections to
> +   an existing commit, even for a single-patch topic.
> +
> +   Make sure that a push refspec for 'refs/notes/amlog' is in the
> +   remote configuration for publishing repositories. A few sample
> +   configurations look like the following:
> +
> +      [remote "github"]
> +        url = https://github.com/gitster/git
> +        pushurl = github.com:gitster/git.git
> +        mirror
> +
> +      [remote "github2"]
> +        url = https://github.com/git/git
> +        fetch = +refs/heads/*:refs/remotes/github2/*
> +        pushurl = github.com:git/git.git
> +        push = refs/heads/maint:refs/heads/maint
> +        push = refs/heads/master:refs/heads/master
> +        push = refs/notes/next:refs/notes/next

Hmm, s/notes/heads/g perhaps?

> +        push = +refs/heads/seen:refs/heads/seen
> +        push = +refs/notes/amlog
> +
>   - Review the last issue of "What's cooking" message, review the
>     topics ready for merging (topic->master and topic->maint).  Use
>     "Meta/cook -w" script (where Meta/ contains a checkout of the
> 

ATB,
Ramsay Jones

Re: [PATCH v3] Documentation: mention the amlog in howto/maintain-git.txt

[Taylor Blau]

On Thu, Oct 03, 2024 at 07:32:04PM +0100, Ramsay Jones wrote:
> > +      [remote "github2"]
> > +        url = https://github.com/git/git
> > +        fetch = +refs/heads/*:refs/remotes/github2/*
> > +        pushurl = github.com:git/git.git
> > +        push = refs/heads/maint:refs/heads/maint
> > +        push = refs/heads/master:refs/heads/master
> > +        push = refs/notes/next:refs/notes/next
>
> Hmm, s/notes/heads/g perhaps?

Ugh, serves me right for trying to send this out late my time last
night.

Perhaps Junio can tweak this when applying? Otherwise I can send out a
new version.

Thanks,
Taylor

Re: Linking topic merges to mailing list threads

[Kristoffer Haugsbakk]

Like Junio explained refs/notes/amlog is a hidden gem for patch
provenance.

> Hi all,
>
> We've been wanting to gather metrics on Git's code review process -
> how long it takes from first contact on list to merge, how many
> iterations are needed, time between iterations, etc. One missing link
> is the actual merge time in `next` and `master` - a human can infer
> the link between the patch and the mailing list thread, but it's more
> challenging for a script to do it.

Is the starting point the email?  I think you could fish out the
Message-ID and do a grep inside the notes tree

    git grep --fixed-string --name-only \
        '00a9fe6b7d77c16c9fd6dfe746aacf9068a76942.1726206484.git.ps@pks.im' \
        refs/notes/amlog --

(the resulting hash will need to be cleaned: fanned directory layout[1])

Then try one commit at a time (because there might be unreachable
commits from rewrite operations) using git-when-merged(1):[2]

    git when-merged --log 7cd8f1cc6e17af54fb78768c259a615b1ccc0205 next
    git when-merged --log 7cd8f1cc6e17af54fb78768c259a615b1ccc0205 master

† 1: e.g. 7c/d8/f1cc6e17af54fb78768c259a615b1ccc0205
🔗 2: https://github.com/mhagger/git-when-merged

-- 
Kristoffer Haugsbakk

Re: Linking topic merges to mailing list threads

[Eric Wong]

Emily Shaffer <nasamuffin@google.com> wrote:
> Hi all,
> 
> We've been wanting to gather metrics on Git's code review process -
> how long it takes from first contact on list to merge, how many
> iterations are needed, time between iterations, etc. One missing link
> is the actual merge time in `next` and `master` - a human can infer
> the link between the patch and the mailing list thread, but it's more
> challenging for a script to do it.

Searching by commit titles as a phrase against email subject (`s:')
can probably make it easy w/o having to look up amlog or explicitly
keep track of human-unmemorizable metadata such as Message-IDs.

Example:
https://80x24.org/lore/pub/scm/git/git.git/365529e1ea19b44a7a253b780f3ae3a1cb2f081f/s/#merged

...In the "find merged patch emails" textarea.  Yeah it's part of a
100% JS-free alternative to cgit, gitweb, etc...

(https://public-inbox.org/meta/20241002223902.4139389-4-e@80x24.org/
 implements the query generation)

Re: Linking topic merges to mailing list threads

[Jeff King]

On Wed, Oct 02, 2024 at 10:50:57PM +0000, Eric Wong wrote:

> Emily Shaffer <nasamuffin@google.com> wrote:
> > Hi all,
> > 
> > We've been wanting to gather metrics on Git's code review process -
> > how long it takes from first contact on list to merge, how many
> > iterations are needed, time between iterations, etc. One missing link
> > is the actual merge time in `next` and `master` - a human can infer
> > the link between the patch and the mailing list thread, but it's more
> > challenging for a script to do it.
> 
> Searching by commit titles as a phrase against email subject (`s:')
> can probably make it easy w/o having to look up amlog or explicitly
> keep track of human-unmemorizable metadata such as Message-IDs.

I do that a lot myself, but it sometimes get tripped up when people put
patches inline, like:

  > blah blah blah

  Yes, good idea. Maybe like this:

  -- >8 --
  foo: frobnicate the bar

  Etc...

In that case you have to do an actual body search. I usually find these
with phrase searches in the body text (I'm usually using notmuch, not
public-inbox, but I think the same would be true). Of course it may also
find more false positives, but usually they're from the same thread
anyway.

Very occasionally somebody posts a patch snippet and Junio writes a
commit message that never even hits the list, but that's pretty rare
these days. :)

-Peff