git-feed-mail-list.sh

git-feed-mail-list.sh

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 404 bytes --]

I've just updated the script which feeds the
git-commits-{head,24}@vger.kernel.org lists -- it now looks like this...

- Git-only; no longer uses cg-*
- No longer uses its own clone repo -- works directly from the original
- Handles MIME encoding of Subject: headers when necessary
  (although this is a bit icky in shell; especially _my_ shell).

Should I be using git-format-patch for this?

-- 
dwmw2

[-- Attachment #2: Type: application/x-shellscript, Size: 4204 bytes --]

Re: git-feed-mail-list.sh

[Junio C Hamano]

David Woodhouse <dwmw2@infradead.org> writes:

> # $FROM specifies the From: header used in the mails. It'll default
> # to GIT_COMMITTER_EMAIL if that exists, or to `whoami`@`hostname`

I am not sure if this part is tested..

> # Unless configured otherwise, just cat it instead of mailing.
> if [ -z "$FROM" ]; then
>     if [ -z "$GIT_COMMITTER_EMAIL" ]; then 
> 	FROM="$GIT_COMMITTER_EMAIL"
>     else
> 	FROM=`whoami`@`hostname`
>     fi
> fi

Maybe you meant 'if test -n "$GIT_COMMITTER_EMAIL"' here?

> # takes an object and generates the object's parent(s)
> createmail () {
>     local commit

If you were to do bashism local, don't you want to also localize
other variables like key, SUBHEX, NEWSUB,...?

It may make sense to enhance format-patch to do the Q encoding,
so that you do not have to do this part by hand...

> 	git-diff -B $parent $commit > $TMPCM
> 	diffstat -p1 $TMPCM 2>/dev/null

With GIT 1.3.0 and later:

	git diff --patch-with-stat $parent..$commit

would be simpler here.

> base=$(git-rev-parse $1)
>
> if [ -z $2 ]; then
>     lastmail=`cat $MAILTAG`
> else
>     lastmail=$(git-rev-parse $2)
> fi

lastmail=`git rev-parse --default "$MAILTAG" ${2+"$2"}`

> if [ -z $1 ]; then
>     base=$(git-rev-parse HEAD) || exit 1
> else
>     base=$(git-rev-parse $1) || exit 1
> fi

I am not sure if earlier base=$(git-rev-parse $1) is needed if
you do this here...

Re: git-feed-mail-list.sh

[Junio C Hamano]

Junio C Hamano <junkio@cox.net> writes:

> David Woodhouse <dwmw2@infradead.org> writes:
>
>> # $FROM specifies the From: header used in the mails. It'll default
>> # to GIT_COMMITTER_EMAIL if that exists, or to `whoami`@`hostname`
>
> I am not sure if this part is tested..
>
>> # Unless configured otherwise, just cat it instead of mailing.
>> if [ -z "$FROM" ]; then
>>     if [ -z "$GIT_COMMITTER_EMAIL" ]; then 
>> 	FROM="$GIT_COMMITTER_EMAIL"
>>     else
>> 	FROM=`whoami`@`hostname`
>>     fi
>> fi
>
> Maybe you meant 'if test -n "$GIT_COMMITTER_EMAIL"' here?
>
>> # takes an object and generates the object's parent(s)
>> createmail () {
>>     local commit
>
> If you were to do bashism local, don't you want to also localize
> other variables like key, SUBHEX, NEWSUB,...?
>
> It may make sense to enhance format-patch to do the Q encoding,
> so that you do not have to do this part by hand...
>
>> 	git-diff -B $parent $commit > $TMPCM
>> 	diffstat -p1 $TMPCM 2>/dev/null
>
> With GIT 1.3.0 and later:
>
> 	git diff --patch-with-stat $parent..$commit
>
> would be simpler here.

Or at least lose "diffstat -p1" and replace it with

	git-apply --stat --status

which would be more pleasant.

>> if [ -z $2 ]; then
>>     lastmail=`cat $MAILTAG`
>> else
>>     lastmail=$(git-rev-parse $2)
>> fi
>
> lastmail=`git rev-parse --default "$MAILTAG" ${2+"$2"}`

As I wrote it this is broken, sorry.

This assumes you stop doing "MAILTAG=.git/refs/tags/MailDone"
by hand and lose "do we have GIT_DIR" logic as well.
Instead define MAILTAG=tags/MailDone or maybe refs/tags/MailDone
and let "git rev-parse --default refs/tags/MailDone" figure out
what to do when GIT_DIR is set or unset.

Re: git-feed-mail-list.sh

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 870 bytes --]

On Wed, 2006-05-03 at 21:35 -0700, Junio C Hamano wrote:
> If you were to do bashism local, don't you want to also localize
> other variables like key, SUBHEX, NEWSUB,...?
> 
> It may make sense to enhance format-patch to do the Q encoding,
> so that you do not have to do this part by hand... 

Yes, that would be useful. We should perhaps to the From: and To:
headers too. Here's my current version (thanks for the feedback)...

The remaining problem is that the invocation of 'date' doesn't work with
new versions of coreutils. This...

   date=(${rest#*> })
   sec=${date[0]}; tz=${date[1]}
   dtz=${tz/+/+ }; dtz=${dtz/-/- }
   pdate="$(date -Rud "1970-01-01 UTC + $sec sec $dtz" 2>/dev/null)"

... doesn't work any more on FC-5, because:

 $ date -Rud '1970-01-01 UTC + 1147104611 sec + 0100'
date: invalid date `1970-01-01 UTC + 1147104611 sec + 0100'

-- 
dwmw2

[-- Attachment #2: git-feed-mail-list.sh --]
[-- Type: application/x-shellscript, Size: 3600 bytes --]

Re: git-feed-mail-list.sh

[Linus Torvalds]

On Tue, 9 May 2006, David Woodhouse wrote:
> 
> The remaining problem is that the invocation of 'date' doesn't work with
> new versions of coreutils. This...
> 
>    date=(${rest#*> })
>    sec=${date[0]}; tz=${date[1]}
>    dtz=${tz/+/+ }; dtz=${dtz/-/- }
>    pdate="$(date -Rud "1970-01-01 UTC + $sec sec $dtz" 2>/dev/null)"
> 
> ... doesn't work any more on FC-5, because:

Well, you might choose to just not use "git-cat-file commit" but instead 
ask git to format the thing for you.

Ie you could probably more easily parse the data from something like

	git show -B --patch-with-stat --pretty=fuller $commit

instead of using "git-cat-file commit $commit" and generating the stat and 
diff manually.

That way you get the dates etc pretty-printed for you by git.

			Linus

Re: git-feed-mail-list.sh

[David Woodhouse]

On Mon, 2006-05-08 at 17:19 -0700, Linus Torvalds wrote:
> Well, you might choose to just not use "git-cat-file commit" but instead 
> ask git to format the thing for you.
> 
> Ie you could probably more easily parse the data from something like
> 
> 	git show -B --patch-with-stat --pretty=fuller $commit
> 
> instead of using "git-cat-file commit $commit" and generating the stat and 
> diff manually.
> 
> That way you get the dates etc pretty-printed for you by git.

Aha, thanks. Git has learned to do a lot more since I first started
hacking up a copy of git-log.sh to feed the mailing lists, and it even
had to walk the commit tree manually :)

The output of (the undocumented) '--pretty=fuller' is probably good
enough that I can just feed the mailing list with it directly. I think I
have to add the commit and the parent manually, but that's easy enough
to do -- the commit is obviously known, and the parent is just
$(git-rev-parse $commit^1).

Anyone got any objections to switching the kernel git-commits-* lists to
this format?

-- 
dwmw2

Re: git-feed-mail-list.sh

[Junio C Hamano]

David Woodhouse <dwmw2@infradead.org> writes:

> ... I think I
> have to add the commit and the parent manually, but that's easy enough
> to do -- the commit is obviously known, and the parent is just
> $(git-rev-parse $commit^1).

If you are going to parse it anyway, this would probably be easier.

	$ git show --patch-with-stat --pretty=fuller --parents "$commit"

Why would you want -B, by the way?

Re: git-feed-mail-list.sh

[Bertrand Jacquin]

On 5/9/06, Junio C Hamano <junkio@cox.net> wrote:
>
> If you are going to parse it anyway, this would probably be easier.
>
>         $ git show --patch-with-stat --pretty=fuller --parents "$commit"
>
> Why would you want -B, by the way?

What is --parents for ?

Is there a way to have a --pretty=fuller with date append to Author
and Commiter name and email instead of have 1 lign after each ?

--
Beber
#e.fr@freenode

Re: git-feed-mail-list.sh

[Junio C Hamano]

"Bertrand Jacquin" <beber.mailing@gmail.com> writes:

> What is --parents for ?

In the part you did not quote, David wanetd a way to grab the
parents of the commit in question, so I gave it to him.

> Is there a way to have a --pretty=fuller with date append to Author
> and Commiter name and email instead of have 1 lign after each ?

sed is your friend.

Re: git-feed-mail-list.sh

[Bertrand Jacquin]

On 5/9/06, Junio C Hamano <junkio@cox.net> wrote:
> "Bertrand Jacquin" <beber.mailing@gmail.com> writes:
>
> > What is --parents for ?
>
> In the part you did not quote, David wanetd a way to grab the
> parents of the commit in question, so I gave it to him.

Ok, sorry. I read too fast.

--
Beber
#e.fr@freenode

Re: git-feed-mail-list.sh

[Linus Torvalds]

On Tue, 9 May 2006, David Woodhouse wrote:
> 
> Anyone got any objections to switching the kernel git-commits-* lists to
> this format?

As long as the "commit <sha1>" id is there (and "--pretty=fuller" does 
have it), I'll be happy. At some point, the commit mailing list didn't 
actually mention the commit ID itself, just the tree/parent IDs.

The "fuller" format should be fine, if you care about committer. Otherwise 
just use the standard "--pretty", which drops committer info.

		Linus

Re: git-feed-mail-list.sh

[David Woodhouse]

On Mon, 2006-05-08 at 17:45 -0700, Linus Torvalds wrote:
> As long as the "commit <sha1>" id is there (and "--pretty=fuller" does 
> have it), I'll be happy.

Ah, right. Those are _commit_ IDs in that strange first line. I'll
reformat those to 'Commit:' and 'Parent:' for the mailing list.

Having 'git-show --pretty=email' would be nice. I think Junio is working
on something which will achieve that, right?

>  At some point, the commit mailing list didn't 
> actually mention the commit ID itself, just the tree/parent IDs.

Yeah, I know -- and I got complaints :)

-- 
dwmw2

Re: git-feed-mail-list.sh

[Junio C Hamano]

David Woodhouse <dwmw2@infradead.org> writes:

> Having 'git-show --pretty=email' would be nice. I think Junio is working
> on something which will achieve that, right?

That's the replacement of "git format-patch".  If you have a
chance, please try out what's in "next".  Johannes did quite a
nice enhancements.

Re: git-feed-mail-list.sh

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 670 bytes --]

On Mon, 2006-05-08 at 18:05 -0700, Junio C Hamano wrote:
> That's the replacement of "git format-patch".  If you have a
> chance, please try out what's in "next".  Johannes did quite a
> nice enhancements. 

I think I'd best wait for it to turn up in the release; preferably
already capable of MIME quoting. I don't like touching the scripts which
feed the mailing lists :)

This is what I currently have... since it changes the format I think
I'll won't deploy it yet -- I'll wait until the "replacement of
git-format-patch" is done, in case that would change the format _again_.
I'd rather not change the format I send to the list twice within a few
weeks.

-- 
dwmw2

[-- Attachment #2: git-feed-mail-list.sh --]
[-- Type: application/x-shellscript, Size: 2999 bytes --]

Re: git-feed-mail-list.sh

[Martin Mares]

Hello!

> I think I'd best wait for it to turn up in the release; preferably
> already capable of MIME quoting.

Wouldn't it be easier to feed the output to a MUA and letting it handle
the MIME stuff for you?

I am using mutt for this purpose:

mutt -x -e 'set charset="utf-8"; set send_charset="us-ascii:iso-8859-2:utf-8"' -s "$subj" "$recipient" <$out

				Have a nice fortnight
-- 
Martin `MJ' Mares   <mj@ucw.cz>   http://atrey.karlin.mff.cuni.cz/~mj/
Faculty of Math and Physics, Charles University, Prague, Czech Rep., Earth
Next lecture on time travel will be held on previous Monday.

Re: git-feed-mail-list.sh

[Linus Torvalds]

On Tue, 9 May 2006, David Woodhouse wrote:
> 
> Ah, right. Those are _commit_ IDs in that strange first line. I'll
> reformat those to 'Commit:' and 'Parent:' for the mailing list.

Right. That first line (that starts with "commit") lists the commit ID, 
and if you say "--parents", the commit ID's of the parents will be 
appended.

So if you want to turn that into "Commit: <id>" and "Parent: <id>", you'll 
want to do something like this:

  git show --no-abbrev -C --patch-with-stat --pretty=fuller --parents $commit |
	sed '1 s/commit \([0-9a-f]*\)/Commit:     \1\nParent:    /'

which should look pretty (count the spaces to make sure it lines up 
right with the other fields).

(And if you ever want to report on merges, you'll want to change that a 
bit, but it should be reasonably close to the above)

		Linus

Re: git-feed-mail-list.sh

[Bertrand Jacquin]

On 5/9/06, Linus Torvalds <torvalds@osdl.org> wrote:
>
> Ie you could probably more easily parse the data from something like
>
>         git show -B --patch-with-stat --pretty=fuller $commit
>

Is there a way to track merge like that ? Documentation is not very
clear and near from empty.

--
Beber
#e.fr@freenode

Re: git-feed-mail-list.sh

[Junio C Hamano]

"Bertrand Jacquin" <beber.mailing@gmail.com> writes:

> Is there a way to track merge like that?

The command line you quoted shows the relevant information for
people who want to know what happened in that merge.

Namely:

 * it always shows the header and the message

 * it shows the changes that are not trivial (i.e. merge parents
   have overlapping different versions and manual resolution
   resulted in something different from either parents).

It is not a replacement for format-patch, but I think the commit
mailing list is not for machines to receive and apply the
received patches, but for humans to inspect, so it would be more
suitable than a naive alternative of showing diff from all
parents concatenated together.

Re: git-feed-mail-list.sh

[Bertrand Jacquin]

On 5/9/06, Junio C Hamano <junkio@cox.net> wrote:
> "Bertrand Jacquin" <beber.mailing@gmail.com> writes:
>
> > Is there a way to track merge like that?
>
> The command line you quoted shows the relevant information for
> people who want to know what happened in that merge.
>
> Namely:
>
>  * it always shows the header and the message
>
>  * it shows the changes that are not trivial (i.e. merge parents
>    have overlapping different versions and manual resolution
>    resulted in something different from either parents).
>
> It is not a replacement for format-patch, but I think the commit
> mailing list is not for machines to receive and apply the
> received patches, but for humans to inspect, so it would be more
> suitable than a naive alternative of showing diff from all
> parents concatenated together.

That's right. And don't want to do that.

But I would like to send an email after merge to inform people that:

o tree ``a'' and ``b'' have been merged.
o made by John Doe at a time
o show a diffstat.
o show a --short-log=oneline from merge base.

--
Beber
#e.fr@freenode

Re: git-feed-mail-list.sh

[Junio C Hamano]

"Bertrand Jacquin" <beber.mailing@gmail.com> writes:

> But I would like to send an email after merge to inform people that:
>
> o tree ``a'' and ``b'' have been merged.
> o made by John Doe at a time
> o show a diffstat.
> o show a --short-log=oneline from merge base.

Forgetting about a fast-forward merge,  (1) and (2) are
available in the commit header and the commit log, so is (4) if
you enable merge.summary configuration like Linus does in his
kernel repository.

The comment on diffstat Linus already made applies to (3), but
if you want you could do "git diff --stat HEAD^..HEAD" to see
what happened to that branch by merging the other branch into
it.

Re: git-feed-mail-list.sh

[Linus Torvalds]

On Mon, 8 May 2006, Junio C Hamano wrote:

> "Bertrand Jacquin" <beber.mailing@gmail.com> writes:
> 
> > But I would like to send an email after merge to inform people that:
> >
> > o tree ``a'' and ``b'' have been merged.
> > o made by John Doe at a time
> > o show a diffstat.
> > o show a --short-log=oneline from merge base.
> 
> Forgetting about a fast-forward merge,  (1) and (2) are
> available in the commit header and the commit log, so is (4) if
> you enable merge.summary configuration like Linus does in his
> kernel repository.

NOTE! Please don't enable "merge.summary" if you ever merge from the 
upstream tree. That just looks ugly. Your merge messages will be just 
filled with crap that has nothing to do with your tree - and everything to 
do with all the _unrelated_ normal development that happened in the tree.

So in general, "merge.summary" makes sense only for trees that pull from 
downstreams, and never merge with anything upstream. My tree obviously 
does that for the kernel. Think of it as a "top-level maintainer" flag, 
although it works find also for sub-maintainers as long as they 
synchronize upwards _purely_ by being pulled from, not by pulling.

But if you want to get it for any random merges, you can always just do

	git log -11 --pretty=oneline ^$commit^ $commit^@ |
		sed 's/[0-9a-f]* // ; 11 s/.*/\.\.\./' 

which will show up to the ten first commits that were merged (and turn the 
eleventh one, if it exists, into "..." - that's a pretty disgusting trick 
to make it show when you left things out).

That "^$commit^ $commit^@" part is important. It may look like some 
deranged git smiley, but it does exactly what you want it to do: take all 
the parents of the commit, but ignore any commit reachable from the first 
one (the "mainline" of the person who did the commit).

The ^@ syntax is obviously pretty new, so it requires a modern git.

		Linus

Re: git-feed-mail-list.sh

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> But if you want to get it for any random merges, you can always just do
>
> 	git log -11 --pretty=oneline ^$commit^ $commit^@ |
> 		sed 's/[0-9a-f]* // ; 11 s/.*/\.\.\./' 
>
> which will show up to the ten first commits that were merged (and turn the 
> eleventh one, if it exists, into "..." - that's a pretty disgusting trick 
> to make it show when you left things out).
>
> That "^$commit^ $commit^@" part is important. It may look like some 
> deranged git smiley, but it does exactly what you want it to do: take all 
> the parents of the commit, but ignore any commit reachable from the first 
> one (the "mainline" of the person who did the commit).
>
> The ^@ syntax is obviously pretty new, so it requires a modern git.

It is indeed very quite new.  Merged into "master" branch at the
beginning of this month.

I often wish we had a straightforward way to tell when a given
feature went into the mainline, not just appeared on a topic
branch.  In this case, I said:

	$ git whatchanged -p -S'"^@"' master -- revision.c

to find ea4a19 commit (Apr 30 00:54:29 2006 -0700).  But that
was when the feature was first made on one of my topic branches,
which is not what I was looking for.

By looking at gitk, I can then tell 83262e (May 1 01:54:27)
merged it to "next", and 746437 (May 1 22:55:40) merged it to
"master".

In general this is an unsolvable question, because I can have a
topic branch forked off of the tip of "master", cook it for a
few days without advancing "master" at all, and merge it to
"master" after that.  But such a merge will be a fast-forward.

Re: git-feed-mail-list.sh

[Linus Torvalds]

On Tue, 9 May 2006, Bertrand Jacquin wrote:

> On 5/9/06, Linus Torvalds <torvalds@osdl.org> wrote:
> > 
> > Ie you could probably more easily parse the data from something like
> > 
> >         git show -B --patch-with-stat --pretty=fuller $commit
> > 
> 
> Is there a way to track merge like that ? Documentation is not very
> clear and near from empty.

Sure.

If you want to track merges and get their patches, add the "--cc" flag, 
which tells git to use the "conflict combination patch" that shows any 
visible conflicts.

(NOTE NOTE NOTE! This is _not_ the same as showing what conflicted: if you 
edited the result to match one of the original branches, it will be quiet 
in --cc, but if the result of a conflict was something that was in 
_neither_ branch, it will be shown! So most clean merges will not show any 
conflict diff at all, but the diffstat will be shown against the "first 
parent").

And you probably don't want to abbreviate the parent commit SHA1's (which 
are shown for merges, but not regular commits), so add "--no-abbrev".

If you want to show parents for _all_ commits, you could do something like

  git show --no-abbrev --cc -C --patch-with-stat --pretty=fuller --parents |
	sed '1 s/commit [0-9a-f]*/\0\nParents:   / ; /^Merge: / d'

which removes a potential "Merge: " line in favour of listing the parents 
on a "Parents:" line, and which also shows merges nicely.

That said, the diffstat for merges is usually just a lot of noise. It's 
sometimes nice (you've merged from a topic branch), but if you have merged 
from the mainline _into_ a topic branch, it's just annoying.

So the above is just a wild suggestion. Caveat emptor.

		Linus

Re: git-feed-mail-list.sh

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> On Tue, 9 May 2006, Bertrand Jacquin wrote:
>
>> On 5/9/06, Linus Torvalds <torvalds@osdl.org> wrote:
>> > 
>> > Ie you could probably more easily parse the data from something like
>> > 
>> >         git show -B --patch-with-stat --pretty=fuller $commit
>> 
>> Is there a way to track merge like that ? Documentation is not very
>> clear and near from empty.
>
> Sure.
>
> If you want to track merges and get their patches, add the "--cc" flag, 
> which tells git to use the "conflict combination patch" that shows any 
> visible conflicts.

Actually, show defaults to --cc so what's shown is good as is.

> That said, the diffstat for merges is usually just a lot of noise. It's 
> sometimes nice (you've merged from a topic branch), but if you have merged 
> from the mainline _into_ a topic branch, it's just annoying.

True.  We made --cc --patch-with-stat to do the combined patch
text with diffstat for first-parent-diff to make it most natural
for merging into upstream, not merging from upstream.