[RFC/RFH] Fun things with git-notes, or: patch tracking backwards

[RFC/RFH] Fun things with git-notes, or: patch tracking backwards

[Thomas Rast]

[-- Attachment #1: Type: text/plain, Size: 2280 bytes --]

Hi everyone

I'll start with the fun first, try this in a git.git:

  git fetch git://repo.or.cz/git/trast.git mailnotes &&
  GIT_NOTES_REF=FETCH_HEAD git log origin/pu

I played around with some python code over the weekend that
automatically filters through git@vger.kernel.org history and scans
for patches, threads, and "What's cooking" messages.  So far it seems
to be working ok.

The net effect is that we get a backwards patch tracker: instead of
tracking the patches via some other means (say a web interface), the
automatic annotations can reconstruct more information about the
patches than what is eventually contained in the commits, and make it
available via git-notes.

Right now it only applies the patches that it finds, and associates
that with known commits to annotate them.  I eventually want to scan
for replies to patch text too, and insert them into the notes too.  I
also plan to publish topics for the patches somewhere (they're already
applied locally) and track at least a rudimentary status, perhaps one
of "unreplied", "replied" and "accepted".  (Distinguishing "rejected"
seems AI-complete.)

The whole process is a bunch of python scripts currently available at

  git://repo.or.cz/trackgit.git
  http://repo.or.cz/w/trackgit.git

It's all a bit after-the-fact right now, and can't cope with a few
things yet, for example patch series that aren't in git.git appearing
out of sequence to the mail reader.

Runtime is okay-ish so that I should be able to run it as a cronjob;
note regeneration is almost negligible (<1min), importing a month's
worth of mails takes on the order of 20min, and scanning history (to
know about the base blobs) since v1.6.0 is around 3min.

So the RFC bit is, is this useful to anyone?  What information would
you like to see in it, and what could be left out?

And the RFH: I don't have a full mail archive, not even since I joined
the list.  There also doesn't seem to be a convenient download button
on gmane.  Does anyone have (or know of) an archive going at least
back to v1.6.0 (not sure if any further back is interesting), which
was released in August last year, that you could send to me?

Thanks in advance :-)

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards

[SZEDER Gábor]

Hi Thomas,

On Mon, Feb 09, 2009 at 03:08:08PM +0100, Thomas Rast wrote:
> And the RFH: I don't have a full mail archive, not even since I joined
> the list.  There also doesn't seem to be a convenient download button
> on gmane.

you can download emails in mbox format from gmane by running

  wget http://download.gmane.org/gmane.comp.version-control.git/X/Y

which will download all emails starting at the email with "gmane id" X
and ending at the email with "gmane id" Y-1.

So, if you want to download the whole archive, you could run

  wget http://download.gmane.org/gmane.comp.version-control.git/1/109090

(but of course that 109090 will be larger by the time you will reading
this).  However, there's a catch, as gmane's script execution time is
limited to 30 sec, so you will not get the whole archive, but only the
first couple of thousands emails (in my case and network I got around
6k emails in one run).  Therefore you'll need some loop to download
only a few thousand emails in each iteration.


Best,
Gábor

Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards

[Thomas Rast]

[-- Attachment #1: Type: text/plain, Size: 367 bytes --]

SZEDER Gábor wrote:
> you can download emails in mbox format from gmane by running
> 
>   wget http://download.gmane.org/gmane.comp.version-control.git/X/Y
> 
> which will download all emails starting at the email with "gmane id" X
> and ending at the email with "gmane id" Y-1.

Thanks, that will do nicely!

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: tool and worktree

[Giuseppe Bilotta]

On Sunday 08 February 2009 20:25, Shawn O. Pearce wrote:
> bill lam <cbill.lam@gmail.com> wrote:
>> I track /etc using a config
>> 
>> [core]
>> 	repositoryformatversion = 0
>> 	filemode = true
>> 	bare = false
>> 	worktree = /etc
>> 	logAllRefUpdates = true
>> 	excludesfile =
>> 
>> But that can not be handled by tools,
>> 
>> git gui : cannot use funny .git directory .
> 
> If someone sends patches for git-gui, maybe.  This use case of
> different repository and worktree isn't very common for git-gui
> so it doesn't support it.

I came across the same problem. I also sent a tentative patchset to
solve the problem about 12 hours ago:

http://thread.gmane.org/gmane.comp.version-control.git/109035

-- 
Giuseppe "Oblomov" Bilotta

Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards

[Thomas Rast]

[-- Attachment #1: Type: text/plain, Size: 785 bytes --]

Thomas Rast wrote:
>   git fetch git://repo.or.cz/git/trast.git mailnotes &&
>   GIT_NOTES_REF=FETCH_HEAD git log origin/pu

An update: I have fully automated the process, it now fetches mails
from Gmane over HTTP which gives it the Gmane URLs for free.  I'm
rather happy with the latter feature, especially since Konsole has a
feature to recognize and open links directly.

I have imported all commits, and mails since roughly July 2008
(starting with Gmane 89000).  In this timeframe there were 1802
non-merge commits, and the mailnotes tree now holds 1122 annotations.
I won't import mails any further back until the parsing-related code
has become reasonably stable, but this at least covers the post-v1.6.0
commits.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards

[Junio C Hamano]

Thomas Rast <trast@student.ethz.ch> writes:

> Thomas Rast wrote:
>>   git fetch git://repo.or.cz/git/trast.git mailnotes &&
>>   GIT_NOTES_REF=FETCH_HEAD git log origin/pu
>
> An update: I have fully automated the process, it now fetches mails
> from Gmane over HTTP which gives it the Gmane URLs for free.  I'm
> rather happy with the latter feature, especially since Konsole has a
> feature to recognize and open links directly.
>
> I have imported all commits, and mails since roughly July 2008
> (starting with Gmane 89000).  In this timeframe there were 1802
> non-merge commits, and the mailnotes tree now holds 1122 annotations.

Wonderful.

Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards

[Junio C Hamano]

Thomas Rast <trast@student.ethz.ch> writes:

> I have imported all commits, and mails since roughly July 2008
> (starting with Gmane 89000).  In this timeframe there were 1802
> non-merge commits, and the mailnotes tree now holds 1122 annotations.

How do you match the mails to commits?

I am curious what the right balance for the matching algorithm should be,
between being forgiving about amending of commit log message and the patch
text to fix minor typos and obvious bugs, and being strict not to cause
false matches to a message that contains the second iteration of the
patch, when what was committed was the first iteration.

Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards

[Thomas Rast]

[-- Attachment #1: Type: text/plain, Size: 880 bytes --]

Junio C Hamano wrote:
> Thomas Rast <trast@student.ethz.ch> writes:
> 
> > I have imported all commits, and mails since roughly July 2008
> > (starting with Gmane 89000).  In this timeframe there were 1802
> > non-merge commits, and the mailnotes tree now holds 1122 annotations.
> 
> How do you match the mails to commits?
> 
> I am curious what the right balance for the matching algorithm should be,
> between being forgiving about amending of commit log message and the patch
> text to fix minor typos and obvious bugs, and being strict not to cause
> false matches to a message that contains the second iteration of the
> patch, when what was committed was the first iteration.

Right now it's just the patch-id.  Maybe filtering (author,subject)
and then picking the one that is the most similar could work.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards

[Junio C Hamano]

Thomas Rast <trast@student.ethz.ch> writes:

> Junio C Hamano wrote:
>> Thomas Rast <trast@student.ethz.ch> writes:
>> 
>> > I have imported all commits, and mails since roughly July 2008
>> > (starting with Gmane 89000).  In this timeframe there were 1802
>> > non-merge commits, and the mailnotes tree now holds 1122 annotations.
>> 
>> How do you match the mails to commits?
>> 
>> I am curious what the right balance for the matching algorithm should be,
>> between being forgiving about amending of commit log message and the patch
>> text to fix minor typos and obvious bugs, and being strict not to cause
>> false matches to a message that contains the second iteration of the
>> patch, when what was committed was the first iteration.
>
> Right now it's just the patch-id.  Maybe filtering (author,subject)
> and then picking the one that is the most similar could work.

Yeah, I actually was thinking about matching the (date, author) tuple and
nothing else, as it is unlikely you would have dups.

By the way, the note to f6b98e4 (git-web--browse: Fix check for
/bin/start, 2009-02-08) is interesting.

Ramsay's 104332 was the same as what ended up in the commit, but his
second patch that can be found by going to the thread from it is obviously
a better alternative.  In short, I screwed up, by not recalling the
previous round.  Sorry.

I find the "Extra-Notes:" tag a bit too loud, but I am probably a minority
who thinks everything but the Message-ID can be dropped, so please don't
take it as a feature request ;-)

Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards

[Thomas Rast]

[-- Attachment #1: Type: text/plain, Size: 1645 bytes --]

Junio C Hamano wrote:
> Thomas Rast <trast@student.ethz.ch> writes:
> > Right now it's just the patch-id.  Maybe filtering (author,subject)
> > and then picking the one that is the most similar could work.
> 
> Yeah, I actually was thinking about matching the (date, author) tuple and
> nothing else, as it is unlikely you would have dups.

Thanks, good idea.  I changed the code to parse the required data, and
we're now up to 1502 annotations.

Unfortunately I noticed there's a bug in the mail input stage:
Python's mailbox module assumes any '^From ' line starts a new mail,
while gmane apparently uses a slightly different format based on the
double newlines too (and they always have the same 'From
news@gmane.org Tue Mar 04 03:33:20 2003' separator too), and doesn't
quote '^From ' in the bodies.  So any mail containing such body lines
got chopped down in the middle, and any patches contained in them
won't apply because of the missing headers.

A quick perl run shows that there are 26 mails affected among the
89000+ mails that I've (again) imported.  The fix should be easy, but
I'm already short on sleep.

> I find the "Extra-Notes:" tag a bit too loud, but I am probably a minority
> who thinks everything but the Message-ID can be dropped, so please don't
> take it as a feature request ;-)

I refactored the final formatting stage a bit to let it do several
notes trees, and you can now take your pick:

  git://repo.or.cz/trackgit.git notes/full
  git://repo.or.cz/trackgit.git notes/terse

The latter only has 'Message-Id' and 'Archived-At'.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards

[Thomas Rast]

[-- Attachment #1: Type: text/plain, Size: 355 bytes --]

Thomas Rast wrote:
> I'm already short on sleep.
[...]
>   git://repo.or.cz/trackgit.git notes/full
>   git://repo.or.cz/trackgit.git notes/terse

Q.E.D.!  The correct repository is

  git://repo.or.cz/git/trast.git notes/full
  git://repo.or.cz/git/trast.git notes/terse

Sorry for the noise.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

tool and worktree

[bill lam]

I track /etc using a config

[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	worktree = /etc
	logAllRefUpdates = true
	excludesfile =

But that can not be handled by tools,

git gui : cannot use funny .git directory .
tig : status view said, The status view requires a working tree

Can these be improved?
 
-- 
regards,
====================================================
GPG key 1024D/4434BAB3 2008-08-24
gpg --keyserver subkeys.pgp.net --recv-keys 4434BAB3
唐詩314 王昌齡  長信怨
    奉帚平明金殿開  且將團扇共徘徊  玉顏不及寒鴉色  猶帶昭陽日影來

Re: tool and worktree

[Jonas Fonseca]

On Sun, Feb 8, 2009 at 04:44, bill lam <cbill.lam@gmail.com> wrote:
> I track /etc using a config
>
> [core]
>        repositoryformatversion = 0
>        filemode = true
>        bare = false
>        worktree = /etc
>        logAllRefUpdates = true
>        excludesfile =
>
> But that can not be handled by tools,
>
> tig : status view said, The status view requires a working tree

Yes, on startup, tig checks git-rev-parse --is-inside-work-tree and
disables things like the status view if it is false. I don't know how
much work it would be to start using core.worktree. It might be as
simple as setting the GIT_DIR environment to getcwd() + output of
git-rev-parse --git-dir and then chdir() to core.worktree.

-- 
Jonas Fonseca

Re: tool and worktree

[Jonas Fonseca]

On Sun, Feb 8, 2009 at 10:47, Jonas Fonseca <jonas.fonseca@gmail.com> wrote:
> It might be as
> simple as setting the GIT_DIR environment to getcwd() + output of
> git-rev-parse --git-dir and then chdir() to core.worktree.

I have pushed a lightly tested patch that does something like this ...

-- 
Jonas Fonseca

Re: tool and worktree

[bill lam]

On Mon, 09 Feb 2009, Jonas Fonseca wrote:
> On Sun, Feb 8, 2009 at 10:47, Jonas Fonseca <jonas.fonseca@gmail.com> wrote:
> > It might be as
> > simple as setting the GIT_DIR environment to getcwd() + output of
> > git-rev-parse --git-dir and then chdir() to core.worktree.
> 
> I have pushed a lightly tested patch that does something like this ...

Thanks. Now the status view works.

Since /etc contains both public and private files.  When browsing the
untracked files in the status view, for files that without read
permission, eg the shadow passwd file, the lower pane displays the
content of the last readable file.  Could it clear the content or just
filter and display only files with read permission for the untracked
files?

-- 
regards,
====================================================
GPG key 1024D/4434BAB3 2008-08-24
gpg --keyserver subkeys.pgp.net --recv-keys 4434BAB3
唐詩215 李商隱  無題
    相見時難別亦難  東風無力百花殘  春蠶到死絲方盡  蠟炬成灰淚始乾
    曉鏡但愁雲鬢改  夜吟應覺月光寒  蓬萊此去無多路  青鳥殷勤為探看

Re: tool and worktree

[Jonas Fonseca]

2009/2/9 bill lam <cbill.lam@gmail.com>:
> Since /etc contains both public and private files.  When browsing the
> untracked files in the status view, for files that without read
> permission, eg the shadow passwd file, the lower pane displays the
> content of the last readable file.  Could it clear the content or just
> filter and display only files with read permission for the untracked
> files?

It now closes the lower pane and displays the reason why it failed to
load the file using strerror().

-- 
Jonas Fonseca

Re: tool and worktree

[Shawn O. Pearce]

bill lam <cbill.lam@gmail.com> wrote:
> I track /etc using a config
> 
> [core]
> 	repositoryformatversion = 0
> 	filemode = true
> 	bare = false
> 	worktree = /etc
> 	logAllRefUpdates = true
> 	excludesfile =
> 
> But that can not be handled by tools,
> 
> git gui : cannot use funny .git directory .

If someone sends patches for git-gui, maybe.  This use case of
different repository and worktree isn't very common for git-gui
so it doesn't support it.

On Windows with some versions of Tcl/Tk I've had trouble in the
past with Git finding the repository when forked out of the wish
process.  To work around it I just required that the repository
be in ".git".

Have you tried making /etc/.git a symlink to the real repository?

BTW, tracking /etc/ in git won't track permissions and ownership
so restoring /etc from git isn't enough on its own.  Usually I
use some sort of shell script to copy /etc to another location
and check it into git, and a different script to go the other way,
with that script handling mode and ownership as it goes.

-- 
Shawn.