Storing Maintainers info around the kernel tree

Storing Maintainers info around the kernel tree

[Kyle Moffett]

[-- Attachment #1: Type: text/plain, Size: 4286 bytes --]

Merging a couple related threads here:

On Aug 16, 2007, at 07:57:23, Rene Herman wrote:
> On 08/16/2007 01:26 PM, Salikh Zakirov wrote:
>> Rene Herman wrote:
>>> Perhaps that immediately suggests an implementation to someone  
>>> already familiar with git internals?
>> perhaps http://www.kernel.org/pub/software/scm/git/docs/ 
>> gitattributes.html and http://www.kernel.org/pub/software/scm/git/ 
>> docs/git-check-attr.html can help you?
>
> No, thanks, saw them, but .gitattributes is in fact in the same  
> category as .gitignore, which would _be_ a property.
>
> If you do this stuff in files scattered around the tree, updating  
> and moving stuff becomes a pain -- the tool would need to go edit  
> files.

 From a practical standpoint we don't want to duplicate someone's  
maintainer information in the attributes of every file they  
maintain.  It would be much easier to put in the "kernel/somesubsys"  
directory a Maintainers file which has:

[SOME RANDOM SUBSYSTEM]
P: J. Random Hacker
M: j.random.hacker@localhost
L: random-subsys-devel@vger.kernel.org
F: *

Anywhere else you had files that you wanted to associate with J.  
Random Hacker's maintainership, you would just use:

[SOME RANDOM SUBSYSTEM]
F: somesubsys.h


I posted a comment describing a mechanism like this a couple days ago:
   http://lkml.org/lkml/2007/8/14/488

Executive overview:
On Aug 15, 2007, at 07:21:04, Rafael J. Wysocki wrote:
> On Wednesday, 15 August 2007 04:51, Kyle Moffett wrote:
>> (a) "Maintainers" files sprinkled around the source tree with  
>> relative pathnames and other data
>>
>> (b) Tool to generate a combined "MAINTAINERS" file from the ones  
>> sprinkled around the source tree
>>
>> (c) Tool to search through the generated "MAINTAINERS" file with  
>> all sorts of useful command-line options
>>
>> (d) Tool to check the generated "MAINTAINERS" file against recent  
>> git history and make suggestions
>
> I like this idea. :-)

Well, to back up this idea with some code, I'm attaching a little  
perl script which does part (b).  Basically you call it as:
   ./maint-combine $(find . -name Maintainers)

It will print any syntax errors on stderr during parsing.  Once it's  
done it will dump to stdout its combined "MAINTAINERS" text.  A  
couple notes:

*  It uses a little "priority" system to figure out what order to  
print the data from each origin in.  For example, the "F:" tag is  
given a score of 0, to force data consisting of just files towards  
the end.  The "P:", "M:", and "L:" tags are given scores of 5, since  
people are generally interesting to know about.  Everything else is  
given a score of "1".  The scores are added up per ($file,  
$subsystem) pair and then during printing each subsystem's data is  
ordered by score (highest comes first).

*  It generally allows any field at all; eventually we might want to  
limit it to a fixed list to help avoid typos.

*  It has a little bit of magic logic for the "F:" field so that it  
figures out the relative directory for each field when generating the  
output.  For example, an entry of "asm-*/suspend.h" in a file  
"include/Maintainers" will produce the output file entry: "F: include/ 
asm-*/suspend.h"

*  The format isn't quite the same as the current MAINTAINERS file,  
to make parsing easier and more dummy-proof I changed the syntax for  
a subsystem-name to use square brackets (IE: "[SUSPEND TO RAM]").   
The samples I gave in my previous email are what I used to test it  
with, plus a little dummy file with some syntax errors to check out  
the error messages:

Maintainers:
> [EVERYTHING ELSE]
> P: Various Linux Kernel Developers
> L: linux-kernel@vger.kernel.org
> F: *

kernel/power/Maintainers:
> [SUSPEND TO RAM]
> P: Pavel Machek
> M: pavel@suse.cz
> P: Rafael J. Wysocki
> M: rjw@sisk.pl
> L: linux-pm@lists.linux-foundation.org
> S: Maintained
> F: *

include/Maintainers:
> [SUSPEND TO RAM]
> F: linux/suspend.h
> F: linux/freezer.h
> F: linux/pm.h
> F: asm-*/suspend.h



If you have any other questions, the perl script is pretty self- 
explanatory and I'll be completely back online this weekend.  With  
any luck I'll have some time in a hotel tomorrow (mmm, slow-as-dirt  
hotel wireless, what fun) to work on parts (c) and (d).

Cheers,
Kyle Moffett


[-- Attachment #2: maint-combine.txt --]
[-- Type: text/plain, Size: 2040 bytes --]

#! /usr/bin/perl

use strict;
use warnings;

## This table is for determining what order to show entries from different
## source "Maintainers" files.  They are prioritized based on the number and
## significance of each field, as listed in this table.  Any field not found
## here will be assumed to have a significance of "1".
my %fieldprio = (
	p => 5,  ## Person
	m => 5,  ## Email
	l => 5,  ## Mailing List
	f => 0,  ## File
);


my %mprio;
my %maint;
my $section;
my $origin;
my $pathprefix;
my $printorigin;

while (<>) {
	unless (defined $origin and $origin eq $ARGV) {
		undef $section;

		## Get rid of extra slashes, unnecessary "./" entries
		$ARGV =~ s{//+}{/}g;
		$ARGV =~ s{(^|/)\./}{$1};

		$origin = $ARGV;
	}

	chomp; s/\s+/ /g; s/^ //; s/ $//;
	/\S+/ or next;

	if (/^\[ ?([^]]+) ?\]$/) {
		$section = uc $1;

		$printorigin = 1 if $origin ne '-';

		$maint{$section}{$origin} ||= [];
		$mprio{$section}{$origin} ||= 0;

		## Figure out a useful path prefix if there is one
		$pathprefix = ($origin =~ m{(^.*/)(?!\.\.?$)[^/]+$})?$1:"";

		next;
	}

	unless (/^([^:]+?) ?: ?(.*)$/) {
		print STDERR "$ARGV:$.: Invalid line: $_\n";
		next;
	}

	my($field,$value) = (lc($1), $2);

	unless (defined $section) {
		print STDERR "$ARGV:$.: Found field '\u$field' before ",
				"any subsystem declaration: $_\n";
		next;
	}

	## Preprocess paths to make them absolute
	$value = $pathprefix.$value if $field eq 'f';

	## Update the sorting order
	if (exists $fieldprio{$field}) {
		$mprio{$section}{$origin} += $fieldprio{$field};
	} else {
		$mprio{$section}{$origin}++;
	}

	push @{$maint{$section}{$origin}}, [$field, $value];
}

sub priosort ( $@ )
{
	my $section = shift;
	return sort {$mprio{$section}{$b} <=> $mprio{$section}{$a}} @_;
}

for $section (sort {$a cmp $b} keys %maint) {
	print "[$section]\n";
	for my $origin (priosort $section, keys %{$maint{$section}}) {
		print "\nOrigin: $origin\n" if $printorigin;
		print "\u$_->[0]: $_->[1]\n" for @{$maint{$section}{$origin}};
	}
	print "\n\n";
}

# vim:set ft=perl:

Re: Storing Maintainers info around the kernel tree

[Rene Herman]

On 08/16/2007 03:04 PM, Kyle Moffett wrote:

> On Aug 16, 2007, at 07:57:23, Rene Herman wrote:

>> category as .gitignore, which would _be_ a property.
>>
>> If you do this stuff in files scattered around the tree, updating and 
>> moving stuff becomes a pain -- the tool would need to go edit files.
> 
> From a practical standpoint we don't want to duplicate someone's 
> maintainer information in the attributes of every file they maintain.  

In a tree structure, you don't have to. As described earlier, the tool 
(git-prop) looks for the requested property being set first on the file 
itself, then on the directory in which it resides, then its parent, and so 
on. If I read things right, this is also how properties work in subversion 
in fact.

So after

$ git prop --set --name maintainer --value \
	"Bartlomiej Zolnierkiewicz <bzolnier@gmail.com>" drivers/ide/
and

$ git prop --set --name maintainer --value \
	"Alan Cox <alan@lxorguk.ukuu.org.uk>" drivers/ide/ide-cd.*

we get:

$ git prop --get --name maintainer drivers/ide/ide-cd.c
Alan Cox <alan@lxorguk.ukuu.org.uk>

$ git prop --get --name maintainer drivers/ide/ide-generic.c
Bartlomiej Zolnierkiewicz <bzolnier@gmail.com>

Now, this override behaviour needs a tree structure ofcourse, but notice I 
set the "maintainer" property only to the name/address. The other 
information from the MAINTAINERS file would be using their own properties:

$ git prop --set --name tree --value \
     "quilt kernel.org/pub/linux/kernel/people/bart/pata-2.6/" drivers/ide/

and nothing under drivers/ide/ would override this value nor would it be 
repeated anywhere. Alan takes care of more than ide-cd but only the actual 
"maintainer" value string would be set on the others as well and repeating 
that much for different "maintenance units" is no different from the current 
MAINTAINERS file where it also is (well, would be, Alan is in fact only 
listed for ide-cd it seems...) repeated in different entries.

(as a slight difference -- in the above example, Alan's information _is_ 
repeated over ide-cd.c and ide-cd.h where the current MAINTAINERS file just 
says "IDE/ATAPI CDROM DRIVER", but that's a bit of an oddbal situation since 
you normally have either single files or a tree that make up a "maintenance 
unit" -- and is in fact just a human versus tool difference).

> It would be much easier to put in the "kernel/somesubsys" directory a 
> Maintainers file which has:

It's ofcourse possible, but note that if we want this stuff to be minimally 
manual, moving files around (and deleting them) then requires editing these 
actual in-tree files via a tool.

With the properties deleting files just requires deleting any file-specific 
properties alongside which is trivial since those are linked from the file.

Moving stuff works by building a list of all properties that are set on the 
source starting at the source and destination's highest shared parent 
directory and then reconstructing this list at the destination, striking 
properties off the list that are already set at the destination.

Adding properties, alongside added files or after the fact, could be done 
via standard patch submissals via the kind of "meta-diff" that already 
exists for "git move".

I really believe this stuff should be meta-data -- and these properties as 
outlined work well it seems.

$ git prop --set --name git.ignore -V ./.gitignore .
$ rm .gitignore

This is something I saw subversion also uses properties for. Takes the value 
from a file instead of the command line. .gitattributes are also easily 
incorporated into the property-system directly.

$ git prop --set --name git.executable scripts/Lindent

Must say I'm not particularly sure if this one has much value over the 
current executable bit storage,but also from svn and example of a boolean 
property.

$ git prop --set --name license --value "GPL v2" .
$ git prop --set --name license --value "GPL" sound/alsa/

and so on. The GPL v2 on the source root only works if you set the property 
on everything that's not, so you may not want to but as a "wouldn't it be 
nice if" kinda thing. Makes for easy license analysis at least...

$ git prop --set --name FIXME drivers/block/floppy.c

Okay, that's probably overdoing it a bit, but as long as I'm having fun here...

Note -- the properties would be versioned themselves ofcourse so that you'd 
always have a tree where data and meta-data matched. Basically, I believe 
you'd view the properties as just more data files, one per property, with 
the exception that they'd not actually live in the working tree and are 
linked from data (files/directories) that do.

Long "letters of intent" this, but I'm by now in love with these things. 
More comments (or implementations obviously :-) welcome. Any significant 
misses in this?

Rene.

Re: [linux-pm] Re: Storing Maintainers info around the kernel tree

[Alan Stern]

On Thu, 16 Aug 2007, Rene Herman wrote:

> > It would be much easier to put in the "kernel/somesubsys" directory a 
> > Maintainers file which has:
> 
> It's ofcourse possible, but note that if we want this stuff to be minimally 
> manual, moving files around (and deleting them) then requires editing these 
> actual in-tree files via a tool.
> 
> With the properties deleting files just requires deleting any file-specific 
> properties alongside which is trivial since those are linked from the file.
> 
> Moving stuff works by building a list of all properties that are set on the 
> source starting at the source and destination's highest shared parent 
> directory and then reconstructing this list at the destination, striking 
> properties off the list that are already set at the destination.
> 
> Adding properties, alongside added files or after the fact, could be done 
> via standard patch submissals via the kind of "meta-diff" that already 
> exists for "git move".
> 
> I really believe this stuff should be meta-data -- and these properties as 
> outlined work well it seems.

Please remember that not everybody uses git.  The MAINTAINERS data 
should be available in the kernel source itself.

(Maybe your suggestion is consistent with this -- I simply wanted to 
raise the point.)

Alan Stern

Re: [linux-pm] Re: Storing Maintainers info around the kernel tree

[Rene Herman]

On 08/16/2007 05:31 PM, Alan Stern wrote:

> Please remember that not everybody uses git.  The MAINTAINERS data 
> should be available in the kernel source itself.

It may be useful to generate a MAINTAINERS file into releases yes.

I must say though that "why?" would also be a question. I personally don't 
think there's a whole lot wrong with more and more expecting people who 
submit patches (for whom this automation is intended) to be using git. Back 
in the BK days there were lots of reasons for resisting any and all 
dependency on the source code management tool but there don't seem to be too 
many left today as far as I'm concerned.

If it's about non-developer users, I suspect it would to a fairly large 
degree be an "in theory" thing to expect that said user does want the 
information in a downloaded releases, but not in git, and not online where 
git-web could also easily display all the information right alongside the files.

But yes, sure, anything can be generated...

Rene.

Re: [linux-pm] Re: Storing Maintainers info around the kernel tree

[Stefan Richter]

Rene Herman wrote:
> I personally don't think there's a whole lot wrong with more and more
> expecting people who submit patches (for whom this automation is
> intended) to be using git.

You mean "people who frequently submit patches for various different
subsystems".
-- 
Stefan Richter
-=====-=-=== =--- =----
http://arcgraph.de/sr/

Re: [linux-pm] Re: Storing Maintainers info around the kernel tree

[Rene Herman]

On 08/16/2007 11:39 PM, Stefan Richter wrote:
> Rene Herman wrote:

>> I personally don't think there's a whole lot wrong with more and more
>> expecting people who submit patches (for whom this automation is
>> intended) to be using git.
> 
> You mean "people who frequently submit patches for various different
> subsystems".

Erm, I guess. Is that agreeing or disagreeing with me?

Rene.

Re: [linux-pm] Re: Storing Maintainers info around the kernel tree

[Alan Stern]

On Fri, 17 Aug 2007, Rene Herman wrote:

> On 08/16/2007 11:39 PM, Stefan Richter wrote:
> > Rene Herman wrote:
> 
> >> I personally don't think there's a whole lot wrong with more and more
> >> expecting people who submit patches (for whom this automation is
> >> intended) to be using git.
> > 
> > You mean "people who frequently submit patches for various different
> > subsystems".
> 
> Erm, I guess. Is that agreeing or disagreeing with me?

Don't forget also that the MAINTAINERS information is (or should be!)
used by people who want to submit bug reports, not just by people who
submit patches.  Bug reporters shouldn't need to use Git.

Alan Stern

Re: [linux-pm] Re: Storing Maintainers info around the kernel tree

[Rene Herman]

On 08/17/2007 03:58 AM, Alan Stern wrote:

> On Fri, 17 Aug 2007, Rene Herman wrote:
> 
>> On 08/16/2007 11:39 PM, Stefan Richter wrote:
>>> Rene Herman wrote:

>>>> I personally don't think there's a whole lot wrong with more and 
>>>> more expecting people who submit patches (for whom this automation 
>>>> is intended) to be using git.
>>> 
>>> You mean "people who frequently submit patches for various different
>>> subsystems".
>> 
>> Erm, I guess. Is that agreeing or disagreeing with me?
> 
> Don't forget also that the MAINTAINERS information is (or should be!)
> used by people who want to submit bug reports, not just by people who
> submit patches.  Bug reporters shouldn't need to use Git.

Like I said:

>> If it's about non-developer users, I suspect it would to a fairly large
>> degree be an "in theory" thing to expect that said user does want the
>> information in a downloaded releases, but not in git, and not online 
>> where git-web could also easily display all the information right 
>> alongside the files.

And again, generating the MAINTAINERS file/info into releases is fine as well.

Rene.

Re: Re: Storing Maintainers info around the kernel tree

[Stefan Richter]

Rene Herman wrote:
> On 08/17/2007 03:58 AM, Alan Stern wrote:
> 
>> On Fri, 17 Aug 2007, Rene Herman wrote:
>>
>>> On 08/16/2007 11:39 PM, Stefan Richter wrote:
>>>> Rene Herman wrote:
> 
>>>>> I personally don't think there's a whole lot wrong with more and
>>>>> more expecting people who submit patches (for whom this automation
>>>>> is intended) to be using git.
>>>>
>>>> You mean "people who frequently submit patches for various different
>>>> subsystems".
>>>
>>> Erm, I guess. Is that agreeing or disagreeing with me?
>>
>> Don't forget also that the MAINTAINERS information is (or should be!)
>> used by people who want to submit bug reports, not just by people who
>> submit patches.  Bug reporters shouldn't need to use Git.

Yes, problem reporters and people who infrequently (or for their first
time) submit patches, and even people who frequently submit patches but
most of the time only to the same one or two subsystems need an obvious,
tool-independent way to get contact information.

> Like I said:
> 
>>> If it's about non-developer users, I suspect it would to a fairly large
>>> degree be an "in theory" thing to expect that said user does want the
>>> information in a downloaded releases, but not in git, and not online
>>> where git-web could also easily display all the information right
>>> alongside the files.
> 
> And again, generating the MAINTAINERS file/info into releases is fine as
> well.

Good.  This generated data will be used by almost everyone except for a
certain special group of submitters.
-- 
Stefan Richter
-=====-=-=== =--- =---=
http://arcgraph.de/sr/