[PATCH 0/4] Scripted update of the MAINTAINERS files

[PATCH 0/4] Scripted update of the MAINTAINERS files

[Joe Perches]

MAINTAINERS is a very large file.

Update the parse-maintainer script to use specific filenames to
allow scripting of the moving of various bits into separate files.

Add a bash script to move various bits into multiple files.

Joe Perches (4):
  parse-maintainers: Add section pattern sorting
  parse-maintainers: Use perl hash references and specific filenames
  parse-maintainers: Move matching sections from MAINTAINERS
  scripts/move_maintainer_sections.bash

 scripts/move_maintainer_sections.bash | 87 +++++++++++++++++++++++++++++++
 scripts/parse-maintainers.pl          | 97 ++++++++++++++++++++++++++---------
 2 files changed, 161 insertions(+), 23 deletions(-)
 create mode 100755 scripts/move_maintainer_sections.bash

-- 
2.10.0.rc2.1.g053435c

[PATCH 1/4] parse-maintainers: Add section pattern sorting

[Joe Perches]

Section [A-Z]: patterns are not currently in any required sorting order.
Add a specific sorting sequence to MAINTAINERS entries.
Sort F: and X: patterns in alphabetic order.

The preferred section ordering is:

SECTION HEADER
M:	Maintainers
R:	Reviewers
P:	Named persons without email addresses
L:	Mailing list addresses
S:	Status of this section (Supported, Maintained, Orphan, etc...)
W:	Any relevant URLs
T:	Source code control type (git, quilt, etc)
Q:	Patchwork patch acceptance queue site
B:	Bug tracking URIs
C:	Chat URIs
F:	Files with wildcard patterns (alphabetic ordered)
X:	Excluded files with wildcard patterns (alphabetic ordered)
N:	Files with regex patterns
K:	Keyword regexes in source code for maintainership identification

Miscellaneous perl neatening:

o Rename %map to %hash, map has a different meaning in perl
o Avoid using \& and local variables for function indirection
o Use return for a little c like clarity
o Use c-like function call style instead of &function

Signed-off-by: Joe Perches <joe@perches.com>
---
 scripts/parse-maintainers.pl | 70 +++++++++++++++++++++++++++++++-------------
 1 file changed, 49 insertions(+), 21 deletions(-)

diff --git a/scripts/parse-maintainers.pl b/scripts/parse-maintainers.pl
index a0fe34349b24..5c8e0504c67e 100644
--- a/scripts/parse-maintainers.pl
+++ b/scripts/parse-maintainers.pl
@@ -2,9 +2,9 @@
 
 use strict;
 
-my %map;
+my %hash;
 
-# sort comparison function
+# sort comparison functions
 sub by_category($$) {
     my ($a, $b) = @_;
 
@@ -15,20 +15,47 @@ sub by_category($$) {
     $a =~ s/THE REST/ZZZZZZ/g;
     $b =~ s/THE REST/ZZZZZZ/g;
 
-    $a cmp $b;
+    return $a cmp $b;
+}
+
+sub by_pattern($$) {
+    my ($a, $b) = @_;
+    my $preferred_order = 'MRPLSWTQBCFXNK';
+
+    my $a1 = uc(substr($a, 0, 1));
+    my $b1 = uc(substr($b, 0, 1));
+
+    my $a_index = index($preferred_order, $a1);
+    my $b_index = index($preferred_order, $b1);
+
+    $a_index = 1000 if ($a_index == -1);
+    $b_index = 1000 if ($b_index == -1);
+
+    if (($a1 =~ /^F$/ && $b1 =~ /^F$/) ||
+	($a1 =~ /^X$/ && $b1 =~ /^X$/)) {
+	return $a cmp $b;
+    }
+
+    if ($a_index < $b_index) {
+	return -1;
+    } elsif ($a_index == $b_index) {
+	return 0;
+    } else {
+	return 1;
+    }
 }
 
 sub alpha_output {
-    my $key;
-    my $sort_method = \&by_category;
-    my $sep = "";
-
-    foreach $key (sort $sort_method keys %map) {
-        if ($key ne " ") {
-            print $sep . $key . "\n";
-            $sep = "\n";
-        }
-        print $map{$key};
+    foreach my $key (sort by_category keys %hash) {
+	if ($key eq " ") {
+	    chomp $hash{$key};
+	    print $hash{$key};
+	} else {
+	    print "\n" . $key . "\n";
+	    foreach my $pattern (sort by_pattern split('\n', $hash{$key})) {
+		print($pattern . "\n");
+	    }
+	}
     }
 }
 
@@ -42,7 +69,7 @@ sub trim {
 sub file_input {
     my $lastline = "";
     my $case = " ";
-    $map{$case} = "";
+    $hash{$case} = "";
 
     while (<>) {
         my $line = $_;
@@ -51,27 +78,28 @@ sub file_input {
         if ($line =~ m/^([A-Z]):\s*(.*)/) {
             $line = $1 . ":\t" . trim($2) . "\n";
             if ($lastline eq "") {
-                $map{$case} = $map{$case} . $line;
+                $hash{$case} = $hash{$case} . $line;
                 next;
             }
             $case = trim($lastline);
-            exists $map{$case} and die "Header '$case' already exists";
-            $map{$case} = $line;
+            exists $hash{$case} and die "Header '$case' already exists";
+            $hash{$case} = $line;
             $lastline = "";
             next;
         }
 
         if ($case eq " ") {
-            $map{$case} = $map{$case} . $lastline;
+            $hash{$case} = $hash{$case} . $lastline;
             $lastline = $line;
             next;
         }
         trim($lastline) eq "" or die ("Odd non-pattern line '$lastline' for '$case'");
         $lastline = $line;
     }
-    $map{$case} = $map{$case} . $lastline;
+    $hash{$case} = $hash{$case} . $lastline;
 }
 
-&file_input;
-&alpha_output;
+file_input();
+alpha_output();
+
 exit(0);
-- 
2.10.0.rc2.1.g053435c

[PATCH 2/4] parse-maintainers: Use perl hash references and specific filenames

[Joe Perches]

Instead of reading STDIN and writing STDOUT, use specific filenames of
MAINTAINERS and MAINTAINERS.new.

Use hash references instead of global hash %hash so future modifications
can read and write specific hashes to split up MAINTAINERS into multiple
files using a script.

Signed-off-by: Joe Perches <joe@perches.com>
---
 scripts/parse-maintainers.pl | 57 ++++++++++++++++++++++++++------------------
 1 file changed, 34 insertions(+), 23 deletions(-)

diff --git a/scripts/parse-maintainers.pl b/scripts/parse-maintainers.pl
index 5c8e0504c67e..c286154a2b68 100644
--- a/scripts/parse-maintainers.pl
+++ b/scripts/parse-maintainers.pl
@@ -2,7 +2,7 @@
 
 use strict;
 
-my %hash;
+my $P = $0;
 
 # sort comparison functions
 sub by_category($$) {
@@ -45,61 +45,72 @@ sub by_pattern($$) {
     }
 }
 
+sub trim {
+    my $s = shift;
+    $s =~ s/\s+$//;
+    $s =~ s/^\s+//;
+    return $s;
+}
+
 sub alpha_output {
-    foreach my $key (sort by_category keys %hash) {
+    my ($hashref, $filename) = (@_);
+
+    open(my $file, '>', "$filename") or die "$P: $filename: open failed - $!\n";
+    foreach my $key (sort by_category keys %$hashref) {
 	if ($key eq " ") {
-	    chomp $hash{$key};
-	    print $hash{$key};
+	    chomp $$hashref{$key};
+	    print $file $$hashref{$key};
 	} else {
-	    print "\n" . $key . "\n";
-	    foreach my $pattern (sort by_pattern split('\n', $hash{$key})) {
-		print($pattern . "\n");
+	    print $file "\n" . $key . "\n";
+	    foreach my $pattern (sort by_pattern split('\n', %$hashref{$key})) {
+		print $file ($pattern . "\n");
 	    }
 	}
     }
-}
-
-sub trim {
-    my $s = shift;
-    $s =~ s/\s+$//;
-    $s =~ s/^\s+//;
-    return $s;
+    close($file);
 }
 
 sub file_input {
+    my ($hashref, $filename) = (@_);
+
     my $lastline = "";
     my $case = " ";
-    $hash{$case} = "";
+    $$hashref{$case} = "";
+
+    open(my $file, '<', "$filename") or die "$P: $filename: open failed - $!\n";
 
-    while (<>) {
+    while (<$file>) {
         my $line = $_;
 
         # Pattern line?
         if ($line =~ m/^([A-Z]):\s*(.*)/) {
             $line = $1 . ":\t" . trim($2) . "\n";
             if ($lastline eq "") {
-                $hash{$case} = $hash{$case} . $line;
+                $$hashref{$case} = $$hashref{$case} . $line;
                 next;
             }
             $case = trim($lastline);
-            exists $hash{$case} and die "Header '$case' already exists";
-            $hash{$case} = $line;
+            exists $$hashref{$case} and die "Header '$case' already exists";
+            $$hashref{$case} = $line;
             $lastline = "";
             next;
         }
 
         if ($case eq " ") {
-            $hash{$case} = $hash{$case} . $lastline;
+            $$hashref{$case} = $$hashref{$case} . $lastline;
             $lastline = $line;
             next;
         }
         trim($lastline) eq "" or die ("Odd non-pattern line '$lastline' for '$case'");
         $lastline = $line;
     }
-    $hash{$case} = $hash{$case} . $lastline;
+    $$hashref{$case} = $$hashref{$case} . $lastline;
+    close($file);
 }
 
-file_input();
-alpha_output();
+my %hash;
+
+file_input(\%hash, "MAINTAINERS");
+alpha_output(\%hash, "MAINTAINERS.new");
 
 exit(0);
-- 
2.10.0.rc2.1.g053435c

[PATCH 3/4] parse-maintainers: Move matching sections from MAINTAINERS

[Joe Perches]

Allow any number of command line arguments to match either the
section header or the section contents and create new files.

Create MAINTAINERS.new and SECTION.new.

This allows scripting of the movement of various sections from
MAINTAINERS.

Signed-off-by: Joe Perches <joe@perches.com>
---
 scripts/parse-maintainers.pl | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/scripts/parse-maintainers.pl b/scripts/parse-maintainers.pl
index c286154a2b68..e40b53db7f9f 100644
--- a/scripts/parse-maintainers.pl
+++ b/scripts/parse-maintainers.pl
@@ -109,8 +109,20 @@ sub file_input {
 }
 
 my %hash;
+my %new_hash;
 
 file_input(\%hash, "MAINTAINERS");
+
+foreach my $type (@ARGV) {
+    foreach my $key (keys %hash) {
+	if ($key =~ /$type/ || $hash{$key} =~ /$type/) {
+	    $new_hash{$key} = $hash{$key};
+	    delete $hash{$key};
+	}
+    }
+}
+
 alpha_output(\%hash, "MAINTAINERS.new");
+alpha_output(\%new_hash, "SECTION.new");
 
 exit(0);
-- 
2.10.0.rc2.1.g053435c

[PATCH 4/4] scripts/move_maintainer_sections.bash

[Joe Perches]

Move MAINTAINERS into a separate directory and reorder it.
Separate various blocks of MAINTAINER sections into separate files.

Signed-off-by: Joe Perches <joe@perches.com>
---
 scripts/move_maintainer_sections.bash | 87 +++++++++++++++++++++++++++++++++++
 1 file changed, 87 insertions(+)
 create mode 100755 scripts/move_maintainer_sections.bash

diff --git a/scripts/move_maintainer_sections.bash b/scripts/move_maintainer_sections.bash
new file mode 100755
index 000000000000..7ea671b6435d
--- /dev/null
+++ b/scripts/move_maintainer_sections.bash
@@ -0,0 +1,87 @@
+# Select various bits out of MAINTAINERS and create new file and commit
+function separate_and_commit () {
+    local filename=$1
+    shift
+    local comment=$1
+    shift
+    local selection=$@
+
+    cd MAINTAINERS
+    perl ../scripts/parse-maintainers.pl $selection
+    mv MAINTAINERS.new MAINTAINERS
+    mv SECTION.new $filename
+    cd ..
+    cat << EOF > commit.msg
+MAINTAINERS: Move $comment sections to separate file
+
+Make it easier to handle merge conflicts.
+EOF
+    git add MAINTAINERS/$filename
+    git commit -s -F commit.msg MAINTAINERS/MAINTAINERS MAINTAINERS/$filename
+    rm commit.msg
+}
+
+# First move the MAINTAINERS file into a separate MAINTAINERS directory
+git mv MAINTAINERS MAINTAINERS.old
+mkdir MAINTAINERS
+git mv MAINTAINERS.old MAINTAINERS/MAINTAINERS
+cat << EOF > commit.msg
+MAINTAINERS: Move to MAINTAINERS directory
+
+This allows breaking up the very large and difficult to merge
+MAINTAINERS files into separate files
+EOF
+git commit -s -F commit.msg MAINTAINERS/MAINTAINERS
+rm commit.msg
+
+# Sort the MAINTAINERS file appropriately
+
+cd MAINTAINERS
+perl ../scripts/parse-maintainers.pl
+mv MAINTAINERS.new MAINTAINERS
+cd ..
+cat << EOF > commit.msg
+MAINTAINERS: Reorder and alphabetize file sections
+
+Move the sections patterns into a standardized order.
+EOF
+git commit -s -F commit.msg MAINTAINERS/MAINTAINERS 
+rm commit.msg
+
+# Move the kernel related sections
+
+separate_and_commit kernel kernel "\\nF:\\s*kernel/(?:.*/)*\\n" "\\nF:\\s*kernel/(?:.*/)*(?:.*\\.[ch]|.*\\*)?\\n"
+separate_and_commit filesystems "filesystems" "\\nF:\\s*fs/"
+separate_and_commit hypervisors "hypervisor subsystems and drivers" "\\nL:\\s*xen-devel" "\\nL:\\s*virtualization" "Hyper-V"
+
+# Move arch specific files
+
+for arch in $(find arch -maxdepth 1 -type d | \
+		  sed 's@^arch/@@' | \
+		  awk '{ if (NR > 1) { print; } }' | \
+		  sort) ; do
+    separate_and_commit arch_$arch arch/$arch "\\nF:\\s*arch/$arch/(?:.*/)*\\n"
+done
+
+# Move various subsystems and driver sections
+
+separate_and_commit networking_core "networking core" "\\nF:\\s*net/"
+separate_and_commit drivers_wireless "wireless drivers" "\\nF:\\s*drivers/net/wireless/" "\\bWIRELESS\\b"
+separate_and_commit drivers_ethernet "ethernet drivers" "\\nF:\\s*drivers/net/ethernet/"
+separate_and_commit drivers_scsi "scsi drivers" "\\nF:\\s*drivers/scsi/"
+separate_and_commit drivers_usb "usb drivers" "\\nF:\\s*drivers/usb/"
+separate_and_commit drivers_media "media drivers" "\\nF:\\s*drivers/media/"
+separate_and_commit drivers_watchdog "watchdog drivers" "\\nF:\\s*drivers/watchdog/"
+separate_and_commit sound "sound subsystems and drivers" "\\nF:\\s*sound/"
+separate_and_commit hardware_monitoring "monitoring subsystem and drivers" "\\nL:\\s*linux-hwmon" "\\nF:\\s*drivers/hwmon/"
+separate_and_commit acpi "ACPI subsystem and drivers" "\\nL:\\s*linux-acpi" "\\nF:\\s*drivers/acpi/"
+separate_and_commit drivers_input "input drivers" "\\nL:\\s*linux-input" "\\nF:\\s*drivers/input/"
+separate_and_commit drivers_video "video drivers" "\\nL:\\s*linux-input" "\\nF:\\s*drivers/video/"
+separate_and_commit drivers_gpio "gpio drivers" "\\nL:\\s*linux-gpio" "\\nF:\\s*drivers/gpio/"
+separate_and_commit drivers_serial "serial/tty drivers" "\\nL:\\s*linux-serial" "\\nF:\\s*drivers/tty/"
+separate_and_commit drivers_gpu_drm "GPU/DRM drivers" "\\nF:\\s*drivers/gpu/"
+separate_and_commit drivers_i2c "I2C drivers" "\\nL:\\s*linux-i2c" "\\nF:\\s*drivers/i2c/"
+separate_and_commit drivers_staging "staging drivers" "\\nL:\\s*devel\@driverdev" "\\nF:\\s*drivers/staging/"
+separate_and_commit power "power management and drivers" "\\nL:\\s*linux-pm" "\\nF:\\s*drivers/cpufreq/" "\\nF:\\s*drivers/cpuidle/" "\\nF:\\s*drivers/devfreq/"
+separate_and_commit pci "PCI drivers" "\\nL:\\s*linux-pci" "\\nF:\\s*drivers/pci/"
+separate_and_commit platform-driver-x86 "x86 platform drivers" "\\nL:\\s*platform-driver-x86" "\\nF:\\s*drivers/platform/x86/"
-- 
2.10.0.rc2.1.g053435c

Re: [PATCH 4/4] scripts/move_maintainer_sections.bash

[Joe Perches]

On Sat, 2017-08-05 at 18:45 -0700, Joe Perches wrote:
> Move MAINTAINERS into a separate directory and reorder it.
> Separate various blocks of MAINTAINER sections into separate files.

Hey Linus.

If/when you try this out, do please let me know what
you think.

There are some oddities in the results like the
"PCI SUBSYSTEM" block ending up in arch_x86, but for
a first pass I think it's fairly reasonable.

Of course the move_maintainer_sections script isn't
meant to end up in the tree, it was just the easiest
way to send it.

Re: [PATCH 4/4] scripts/move_maintainer_sections.bash

[Linus Torvalds]

On Mon, Aug 7, 2017 at 10:04 AM, Joe Perches <joe@perches.com> wrote:
> On Sat, 2017-08-05 at 18:45 -0700, Joe Perches wrote:
>> Move MAINTAINERS into a separate directory and reorder it.
>> Separate various blocks of MAINTAINER sections into separate files.
>
> Hey Linus.
>
> If/when you try this out, do please let me know what
> you think.

Ok, I've applied the preparatory patches, but not run the script.

Or rather, I ran the script to see what happens, but I'm not going to
push the end result.

>From a quick look at the end result, I note:

 - the arch maintainer split is pointless. It ends up being just one
entry per architecture, and for x86_64 not even that (because the x86
pattern matched all of them)

 - the two arch maintainer lists that end up being bigger is x86 and
arm, but the x86 one picked up a log of misleading ones (not just PCI:
EFI, various random other things too)

 - they all end up having the empty line at the top because of how the
parse-maintainers.pl script works.

But *some* of it looks really nice.

The other thing I note is that the way the patches look, this is going
to be a disaster to merge with any other work - and there really tends
to be a lot of things touching MAINTAINERS.

I'll have to think about it.

But at least the infrastructure patches are applied,

                Linus

Re: [PATCH 4/4] scripts/move_maintainer_sections.bash

[Joe Perches]

On Tue, 2017-08-08 at 11:33 -0700, Linus Torvalds wrote:
> On Mon, Aug 7, 2017 at 10:04 AM, Joe Perches <joe@perches.com> wrote:
> > On Sat, 2017-08-05 at 18:45 -0700, Joe Perches wrote:
> > > Move MAINTAINERS into a separate directory and reorder it.
> > > Separate various blocks of MAINTAINER sections into separate files.
> > 
> > Hey Linus.
> > 
> > If/when you try this out, do please let me know what
> > you think.
> 
> Ok, I've applied the preparatory patches, but not run the script.
> 
> Or rather, I ran the script to see what happens, but I'm not going to
> push the end result.
> 
> From a quick look at the end result, I note:
> 
>  - the arch maintainer split is pointless. It ends up being just one
> entry per architecture

I think that's just preliminary.

Many of the arches have a single maintainer and many
of the drivers specific to that arch could/should be
moved into the arch_<foo> file.

But that doesn't seem easily scriptable.

> for x86_64 not even that (because the x86> pattern matched all of
> them)

What x86_64 section is that?  There isn't an arch/x86_64
directory and only 1 combined entry for x86 and x86_64.

>  - the two arch maintainer lists that end up being bigger is x86 and
> arm, but the x86 one picked up a log of misleading ones (not just PCI:
> EFI, various random other things too)

Yeah, it's imperfect.  Suggestions welcomed.

>  - they all end up having the empty line at the top because of how the
> parse-maintainers.pl script works.

That's because I was a bit lazy about the output.
	awk '{ if (NR > 1) { print; } }'
would fix it.

Also a 00-README type file for the introductory section
could be useful.

> But *some* of it looks really nice.
> 
> The other thing I note is that the way the patches look, this is going
> to be a disaster to merge with any other work - and there really tends
> to be a lot of things touching MAINTAINERS.

It is going to be messy.
Once done it should be easier though.

Even today there's a merge conflict (kinda) with -next and
the alphabetic
reordering movement as one of the sections
is duplicated again. (SYNC
FILE FRAMEWORK)

Collecting the remainder MAINTAINERS patches after an -rc1
and applying them with a quilt like merge might help too
if Andrew feels up to it.

> I'll have to think about it.
> 
> But at least the infrastructure patches are applied,

Thanks.