git clone svn: authors from authors file are ignored, authors-prog works, but crashes on branch points

git clone svn: authors from authors file are ignored, authors-prog works, but crashes on branch points

[Till Schäfer]

Hi,
i am observing some weired "git svn clone" behavior during my try to migrate the Scaffold Hunter [1] SVN repository [2] to Git: 

if i just use the command 

$ git svn clone svn://svn.code.sf.net/p/scaffoldhunter/code --no-metadata -s code

everything went smoothly. Now i wanted to replace the authors SVN-logins by useful names and created a mapping file with the following content: 

anjenson = Andrew Zhilka <>
bernhard.dick = Bernhard Dick <>
dominic.sacre = Dominic Sacré <>
doxmoxbox = doxmoxbox <>
falkn = Falk Nette <>
henning.garus = Henning Garus <>
kakl = Karsten Klein <>
klein = Karsten Klein <>
lappie00 = Jeroen Lappenschaar <>
michael.hesse = Michael Hesse <>
nlskrg = Nils Kriege <nlskrg@gmail.com>
philipp.kopp = Philipp Kopp <>
philipp.lewe = Philipp Lewe <>
schrins = Sven Schrinner <>
shamshadnpti = Shamshad Alam <>
srenner7 = Steffen Renner <>
sturm89 = Werner Sturm <>
thomas.schmitz = Thomas Schmitz <>
thorsten.fluegel = Thorsten Flügel <>
till.schaefer = Till Schäfer <till2.schaefer@tu-dortmund.de>
tillschaefer = Till Schäfer <till2.schaefer@tu-dortmund.de>


I executed:

$ git svn clone svn://svn.code.sf.net/p/scaffoldhunter/code --authors-file=/home/till/temp/code-authors-transform.txt --no-metadata -s code

and got the error message "Author: klein not defined in /home/till/temp/code-authors-transform.txt file". 
I tried a workaround by using a python script to return the entries i have already defined in the authors file: 

$ git svn clone svn://svn.code.sf.net/p/scaffoldhunter/code --authors-file=/home/till/temp/code-authors-transform.txt --authors-prog=/home/till/temp/authors.py --no-metadata -s code

The import process now went over the "klein" commit. The weired thing is, that a few SVN usernames seem to be recognized in the authors file and a few are passed to my script (I logged the output). However, at the first revision, where a tag was added in the SVN repo the cloning process crashed with the error message: 

Found possible branch point: svn://svn.code.sf.net/p/scaffoldhunter/code/trunk => svn://svn.code.sf.net/p/scaffoldhunter/code/branches/subsearch, 17
Use of uninitialized value $u in substitution (s///) at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
Use of uninitialized value $u in concatenation (.) or string at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
refs/remotes/origin/trunk: 'svn://svn.code.sf.net/p/scaffoldhunter/code' not found in ''

One problem I am seeing in the SVN repo is, that between the revisions 97 and 102 the trunk folder was absent. the old one was moved to some subfolder and a new one was created. Therefore, I started the cloning beginning with rev 102 using the command line option "-r102:HEAD". However, the same error occurred for some later tag (the first tag after rev 102). 

Found possible branch point: svn://svn.code.sf.net/p/scaffoldhunter/code/trunk => svn://svn.code.sf.net/p/scaffoldhunter/code/tags/release-2.0, 1565
Use of uninitialized value $u in substitution (s///) at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
Use of uninitialized value $u in concatenation (.) or string at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
refs/remotes/origin/trunk: 'svn://svn.code.sf.net/p/scaffoldhunter/code' not found in ''

I would be very glad if someone has a hint about what is going wrong here. Is this a bug in git or is something wrong with the SVN repo?


used software versions: 
- git 2.5.0 
- subversion 1.8.14


Regards,
Till

[1] http://scaffoldhunter.sourceforge.net/
[2] https://svn.code.sf.net/p/scaffoldhunter/code/trunk

-- 
Dipl.-Inf. Till Schäfer
TU Dortmund University
Chair 11 - Algorithm Engineering
Otto-Hahn-Str. 14 / Room 237
44227 Dortmund, Germany

e-mail: till.schaefer@cs.tu-dortmund.de
phone: +49(231)755-7706
fax: +49(231)755-7740
web: http://ls11-www.cs.uni-dortmund.de/staff/schaefer
pgp: https://keyserver2.pgp.com/vkd/SubmitSearch.event?&&SearchCriteria=0xD84DED79

Re: git clone svn: authors from authors file are ignored, authors-prog works, but crashes on branch points

[Michael J Gruber]

Till Schäfer venit, vidit, dixit 26.08.2015 21:57:
> Hi,
> i am observing some weired "git svn clone" behavior during my try to migrate the Scaffold Hunter [1] SVN repository [2] to Git: 
> 
> if i just use the command 
> 
> $ git svn clone svn://svn.code.sf.net/p/scaffoldhunter/code --no-metadata -s code

Is there a specific reason for "--no-metadata"? The man page advises
against it. Basically, it keeps you from converting the history in
chunks. (You can always filter-branch everything after a complete, "one
shot" conversion which may be done in stages.)

> everything went smoothly. Now i wanted to replace the authors SVN-logins by useful names and created a mapping file with the following content: 
> 
> anjenson = Andrew Zhilka <>
> bernhard.dick = Bernhard Dick <>
> dominic.sacre = Dominic Sacré <>
> doxmoxbox = doxmoxbox <>
> falkn = Falk Nette <>
> henning.garus = Henning Garus <>
> kakl = Karsten Klein <>
> klein = Karsten Klein <>
> lappie00 = Jeroen Lappenschaar <>
> michael.hesse = Michael Hesse <>
> nlskrg = Nils Kriege <nlskrg@gmail.com>
> philipp.kopp = Philipp Kopp <>
> philipp.lewe = Philipp Lewe <>
> schrins = Sven Schrinner <>
> shamshadnpti = Shamshad Alam <>
> srenner7 = Steffen Renner <>
> sturm89 = Werner Sturm <>
> thomas.schmitz = Thomas Schmitz <>
> thorsten.fluegel = Thorsten Flügel <>
> till.schaefer = Till Schäfer <till2.schaefer@tu-dortmund.de>
> tillschaefer = Till Schäfer <till2.schaefer@tu-dortmund.de>
> 
> 
> I executed:
> 
> $ git svn clone svn://svn.code.sf.net/p/scaffoldhunter/code --authors-file=/home/till/temp/code-authors-transform.txt --no-metadata -s code
> 
> and got the error message "Author: klein not defined in /home/till/temp/code-authors-transform.txt file". 
> I tried a workaround by using a python script to return the entries i have already defined in the authors file: 
> 
> $ git svn clone svn://svn.code.sf.net/p/scaffoldhunter/code --authors-file=/home/till/temp/code-authors-transform.txt --authors-prog=/home/till/temp/authors.py --no-metadata -s code
> 
> The import process now went over the "klein" commit. The weired thing is, that a few SVN usernames seem to be recognized in the authors file and a few are passed to my script (I logged the output). However, at the first revision, where a tag was added in the SVN repo the cloning process crashed with the error message: 

That first part is weird already. The reason is that author-file
processing and author-prog output processing use different perl regexps.
I will follow up with a patch.

> Found possible branch point: svn://svn.code.sf.net/p/scaffoldhunter/code/trunk => svn://svn.code.sf.net/p/scaffoldhunter/code/branches/subsearch, 17
> Use of uninitialized value $u in substitution (s///) at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
> Use of uninitialized value $u in concatenation (.) or string at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
> refs/remotes/origin/trunk: 'svn://svn.code.sf.net/p/scaffoldhunter/code' not found in ''
> 
> One problem I am seeing in the SVN repo is, that between the revisions 97 and 102 the trunk folder was absent. the old one was moved to some subfolder and a new one was created. Therefore, I started the cloning beginning with rev 102 using the command line option "-r102:HEAD". However, the same error occurred for some later tag (the first tag after rev 102). 
> 
> Found possible branch point: svn://svn.code.sf.net/p/scaffoldhunter/code/trunk => svn://svn.code.sf.net/p/scaffoldhunter/code/tags/release-2.0, 1565
> Use of uninitialized value $u in substitution (s///) at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
> Use of uninitialized value $u in concatenation (.) or string at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
> refs/remotes/origin/trunk: 'svn://svn.code.sf.net/p/scaffoldhunter/code' not found in ''
> 
> I would be very glad if someone has a hint about what is going wrong here. Is this a bug in git or is something wrong with the SVN repo?

The layout change (no root) certainly doesn't help. We shouldn't crash,
though.

> used software versions: 
> - git 2.5.0 
> - subversion 1.8.14

The conversion goes through for me with (your other options and) your
authors file and specifying a stupid authors prog:

--->8---
#!/bin/sh
grep "$1" /tmp/t/authors | head -n 1 | cut -d'=' -f2 | cut -c'2-'
--->8---

and:

git version 2.6.0.rc0.178.g4177fa5
svn, Version 1.8.13 (r1667537)

with our without --authors-file...

(It went also through with a patched git-svn and just the authors file.)

I don't think git-svn has changed since git 2.5.0. Maybe you can
downgrade svn for the one-shot conversion? I don't see anything
suspicious in their changelog but don't remember it as being overly
complete.

git svn served us well in making itself mostly obsolete. The downside is
that not many are using it any more. Keeping up with subversion changes
(to keep git svn working) has become more and more difficult.

Michael

[PATCH] git-svn: parse authors file more leniently

[Michael J Gruber]

Currently, git-svn parses an authors file using the perl regex

/^(.+?|\(no author\))\s*=\s*(.+?)\s*<(.+)>\s*$/

in order to extract svn user name, real name and e-mail.
This does not match an empty e-mail field like "<>". On the other hand,
the output of an authors-prog is parsed with the perl regex

/^\s*(.+?)\s*<(.*)>\s*$/

in order to extract real name and e-mail.

So, specifying a trivial file grep such as

grep "$1" /tmp/authors | head -n 1 | cut -d'=' -f2 | cut -c'2-'

as the authors prog gives different results compared to specifying
/tmp/authors as the authors file directly.

Instead, make git svn uses the perl regex

/^(.+?|\(no author\))\s*=\s*(.+?)\s*<(.*)>\s*$/

for parsing the authors file so that the same (slightly more lenient)
regex is used in both cases.

Reported-by: Till Schäfer <till2.schaefer@tu-dortmund.de>
Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
---
 git-svn.perl | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/git-svn.perl b/git-svn.perl
index 36f7240..fa5f253 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -1924,7 +1924,7 @@ sub load_authors {
 	my $log = $cmd eq 'log';
 	while (<$authors>) {
 		chomp;
-		next unless /^(.+?|\(no author\))\s*=\s*(.+?)\s*<(.+)>\s*$/;
+		next unless /^(.+?|\(no author\))\s*=\s*(.+?)\s*<(.*)>\s*$/;
 		my ($user, $name, $email) = ($1, $2, $3);
 		if ($log) {
 			$Git::SVN::Log::rusers{"$name <$email>"} = $user;
-- 
2.6.0.rc0.178.g4177fa5

Re: git clone svn: authors from authors file are ignored, authors-prog works, but crashes on branch points

[Till Schäfer]

Hi, 
I can confirm, that the patch (see mail: "[PATCH] git-svn: parse authors file more leniently") fixes the "Author: **** not defined in **** file" problem in my case. 

Downgrading to subversion 1.8.13 (from 1.8.14) also fixes the branching crash in revision 17. With subversion 1.8.14 i still get:  

Found possible branch point: svn://svn.code.sf.net/p/scaffoldhunter/code/trunk => svn://svn.code.sf.net/p/scaffoldhunter/code/branches/subsearch, 17
Use of uninitialized value $u in substitution (s///) at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
Use of uninitialized value $u in concatenation (.) or string at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
refs/remotes/origin/trunk: 'svn://svn.code.sf.net/p/scaffoldhunter/code' not found in ''


Am Donnerstag, 10. September 2015, 14:32:07 schrieb Michael J Gruber:
> Till Schäfer venit, vidit, dixit 26.08.2015 21:57:
> > Hi,
> > i am observing some weired "git svn clone" behavior during my try to migrate the Scaffold Hunter [1] SVN repository [2] to Git: 
> > 
> > if i just use the command 
> > 
> > $ git svn clone svn://svn.code.sf.net/p/scaffoldhunter/code --no-metadata -s code
> 
> Is there a specific reason for "--no-metadata"? The man page advises
> against it. Basically, it keeps you from converting the history in
> chunks. (You can always filter-branch everything after a complete, "one
> shot" conversion which may be done in stages.)

i was basically following the tutorial at git-scm [1] (which is also somewhat outdated in the later svn branch conversion). Anyway, i did not read the manual :-/ (man-page)


Regards, 
Till

[1] https://git-scm.com/book/en/v2/Git-and-Other-Systems-Migrating-to-Git










> 
> > everything went smoothly. Now i wanted to replace the authors SVN-logins by useful names and created a mapping file with the following content: 
> > 
> > anjenson = Andrew Zhilka <>
> > bernhard.dick = Bernhard Dick <>
> > dominic.sacre = Dominic Sacré <>
> > doxmoxbox = doxmoxbox <>
> > falkn = Falk Nette <>
> > henning.garus = Henning Garus <>
> > kakl = Karsten Klein <>
> > klein = Karsten Klein <>
> > lappie00 = Jeroen Lappenschaar <>
> > michael.hesse = Michael Hesse <>
> > nlskrg = Nils Kriege <nlskrg@gmail.com>
> > philipp.kopp = Philipp Kopp <>
> > philipp.lewe = Philipp Lewe <>
> > schrins = Sven Schrinner <>
> > shamshadnpti = Shamshad Alam <>
> > srenner7 = Steffen Renner <>
> > sturm89 = Werner Sturm <>
> > thomas.schmitz = Thomas Schmitz <>
> > thorsten.fluegel = Thorsten Flügel <>
> > till.schaefer = Till Schäfer <till2.schaefer@tu-dortmund.de>
> > tillschaefer = Till Schäfer <till2.schaefer@tu-dortmund.de>
> > 
> > 
> > I executed:
> > 
> > $ git svn clone svn://svn.code.sf.net/p/scaffoldhunter/code --authors-file=/home/till/temp/code-authors-transform.txt --no-metadata -s code
> > 
> > and got the error message "Author: klein not defined in /home/till/temp/code-authors-transform.txt file". 
> > I tried a workaround by using a python script to return the entries i have already defined in the authors file: 
> > 
> > $ git svn clone svn://svn.code.sf.net/p/scaffoldhunter/code --authors-file=/home/till/temp/code-authors-transform.txt --authors-prog=/home/till/temp/authors.py --no-metadata -s code
> > 
> > The import process now went over the "klein" commit. The weired thing is, that a few SVN usernames seem to be recognized in the authors file and a few are passed to my script (I logged the output). However, at the first revision, where a tag was added in the SVN repo the cloning process crashed with the error message: 
> 
> That first part is weird already. The reason is that author-file
> processing and author-prog output processing use different perl regexps.
> I will follow up with a patch.
> 
> > Found possible branch point: svn://svn.code.sf.net/p/scaffoldhunter/code/trunk => svn://svn.code.sf.net/p/scaffoldhunter/code/branches/subsearch, 17
> > Use of uninitialized value $u in substitution (s///) at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
> > Use of uninitialized value $u in concatenation (.) or string at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
> > refs/remotes/origin/trunk: 'svn://svn.code.sf.net/p/scaffoldhunter/code' not found in ''
> > 
> > One problem I am seeing in the SVN repo is, that between the revisions 97 and 102 the trunk folder was absent. the old one was moved to some subfolder and a new one was created. Therefore, I started the cloning beginning with rev 102 using the command line option "-r102:HEAD". However, the same error occurred for some later tag (the first tag after rev 102). 
> > 
> > Found possible branch point: svn://svn.code.sf.net/p/scaffoldhunter/code/trunk => svn://svn.code.sf.net/p/scaffoldhunter/code/tags/release-2.0, 1565
> > Use of uninitialized value $u in substitution (s///) at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
> > Use of uninitialized value $u in concatenation (.) or string at /usr/lib64/perl5/vendor_perl/5.20.2/Git/SVN.pm line 101.
> > refs/remotes/origin/trunk: 'svn://svn.code.sf.net/p/scaffoldhunter/code' not found in ''
> > 
> > I would be very glad if someone has a hint about what is going wrong here. Is this a bug in git or is something wrong with the SVN repo?
> 
> The layout change (no root) certainly doesn't help. We shouldn't crash,
> though.
> 
> > used software versions: 
> > - git 2.5.0 
> > - subversion 1.8.14
> 
> The conversion goes through for me with (your other options and) your
> authors file and specifying a stupid authors prog:
> 
> --->8---
> #!/bin/sh
> grep "$1" /tmp/t/authors | head -n 1 | cut -d'=' -f2 | cut -c'2-'
> --->8---
> 
> and:
> 
> git version 2.6.0.rc0.178.g4177fa5
> svn, Version 1.8.13 (r1667537)
> 
> with our without --authors-file...
> 
> (It went also through with a patched git-svn and just the authors file.)
> 
> I don't think git-svn has changed since git 2.5.0. Maybe you can
> downgrade svn for the one-shot conversion? I don't see anything
> suspicious in their changelog but don't remember it as being overly
> complete.
> 
> git svn served us well in making itself mostly obsolete. The downside is
> that not many are using it any more. Keeping up with subversion changes
> (to keep git svn working) has become more and more difficult.
> 
> Michael





-- 
Dipl.-Inf. Till Schäfer
TU Dortmund University
Chair 11 - Algorithm Engineering
Otto-Hahn-Str. 14 / Room 237
44227 Dortmund, Germany

e-mail: till.schaefer@cs.tu-dortmund.de
phone: +49(231)755-7706
fax: +49(231)755-7740
web: http://ls11-www.cs.uni-dortmund.de/staff/schaefer
pgp: https://keyserver2.pgp.com/vkd/SubmitSearch.event?&&SearchCriteria=0xD84DED79

Re: [PATCH] git-svn: parse authors file more leniently

[Eric Wong]

Michael J Gruber <git@drmicha.warpmail.net> wrote:
> Instead, make git svn uses the perl regex
> 
> /^(.+?|\(no author\))\s*=\s*(.+?)\s*<(.*)>\s*$/
> 
> for parsing the authors file so that the same (slightly more lenient)
> regex is used in both cases.
> 
> Reported-by: Till Schäfer <till2.schaefer@tu-dortmund.de>
> Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>

Thanks.
Signed-off-by: Eric Wong <normalperson@yhbt.net>

And pushed to master of git://bogomips.org/git-svn
(commit f7c6de0ea1bd5722a1181c6279676c6831b38a34)

By the way, I also had some other patches sitting around for you.
Did you ever have time to revisit them?  (I haven't)

      t/lib-httpd: load mod_unixd
      t/lib-git-svn: check same httpd module dirs as lib-httpd

Re: [PATCH] git-svn: parse authors file more leniently

[Junio C Hamano]

Eric Wong <normalperson@yhbt.net> writes:

> Michael J Gruber <git@drmicha.warpmail.net> wrote:
>> Instead, make git svn uses the perl regex
>> 
>> /^(.+?|\(no author\))\s*=\s*(.+?)\s*<(.*)>\s*$/
>> 
>> for parsing the authors file so that the same (slightly more lenient)
>> regex is used in both cases.
>> 
>> Reported-by: Till Schäfer <till2.schaefer@tu-dortmund.de>
>> Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
>
> Thanks.
> Signed-off-by: Eric Wong <normalperson@yhbt.net>
>
> And pushed to master of git://bogomips.org/git-svn
> (commit f7c6de0ea1bd5722a1181c6279676c6831b38a34)

Thanks, pulled.

Re: [PATCH] git-svn: parse authors file more leniently

[Michael J Gruber]

Eric Wong venit, vidit, dixit 10.09.2015 20:08:
> Michael J Gruber <git@drmicha.warpmail.net> wrote:
>> Instead, make git svn uses the perl regex
>>
>> /^(.+?|\(no author\))\s*=\s*(.+?)\s*<(.*)>\s*$/
>>
>> for parsing the authors file so that the same (slightly more lenient)
>> regex is used in both cases.
>>
>> Reported-by: Till Schäfer <till2.schaefer@tu-dortmund.de>
>> Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
> 
> Thanks.
> Signed-off-by: Eric Wong <normalperson@yhbt.net>
> 
> And pushed to master of git://bogomips.org/git-svn
> (commit f7c6de0ea1bd5722a1181c6279676c6831b38a34)
> 
> By the way, I also had some other patches sitting around for you.
> Did you ever have time to revisit them?  (I haven't)
> 
>       t/lib-httpd: load mod_unixd
>       t/lib-git-svn: check same httpd module dirs as lib-httpd
> 

Also "from me".

Short answer: No

If I remember correctly, they were correct bit not complete in the sense
that on a standard Fedora install (with newer apache), svn tests still
wouldn't run over http. But I/we learned that those tests were simply
run over local file protocol instead when svn over http didn't work. On
a standard debian install (which apparantly has non-standard, thus
downwards compatible apache config) everything was fine with or without
those patches.

I still plan to look at them when I find time. (I'll be retiring
sometime between 20 and 30 years from now, so there's hope.)

Michael