[PATCH] git-mv is not able to handle big directories

[PATCH] git-mv is not able to handle big directories

[Alexander Litvinov]

[-- Attachment #1: Type: text/plain, Size: 473 bytes --]

When moving directory with large number of files git-mv says:
> git-mv jsp* .
Can't exec "git-update-index": Argument list too long at /usr/local/bin/git-mv 
line 193.
git-update-index failed to add new names with code -1

This patch fixes this by building list of files with limited len (currently 
5000) and executing git-update-index few times until all files will be 
processed. I don't know how to determinate limit of command line but 5000 
seems safe enougth to me.

[-- Attachment #2: git-mv.perl.patch --]
[-- Type: text/x-diff, Size: 1492 bytes --]

--- git-mv.perl.orig	2005-11-23 11:24:10.000000000 +0600
+++ git-mv.perl	2005-11-23 11:33:31.000000000 +0600
@@ -185,13 +185,36 @@
 }
 	
 my $rc;
-if (scalar @changedfiles >0) {
-	$rc = system("git-update-index","--",@changedfiles);
+while (scalar @changedfiles >0) {
+	my @toHandle = ();
+	my $len = 0;
+	while ($len < 5000 && scalar(@changedfiles) >0) {
+		my $f = pop(@changedfiles);
+		$len += length($f) + 1;
+		push(@toHandle, $f);
+	}
+	$rc = system("git-update-index","--",@toHandle);
 	die "git-update-index failed to update changed files with code $?\n" if $rc;
 }
-if (scalar @addedfiles >0) {
-	$rc = system("git-update-index","--add","--",@addedfiles);
+while (scalar @addedfiles >0) {
+	my @toHandle = ();
+	my $len = 0;
+	while ($len < 5000 && scalar(@addedfiles) >0) {
+		my $f = pop(@addedfiles);
+		$len += length($f) + 1;
+		push(@toHandle, $f);
+	}
+	$rc = system("git-update-index","--add","--",@toHandle);
 	die "git-update-index failed to add new names with code $?\n" if $rc;
 }
-$rc = system("git-update-index","--remove","--",@deletedfiles);
-die "git-update-index failed to remove old names with code $?\n" if $rc;
+while (scalar @deletedfiles > 0) {
+	my @toHandle = ();
+	my $len = 0;
+	while ($len < 5000 && scalar(@deletedfiles) >0) {
+		my $f = pop(@deletedfiles);
+		$len += length($f) + 1;
+		push(@toHandle, $f);
+	}
+	$rc = system("git-update-index","--remove","--",@toHandle);
+	die "git-update-index failed to remove old names with code $?\n" if $rc;
+}

Re: [PATCH] git-mv is not able to handle big directories

[Junio C Hamano]

Alexander Litvinov <lan@ac-sw.com> writes:

> When moving directory with large number of files git-mv says:
>> git-mv jsp* .
> Can't exec "git-update-index": Argument list too long at /usr/local/bin/git-mv 
> line 193.
> git-update-index failed to add new names with code -1
>
> This patch fixes this by building list of files with limited len (currently 
> 5000) and executing git-update-index few times until all files will be 
> processed. I don't know how to determinate limit of command line but 5000 
> seems safe enougth to me.

Two comments.

(1) the argument limit is enforced by the operating system in
    bytes (including environment size unfortunately) so we might
    want to count bytes not number of paths.  I heard GNU xargs
    uses 131072 as the default limit.

(2) I wonder if we can detect this particular failure case and
    then fall back on splitting the arguments dynamically, maybe
    something like this:

	sub xargs_system {
        	my ($cmd, @args) = @_;
                my $rc = system(@$cmd, @args);
                if ($rc == 'argument list too long error') {
                	my (@args0) = splice(@args, 0, @args/2);
			$rc = xargs_system($cmd, @args0);
                        return $c if ($rc);
                        return xargs_system($cmd, @args);
		}
		return $rc;
	}

    and:

	$rc = xargs_system([qw(git-update-index --)], @changedfiles);
	$rc = xargs_system([qw(git-update-index --add --)], @addedfiles);
	...

Re: [PATCH] git-mv is not able to handle big directories

[Junio C Hamano]

Junio C Hamano <junkio@cox.net> writes:

> Two comments.
>
> (1) the argument limit is enforced by the operating system in
> ...
> (2) I wonder if we can detect this particular failure case and
> ...

(3) Even better, 'git-update-index -z --stdin'

	if (@changedfiles) {
            open my $oh, qw(|- git-update-index -z --stdin)
                    or die "oops";
            for (@changedfiles) {
                print $oh "$_\0";
	    }
	    close $oh;
	}



	JC "added too many features that myself cannot remember" Hamano

Re: [PATCH] git-mv is not able to handle big directories

[Randal L. Schwartz]

>>>>> "Junio" == Junio C Hamano <junkio@cox.net> writes:

Junio>             open my $oh, qw(|- git-update-index -z --stdin)
Junio>                     or die "oops";

This is Perl 5.6 or later.  Breaks on Perl 5.5, which is still in use
in some places.

To be compatible with 5.5, you have to create a handle explicitly:

        require IO::Handle;
        my $oh = IO::Handle->new;
        open $oh, qw(...) ...;

That works all the way back to 5.4, which is the earliest Perl
supported by the core team.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Re: [PATCH] git-mv is not able to handle big directories

[Junio C Hamano]

merlyn@stonehenge.com (Randal L. Schwartz) writes:

>>>>>> "Junio" == Junio C Hamano <junkio@cox.net> writes:
>
> Junio>             open my $oh, qw(|- git-update-index -z --stdin)
> Junio>                     or die "oops";
>
> This is Perl 5.6 or later.  Breaks on Perl 5.5, which is still in use
> in some places.

I should have known better than posting Perl code to the list
where a real guru is lurking and making afool of myself ;-).

Thanks for the advice.  How much do we care about 5.5?  IOW, is
"in some places" wide enough to matter?

git-cvsimport, git-svnimport and git-shortlog share the same
problem (svnimport declares that it wants 5.8).  perl58delta.pod
does say list form of open for pipes is new in that version.  So
what I wrote above requires 5.8 or better, perhaps?  cvsimport
uses that same structure.  That is doubly Ouch.

Re: [PATCH] git-mv is not able to handle big directories

[Ryan Anderson]

On Wed, Nov 23, 2005 at 12:37:59AM -0800, Junio C Hamano wrote:
> merlyn@stonehenge.com (Randal L. Schwartz) writes:
> 
> >>>>>> "Junio" == Junio C Hamano <junkio@cox.net> writes:
> >
> > Junio>             open my $oh, qw(|- git-update-index -z --stdin)
> > Junio>                     or die "oops";
> >
> > This is Perl 5.6 or later.  Breaks on Perl 5.5, which is still in use
> > in some places.
> 
> I should have known better than posting Perl code to the list
> where a real guru is lurking and making afool of myself ;-).
> 
> Thanks for the advice.  How much do we care about 5.5?  IOW, is
> "in some places" wide enough to matter?
> 
> git-cvsimport, git-svnimport and git-shortlog share the same
> problem (svnimport declares that it wants 5.8).  perl58delta.pod
> does say list form of open for pipes is new in that version.  So
> what I wrote above requires 5.8 or better, perhaps?  cvsimport
> uses that same structure.  That is doubly Ouch.

No, you're not using the list form for pipes.  I use that in
git-graft-ripple:

	open(P,"-|","git-rev-list","--parents","--merge-order",$range)

For the kernel, requiring 5.8 shouldn't be a big issue. I suspect it's
really only the commercial Unixes where requiring 5.8 would be annoying.

Randal, is my guess even remotely accurate?

-- 

Ryan Anderson
  sometimes Pug Majere

Perl version support (was Re: [PATCH] git-mv is not able to handle big directories)

[Randal L. Schwartz]

>>>>> "Ryan" == Ryan Anderson <ryan@michonline.com> writes:

Ryan> For the kernel, requiring 5.8 shouldn't be a big issue. I suspect it's
Ryan> really only the commercial Unixes where requiring 5.8 would be annoying.

Ryan> Randal, is my guess even remotely accurate?

I'd say that 50% of the Perl-using population is at 5.6, with 25% each
at 5.5 and 5.8.  Those on 5.5 are generally unable to upgrade Perl
for corporate reasons.

Targetting Perl 5.6 would assist broad acceptance of git for the
typical commercial end user.  Targetting 5.5 where possible would
ensure practical success for everyone.

However, I have not seen the "target market" of git discussed yet
(I came late to the party), so if support for 5.6 (or 5.5) is not chosen,
it merely limits the market.

If you'd like, I can review all the Perl code with a tool that
determines the minimum Perl version, and provide patches to bring the
code to 5.5 level.  But if it's likely that someone will say "this
is not important to us", I'd rather not waste my time. :)

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Re: Perl version support

[Junio C Hamano]

merlyn@stonehenge.com (Randal L. Schwartz) writes:

> I'd say that 50% of the Perl-using population is at 5.6, with 25% each
> at 5.5 and 5.8.  Those on 5.5 are generally unable to upgrade Perl
> for corporate reasons.

I think the target market is people who use whatever POSIXy
systems for their own work.  To them, git and its dependencies
are merely tools to get their job done, and while they might be
capable and even willing to match the version requirements if it
is absolutely necessary, they'd rather spend their time on what
they do best (i.e. write software or documentation or whatever,
whose changes are to be managed by git), than installing or
upgrading base software just to satisfy git.  And I'd like to
see their time spent on what they do best, instead of caring and
feeding of git, too.

So in that sense, excluding 75% of people with "5.8 only" is not
acceptable at all, and covering 75% with "5.6 or better" might
be good enough, at least as a starter.

> If you'd like, I can review all the Perl code with a tool that
> determines the minimum Perl version, and provide patches to bring the
> code to 5.5 level.

If it is not too much trouble, I'd appreciate it, at least the
first half that can be done without taking too much of your time.
Is it a mechanical "lint" like thing, that says "line 47 you
have list form of pipe open -- that's 5.8 and better"?

Re: Perl version support

[Randal L. Schwartz]

>>>>> "Junio" == Junio C Hamano <junkio@cox.net> writes:

Junio> I think the target market is people who use whatever POSIXy
Junio> systems for their own work.  To them, git and its dependencies
Junio> are merely tools to get their job done, and while they might be
Junio> capable and even willing to match the version requirements if it
Junio> is absolutely necessary, they'd rather spend their time on what
Junio> they do best (i.e. write software or documentation or whatever,
Junio> whose changes are to be managed by git), than installing or
Junio> upgrading base software just to satisfy git.  And I'd like to
Junio> see their time spent on what they do best, instead of caring and
Junio> feeding of git, too.

Yes, this is why I keep pointing out GNU tools being used instead of
POSIXy versions. :)  Mostly because they break on either my Darwin
laptop or my OpenBSD host server.

Junio> So in that sense, excluding 75% of people with "5.8 only" is not
Junio> acceptable at all, and covering 75% with "5.6 or better" might
Junio> be good enough, at least as a starter.

I think we're in agreement.

>> If you'd like, I can review all the Perl code with a tool that
>> determines the minimum Perl version, and provide patches to bring the
>> code to 5.5 level.

Junio> If it is not too much trouble, I'd appreciate it, at least the
Junio> first half that can be done without taking too much of your time.
Junio> Is it a mechanical "lint" like thing, that says "line 47 you
Junio> have list form of pipe open -- that's 5.8 and better"?

The "perlver" tool installed with the "Perl::MinimumVersion" module
from the CPAN will do about 80% of it.  The other 20% are things
that I can use my brain for.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Re: Perl version support (was Re: [PATCH] git-mv is not able to handle big directories)

[H. Peter Anvin]

Randal L. Schwartz wrote:
> 
> I'd say that 50% of the Perl-using population is at 5.6, with 25% each
> at 5.5 and 5.8.  Those on 5.5 are generally unable to upgrade Perl
> for corporate reasons.
> 
> Targetting Perl 5.6 would assist broad acceptance of git for the
> typical commercial end user.  Targetting 5.5 where possible would
> ensure practical success for everyone.
> 
> However, I have not seen the "target market" of git discussed yet
> (I came late to the party), so if support for 5.6 (or 5.5) is not chosen,
> it merely limits the market.
> 

There are a lot of Perl modules we use, so limiting it to 5.5 is 
probably a showstopper.

I'm very surprised you say that 5.6 is more prevalent than 5.8.

	-hpa

Re: Perl version support (was Re: [PATCH] git-mv is not able to handle big directories)

[Randal L. Schwartz]

>>>>> "H" == H Peter Anvin <hpa@zytor.com> writes:

H> There are a lot of Perl modules we use, so limiting it to 5.5 is
H> probably a showstopper.

Hmm.  I should look at that then.  Most Perl Modules are 5.5
compatible, unless they've been written by naive people recently. :)

H> I'm very surprised you say that 5.6 is more prevalent than 5.8.

For individual early adopters, 5.8 is nearly universal.  But a lot of
my corporate clients upgrade *very* slowly, and are on 5.6 for now.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Re: Perl version support (was Re: [PATCH] git-mv is not able to handle big directories)

[Morten Welinder]

> I'm very surprised you say that 5.6 is more prevalent than 5.8.

You are overestimating the update-eagerness of the world.

SunOS 5.8 seems to come with 5.005_03
SunOS 5.9 seems to come with v5.6.1.

("Seems" because I am only 99% sure no-one around here has mucked with
the perl in /usr/bin/.)

M.

Re: Perl version support (was Re: [PATCH] git-mv is not able to handle big directories)

[Ryan Anderson]

[-- Attachment #1: Type: text/plain, Size: 1182 bytes --]

Randal L. Schwartz wrote:
>>>>>>"Ryan" == Ryan Anderson <ryan@michonline.com> writes:
> Ryan> For the kernel, requiring 5.8 shouldn't be a big issue. I suspect it's
> Ryan> really only the commercial Unixes where requiring 5.8 would be annoying.
> 
> Ryan> Randal, is my guess even remotely accurate?
> 
> I'd say that 50% of the Perl-using population is at 5.6, with 25% each
> at 5.5 and 5.8.  Those on 5.5 are generally unable to upgrade Perl
> for corporate reasons.
> 
> Targetting Perl 5.6 would assist broad acceptance of git for the
> typical commercial end user.  Targetting 5.5 where possible would
> ensure practical success for everyone.
> 
> However, I have not seen the "target market" of git discussed yet
> (I came late to the party), so if support for 5.6 (or 5.5) is not chosen,
> it merely limits the market.

Well, I think the general target market[1] for Linux probably applies
here, as well.  On the other hand, the users limited to Perl 5.5 for
corporate reasons are probably also prevented from installing new tools
for general use, so the difference may not matter very much.

[1] - "World domination", of course.

-- 

Ryan Anderson
  sometimes Pug Majere

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: Perl version support (was Re: [PATCH] git-mv is not able to handle big directories)

[Andreas Ericsson]

Ryan Anderson wrote:
> Randal L. Schwartz wrote:
> 
>>However, I have not seen the "target market" of git discussed yet
>>(I came late to the party), so if support for 5.6 (or 5.5) is not chosen,
>>it merely limits the market.
> 
> 
> Well, I think the general target market[1] for Linux probably applies
> here, as well.  On the other hand, the users limited to Perl 5.5 for
> corporate reasons are probably also prevented from installing new tools
> for general use, so the difference may not matter very much.
> 
> [1] - "World domination", of course.
> 

Servers with software installation restrictions are rarely used for 
development either and the server side part of git is all C. So long as 
that's true I don't think we need to bother very much with staying 
backwards compatible. It's nice if we manage it, but we shouldn't, imho, 
bend over backwards for the sake of perl 5.5 (or even 5.6).

Just my two öre. :)

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: [PATCH] git-mv is not able to handle big directories

[Junio C Hamano]

Ryan Anderson <ryan@michonline.com> writes:

> On Wed, Nov 23, 2005 at 12:37:59AM -0800, Junio C Hamano wrote:
>> merlyn@stonehenge.com (Randal L. Schwartz) writes:
>> 
>> >>>>>> "Junio" == Junio C Hamano <junkio@cox.net> writes:
>> >
>> > Junio>             open my $oh, qw(|- git-update-index -z --stdin)
>> >
> ...
> No, you're not using the list form for pipes.  I use that in
> git-graft-ripple:
>
> 	open(P,"-|","git-rev-list","--parents","--merge-order",$range)

Oh, I didn't?

Re: [PATCH] git-mv is not able to handle big directories

[Ryan Anderson]

[-- Attachment #1: Type: text/plain, Size: 625 bytes --]

Junio C Hamano wrote:
> Ryan Anderson <ryan@michonline.com> writes:
> 
> 
>>On Wed, Nov 23, 2005 at 12:37:59AM -0800, Junio C Hamano wrote:
>>
>>>merlyn@stonehenge.com (Randal L. Schwartz) writes:
>>>
>>>
>>>>>>>>>"Junio" == Junio C Hamano <junkio@cox.net> writes:
>>>>
>>>>Junio>             open my $oh, qw(|- git-update-index -z --stdin)
>>>>
>>
>>...
>>No, you're not using the list form for pipes.  I use that in
>>git-graft-ripple:
>>
>>	open(P,"-|","git-rev-list","--parents","--merge-order",$range)
> 
> 
> Oh, I didn't?

Sorry, I read after waking up and totally didn't see the "qw" in there.
   You did, my bad.




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH] git-mv is not able to handle big directories

[Randal L. Schwartz]

>>>>> "Alexander" == Alexander Litvinov <lan@ac-sw.com> writes:

Alexander> I have made this change. I also belive it will work on earlier perl but I 
Alexander> can't test this.

This patch looks good back to 5.5.  I didn't execute it either, but
the Perl version installed in my head gave it a thumbs-up. :)

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Re: git-mv is not able to handle directory with one file in it

[Alexander Litvinov]

I have found one error during directory movig: If I move directory with one 
file somewhere in it this script will try to add target directory instead of 
file. Commenting lines starting from 190 solve this error. But I don't 
understand what is the logic behind this case ? Why do target directory 
checked instead of target file ? Should we replace $dst my $destfiles[0] ?

at line 190 in git-mv:
    if (scalar @srcfiles == 1) {
	if ($overwritten{$dst} ==1) {
	    push @changedfiles, $dst;
	} else {
	    push @addedfiles, $dst;
	}
    }
    else {
	push @addedfiles, @dstfiles;
    }

Re: git-mv is not able to handle directory with one file in it

[Andreas Ericsson]

Alexander Litvinov wrote:
> I have found one error during directory movig: If I move directory with one 
> file somewhere in it this script will try to add target directory instead of 
> file.

Are you saying this setup
	foodir/somefile.c     <--file
	newdir/               <--directory

with this command
	git-mv foodir/ newdir

tries to create
	newdir/foodir/somefile.c  <-- directory

or does it create
	newdir/somefile.c         <-- file

?

It should create
	newdir/foodir/somefile.c  <-- file

Otherwise it's misbehaving.

Try running it with the -v switch to make it shout out loud what it's 
trying to do, and then paste the output here.

> Commenting lines starting from 190 solve this error. But I don't 
> understand what is the logic behind this case ? Why do target directory 
> checked instead of target file ? Should we replace $dst my $destfiles[0] ?
> 
> at line 190 in git-mv:
>     if (scalar @srcfiles == 1) {
> 	if ($overwritten{$dst} ==1) {
> 	    push @changedfiles, $dst;
> 	} else {
> 	    push @addedfiles, $dst;
> 	}
>     }
>     else {
> 	push @addedfiles, @dstfiles;
>     }

This is broken. It only checks if there's just one source-file 
regardless of whether or not it resided in a subdirectory. I'm not 
exactly fluent in perl so I can't submit a patch, but the src option 
needs to be directory aware, traverse all source directories and then 
move the files axing everything but the bottom-most dirname to the 
destination directory.

Any takers?

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: git-mv is not able to handle directory with one file in it

[Alexander Litvinov]

On Wednesday 23 November 2005 13:57, Andreas Ericsson wrote:
> Are you saying this setup
> 	foodir/somefile.c     <--file
> 	newdir/               <--directory
>
> with this command
> 	git-mv foodir/ newdir

It is calling git-update-index --add newdir but actiual file structire is 
correct: 
newdir/somefile.c - is a file

Re: git-mv is not able to handle directory with one file in it

[Alexander Litvinov]

On Wednesday 23 November 2005 13:57, Andreas Ericsson wrote:
> This is broken. It only checks if there's just one source-file
> regardless of whether or not it resided in a subdirectory. I'm not
> exactly fluent in perl so I can't submit a patch, but the src option
> needs to be directory aware, traverse all source directories and then
> move the files axing everything but the bottom-most dirname to the
> destination directory.
>
> Any takers?

I still does not understand what this part should do. I know perl enought to 
fix it but I don't understand the logic.

Re: git-mv is not able to handle directory with one file in it

[Josef Weidendorfer]

On Wednesday 23 November 2005 11:21, Alexander Litvinov wrote:
> On Wednesday 23 November 2005 13:57, Andreas Ericsson wrote:
> > This is broken. It only checks if there's just one source-file
> > regardless of whether or not it resided in a subdirectory.

Yes.
For git-update-index we have to use the file inside the directory.

I just sent a patch for this.

Josef

Re: git-mv is not able to handle directory with one file in it

[Josef Weidendorfer]

On Wednesday 23 November 2005 08:26, Alexander Litvinov wrote:
> I have found one error during directory movig: If I move directory with one 
> file somewhere in it this script will try to add target directory instead of 
> file. Commenting lines starting from 190 solve this error. But I don't 
> understand what is the logic behind this case ? Why do target directory 
> checked instead of target file ?

$src/$dst pairs match the arguments given on the command line of git-mv,
And git-mv (as was git-rename) is able to move directory trees around.

It is simplier to keep the granularity at directories, as multiple moves
of directories can not overlap each other. If you do it at file level, you
have to remote and create directories yourself.

Of course, for git it is done at file level (that are the @srcfiles/@dstfiles
arrays), but this is quite trivial as git does not work with directories, but
only with files.

> Should we replace $dst my $destfiles[0] ?

Yes, that was part of my patch.

Josef