[PATCHv2 0/2] git p4: use "move" command for renames

[PATCHv2 0/2] git p4: use "move" command for renames

[Pete Wyckoff]

Difference from v1:
    * Remove "tee" usage in t9814, just send output to a file
      and grep it directly.

Recent p4 supports a "move" command that records explicitly that
a file was moved from one place to another.  It can be changed a bit
during the move, too.  Use this feature, if it exists, when renames
are detected.

Gary sent these patches months ago, and I've been sitting on them
far too long.  Was hoping to move some other work out first, but it
was not ready.

These commits are on origin/next, as they depend on changes
in pw/git-p4-tests that was merged into next on Jul 3.  They
do not conflict with the other series in flight, "notice Jobs: ...".


Gary Gibbons (2):
  git p4: refactor diffOpts calculation
  git p4: add support for 'p4 move' in P4Submit

 Documentation/git-p4.txt | 10 +++---
 git-p4.py                | 86 ++++++++++++++++++++++++++++++++----------------
 t/t9814-git-p4-rename.sh | 16 ++++-----
 3 files changed, 72 insertions(+), 40 deletions(-)

-- 
1.7.11.1.72.g3344471

[PATCHv2 1/2] git p4: refactor diffOpts calculation

[Pete Wyckoff]

From: Gary Gibbons <ggibbons@perforce.com>

P4Submit.applyCommit()

To avoid recalculating the same diffOpts for each commit, move it
out of applyCommit() and into the top-level run().  Also fix a bug
in that code which interpreted the value of detectRenames as a
string rather than as a boolean.

[pw: fix documentation, rearrange code a bit]

Signed-off-by: Gary Gibbons <ggibbons@perforce.com>
Signed-off-by: Pete Wyckoff <pw@padd.com>
---
 Documentation/git-p4.txt | 10 ++++++----
 git-p4.py                | 52 +++++++++++++++++++++++++++++-------------------
 2 files changed, 38 insertions(+), 24 deletions(-)

diff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt
index fe1f49b..8228f33 100644
--- a/Documentation/git-p4.txt
+++ b/Documentation/git-p4.txt
@@ -255,7 +255,7 @@ These options can be used to modify 'git p4 submit' behavior.
 	p4.  By default, this is the most recent p4 commit reachable
 	from 'HEAD'.
 
--M[<n>]::
+-M::
 	Detect renames.  See linkgit:git-diff[1].  Renames will be
 	represented in p4 using explicit 'move' operations.  There
 	is no corresponding option to detect copies, but there are
@@ -465,13 +465,15 @@ git-p4.useClientSpec::
 Submit variables
 ~~~~~~~~~~~~~~~~
 git-p4.detectRenames::
-	Detect renames.  See linkgit:git-diff[1].
+	Detect renames.  See linkgit:git-diff[1].  This can be true,
+	false, or a score as expected by 'git diff -M'.
 
 git-p4.detectCopies::
-	Detect copies.  See linkgit:git-diff[1].
+	Detect copies.  See linkgit:git-diff[1].  This can be true,
+	false, or a score as expected by 'git diff -C'.
 
 git-p4.detectCopiesHarder::
-	Detect copies harder.  See linkgit:git-diff[1].
+	Detect copies harder.  See linkgit:git-diff[1].  A boolean.
 
 git-p4.preserveUser::
 	On submit, re-author changes to reflect the git author,
diff --git a/git-p4.py b/git-p4.py
index f895a24..5fe509f 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -1046,27 +1046,8 @@ class P4Submit(Command, P4UserMap):
 
         (p4User, gitEmail) = self.p4UserForCommit(id)
 
-        if not self.detectRenames:
-            # If not explicitly set check the config variable
-            self.detectRenames = gitConfig("git-p4.detectRenames")
-
-        if self.detectRenames.lower() == "false" or self.detectRenames == "":
-            diffOpts = ""
-        elif self.detectRenames.lower() == "true":
-            diffOpts = "-M"
-        else:
-            diffOpts = "-M%s" % self.detectRenames
-
-        detectCopies = gitConfig("git-p4.detectCopies")
-        if detectCopies.lower() == "true":
-            diffOpts += " -C"
-        elif detectCopies != "" and detectCopies.lower() != "false":
-            diffOpts += " -C%s" % detectCopies
 
-        if gitConfig("git-p4.detectCopiesHarder", "--bool") == "true":
-            diffOpts += " --find-copies-harder"
-
-        diff = read_pipe_lines("git diff-tree -r %s \"%s^\" \"%s\"" % (diffOpts, id, id))
+        diff = read_pipe_lines("git diff-tree -r %s \"%s^\" \"%s\"" % (self.diffOpts, id, id))
         filesToAdd = set()
         filesToDelete = set()
         editedFiles = set()
@@ -1433,6 +1414,37 @@ class P4Submit(Command, P4UserMap):
         if self.preserveUser:
             self.checkValidP4Users(commits)
 
+        #
+        # Build up a set of options to be passed to diff when
+        # submitting each commit to p4.
+        #
+        if self.detectRenames:
+            # command-line -M arg
+            self.diffOpts = "-M"
+        else:
+            # If not explicitly set check the config variable
+            detectRenames = gitConfig("git-p4.detectRenames")
+
+            if detectRenames.lower() == "false" or detectRenames == "":
+                self.diffOpts = ""
+            elif detectRenames.lower() == "true":
+                self.diffOpts = "-M"
+            else:
+                self.diffOpts = "-M%s" % detectRenames
+
+        # no command-line arg for -C or --find-copies-harder, just
+        # config variables
+        detectCopies = gitConfig("git-p4.detectCopies")
+        if detectCopies.lower() == "false" or detectCopies == "":
+            pass
+        elif detectCopies.lower() == "true":
+            self.diffOpts += " -C"
+        else:
+            self.diffOpts += " -C%s" % detectCopies
+
+        if gitConfig("git-p4.detectCopiesHarder", "--bool") == "true":
+            self.diffOpts += " --find-copies-harder"
+
         while len(commits) > 0:
             commit = commits[0]
             commits = commits[1:]
-- 
1.7.11.1.72.g3344471

[PATCHv2 2/2] git p4: add support for 'p4 move' in P4Submit

[Pete Wyckoff]

From: Gary Gibbons <ggibbons@perforce.com>

For -M option (detectRenames) in P4Submit, use 'p4 move' rather
than 'p4 integrate'.  Check Perforce server for exisitence of
'p4 move' and use it if present, otherwise revert to 'p4 integrate'.

[pw: wildcard-encode src/dest, add/update tests, tweak code]

Signed-off-by: Gary Gibbons <ggibbons@perforce.com>
Signed-off-by: Pete Wyckoff <pw@padd.com>
---
 git-p4.py                | 34 ++++++++++++++++++++++++++--------
 t/t9814-git-p4-rename.sh | 16 ++++++++--------
 2 files changed, 34 insertions(+), 16 deletions(-)

diff --git a/git-p4.py b/git-p4.py
index 5fe509f..b79e6f0 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -120,6 +120,15 @@ def p4_read_pipe_lines(c):
     real_cmd = p4_build_cmd(c)
     return read_pipe_lines(real_cmd)
 
+def p4_has_command(cmd):
+    """Ask p4 for help on this command.  If it returns an error, the
+       command does not exist in this version of p4."""
+    real_cmd = p4_build_cmd(["help", cmd])
+    p = subprocess.Popen(real_cmd, stdout=subprocess.PIPE,
+                                   stderr=subprocess.PIPE)
+    p.communicate()
+    return p.returncode == 0
+
 def system(cmd):
     expand = isinstance(cmd,basestring)
     if verbose:
@@ -157,6 +166,9 @@ def p4_revert(f):
 def p4_reopen(type, f):
     p4_system(["reopen", "-t", type, wildcard_encode(f)])
 
+def p4_move(src, dest):
+    p4_system(["move", "-k", wildcard_encode(src), wildcard_encode(dest)])
+
 #
 # Canonicalize the p4 type and return a tuple of the
 # base type, plus any modifiers.  See "p4 help filetypes"
@@ -850,6 +862,7 @@ class P4Submit(Command, P4UserMap):
         self.preserveUser = gitConfig("git-p4.preserveUser").lower() == "true"
         self.isWindows = (platform.system() == "Windows")
         self.exportLabels = False
+        self.p4HasMoveCommand = p4_has_command("move") 
 
     def check(self):
         if len(p4CmdList("opened ...")) > 0:
@@ -1046,7 +1059,6 @@ class P4Submit(Command, P4UserMap):
 
         (p4User, gitEmail) = self.p4UserForCommit(id)
 
-
         diff = read_pipe_lines("git diff-tree -r %s \"%s^\" \"%s\"" % (self.diffOpts, id, id))
         filesToAdd = set()
         filesToDelete = set()
@@ -1087,17 +1099,23 @@ class P4Submit(Command, P4UserMap):
                 editedFiles.add(dest)
             elif modifier == "R":
                 src, dest = diff['src'], diff['dst']
-                p4_integrate(src, dest)
-                if diff['src_sha1'] != diff['dst_sha1']:
-                    p4_edit(dest)
+                if self.p4HasMoveCommand:
+                    p4_edit(src)        # src must be open before move
+                    p4_move(src, dest)  # opens for (move/delete, move/add)
                 else:
-                    pureRenameCopy.add(dest)
+                    p4_integrate(src, dest)
+                    if diff['src_sha1'] != diff['dst_sha1']:
+                        p4_edit(dest)
+                    else:
+                        pureRenameCopy.add(dest)
                 if isModeExecChanged(diff['src_mode'], diff['dst_mode']):
-                    p4_edit(dest)
+                    if not self.p4HasMoveCommand:
+                        p4_edit(dest)   # with move: already open, writable
                     filesToChangeExecBit[dest] = diff['dst_mode']
-                os.unlink(dest)
+                if not self.p4HasMoveCommand:
+                    os.unlink(dest)
+                    filesToDelete.add(src)
                 editedFiles.add(dest)
-                filesToDelete.add(src)
             else:
                 die("unknown modifier %s for %s" % (modifier, path))
 
diff --git a/t/t9814-git-p4-rename.sh b/t/t9814-git-p4-rename.sh
index 84fffb3..3bf1224 100755
--- a/t/t9814-git-p4-rename.sh
+++ b/t/t9814-git-p4-rename.sh
@@ -77,16 +77,16 @@ test_expect_success 'detect renames' '
 		git commit -a -m "Rename file1 to file4" &&
 		git diff-tree -r -M HEAD &&
 		git p4 submit &&
-		p4 filelog //depot/file4 &&
-		p4 filelog //depot/file4 | test_must_fail grep -q "branch from" &&
+		p4 filelog //depot/file4 >filelog &&
+		! grep " from //depot" filelog &&
 
 		git mv file4 file5 &&
 		git commit -a -m "Rename file4 to file5" &&
 		git diff-tree -r -M HEAD &&
 		git config git-p4.detectRenames true &&
 		git p4 submit &&
-		p4 filelog //depot/file5 &&
-		p4 filelog //depot/file5 | grep -q "branch from //depot/file4" &&
+		p4 filelog //depot/file5 >filelog &&
+		grep " from //depot/file4" filelog &&
 
 		git mv file5 file6 &&
 		echo update >>file6 &&
@@ -97,8 +97,8 @@ test_expect_success 'detect renames' '
 		test -n "$level" && test "$level" -gt 0 && test "$level" -lt 98 &&
 		git config git-p4.detectRenames $(($level + 2)) &&
 		git p4 submit &&
-		p4 filelog //depot/file6 &&
-		p4 filelog //depot/file6 | test_must_fail grep -q "branch from" &&
+		p4 filelog //depot/file6 >filelog &&
+		! grep " from //depot" filelog &&
 
 		git mv file6 file7 &&
 		echo update >>file7 &&
@@ -109,8 +109,8 @@ test_expect_success 'detect renames' '
 		test -n "$level" && test "$level" -gt 2 && test "$level" -lt 100 &&
 		git config git-p4.detectRenames $(($level - 2)) &&
 		git p4 submit &&
-		p4 filelog //depot/file7 &&
-		p4 filelog //depot/file7 | grep -q "branch from //depot/file6"
+		p4 filelog //depot/file7 >filelog &&
+		grep " from //depot/file6" filelog
 	)
 '
 
-- 
1.7.11.1.72.g3344471

Re: [PATCHv2 2/2] git p4: add support for 'p4 move' in P4Submit

[Vitor Antunes]

Pete Wyckoff <pw <at> padd.com> writes:

> 
> From: Gary Gibbons <ggibbons <at> perforce.com>
> 
> For -M option (detectRenames) in P4Submit, use 'p4 move' rather
> than 'p4 integrate'.  Check Perforce server for exisitence of
> 'p4 move' and use it if present, otherwise revert to 'p4 integrate'.
> 

Hi Pete,

I've just been hit by a situation where this command is available but is
disabled in the server. I don't know what is the best approach to avoid
this issue.

Thanks,
Vitor

Re: [PATCHv2 2/2] git p4: add support for 'p4 move' in P4Submit

[Pete Wyckoff]

vitor.hda@gmail.com wrote on Mon, 05 Nov 2012 17:37 +0000:
> Pete Wyckoff <pw <at> padd.com> writes:
> 
> > 
> > From: Gary Gibbons <ggibbons <at> perforce.com>
> > 
> > For -M option (detectRenames) in P4Submit, use 'p4 move' rather
> > than 'p4 integrate'.  Check Perforce server for exisitence of
> > 'p4 move' and use it if present, otherwise revert to 'p4 integrate'.
> > 
> 
> Hi Pete,
> 
> I've just been hit by a situation where this command is available but is
> disabled in the server. I don't know what is the best approach to avoid
> this issue.

Really?  The command exists in the server because it returns the
text output for "p4 help move".  But "p4 move" itself fails
because it is somehow disabled in the server?

I didn't even know it was possible to administratively disable
commands.

What's the actual error message?  And versions of your client and
server (p4 -V, p4d -V, p4 info).

Any ideas Gary?

		-- Pete

Re: [PATCHv2 2/2] git p4: add support for 'p4 move' in P4Submit

[Vitor Antunes]

On Mon, Nov 5, 2012 at 5:57 PM, Pete Wyckoff <pw@padd.com> wrote:
> vitor.hda@gmail.com wrote on Mon, 05 Nov 2012 17:37 +0000:
>> Hi Pete,
>>
>> I've just been hit by a situation where this command is available but is
>> disabled in the server. I don't know what is the best approach to avoid
>> this issue.
>
> Really?  The command exists in the server because it returns the
> text output for "p4 help move".  But "p4 move" itself fails
> because it is somehow disabled in the server?
>
> I didn't even know it was possible to administratively disable
> commands.
>
> What's the actual error message?  And versions of your client and
> server (p4 -V, p4d -V, p4 info).
>
> Any ideas Gary?

I don't feel comfortable in testing this again because I'm working in a
production server. Can Gary provide any details on this type of
configuration on the server side?

Vitor

[PATCH] git p4: handle servers without move support

[Pete Wyckoff]

Support for the "p4 move" command was added in 8e9497c (git p4:
add support for 'p4 move' in P4Submit, 2012-07-12), which checks
to make sure the client and server support the command.

But older versions of p4d may not handle the "-k" argument, and
newer p4d allow disabling "p4 move" with a configurable.  Check
for both these cases by testing a p4 move command on bogus
filenames and looking for strings in the error messages.

Signed-off-by: Pete Wyckoff <pw@padd.com>
---
vitor.hda@gmail.com wrote on Wed, 07 Nov 2012 21:06 +0000:
> On Mon, Nov 5, 2012 at 5:57 PM, Pete Wyckoff <pw@padd.com> wrote:
> > vitor.hda@gmail.com wrote on Mon, 05 Nov 2012 17:37 +0000:
> >> Hi Pete,
> >>
> >> I've just been hit by a situation where this command is available but is
> >> disabled in the server. I don't know what is the best approach to avoid
> >> this issue.
> >
> > Really?  The command exists in the server because it returns the
> > text output for "p4 help move".  But "p4 move" itself fails
> > because it is somehow disabled in the server?
> >
> > I didn't even know it was possible to administratively disable
> > commands.
> >
> > What's the actual error message?  And versions of your client and
> > server (p4 -V, p4d -V, p4 info).
> >
> > Any ideas Gary?
> 
> I don't feel comfortable in testing this again because I'm working in a
> production server. Can Gary provide any details on this type of
> configuration on the server side?

Gary explained how to administratively disable "p4 move".  This
patch checks for that condition, and "p4 move -k", which is
missing in p4d 2009.1.

I'm not super happy with it, due to the string-searching in the
error messages.  But I can't think of a better way:  if "p4 move
-k src dest" returns 1, is it because the server doesn't support
it?  arguments were wrong?  printer on fire?

Vitor, can you test that it works for you?  It's probably worth
putting in the code just so no one else finds the same mysterious
failure.

		-- Pete

 git-p4.py                | 21 ++++++++++++++++++++-
 t/t9814-git-p4-rename.sh | 35 +++++++++++++++++++++++++++++++++++
 2 files changed, 55 insertions(+), 1 deletion(-)

diff --git a/git-p4.py b/git-p4.py
index 882b1bb..417a66a 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -129,6 +129,25 @@ def p4_has_command(cmd):
     p.communicate()
     return p.returncode == 0
 
+def p4_has_move_command():
+    """See if the move command exists, that it supports -k, and that
+       it has not been administratively disabled.  The arguments
+       must be correct, but the filenames do not have to exist.  Use
+       ones with wildcards so even if they exist, it will fail."""
+
+    if not p4_has_command("move"):
+        return False
+    cmd = p4_build_cmd(["move", "-k", "@from", "@to"])
+    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = p.communicate()
+    # return code will be 1 in either case
+    if err.find("Invalid option") >= 0:
+        return False
+    if err.find("disabled") >= 0:
+        return False
+    # assume it failed because @... was invalid changelist
+    return True
+
 def system(cmd):
     expand = isinstance(cmd,basestring)
     if verbose:
@@ -871,7 +890,7 @@ class P4Submit(Command, P4UserMap):
         self.conflict_behavior = None
         self.isWindows = (platform.system() == "Windows")
         self.exportLabels = False
-        self.p4HasMoveCommand = p4_has_command("move")
+        self.p4HasMoveCommand = p4_has_move_command()
 
     def check(self):
         if len(p4CmdList("opened ...")) > 0:
diff --git a/t/t9814-git-p4-rename.sh b/t/t9814-git-p4-rename.sh
index 3bf1224..be802e0 100755
--- a/t/t9814-git-p4-rename.sh
+++ b/t/t9814-git-p4-rename.sh
@@ -199,6 +199,41 @@ test_expect_success 'detect copies' '
 	)
 '
 
+# See if configurables can be set, and in particular if the run.move.allow
+# variable exists, which allows admins to disable the "p4 move" command.
+test_expect_success 'p4 configure command and run.move.allow are available' '
+	p4 configure show run.move.allow >out ; retval=$? &&
+	test $retval = 0 &&
+	{
+		egrep ^run.move.allow: out &&
+		test_set_prereq P4D_HAVE_CONFIGURABLE_RUN_MOVE_ALLOW ||
+		true
+	} || true
+'
+
+# If move can be disabled, turn it off and test p4 move handling
+test_expect_success P4D_HAVE_CONFIGURABLE_RUN_MOVE_ALLOW \
+		    'do not use p4 move when administratively disabled' '
+	test_when_finished "p4 configure set run.move.allow=1" &&
+	p4 configure set run.move.allow=0 &&
+	(
+		cd "$cli" &&
+		echo move-disallow-file >move-disallow-file &&
+		p4 add move-disallow-file &&
+		p4 submit -d "add move-disallow-file"
+	) &&
+	test_when_finished cleanup_git &&
+	git p4 clone --dest="$git" //depot &&
+	(
+		cd "$git" &&
+		git config git-p4.skipSubmitEdit true &&
+		git config git-p4.detectRenames true &&
+		git mv move-disallow-file move-disallow-file-moved &&
+		git commit -m "move move-disallow-file" &&
+		git p4 submit
+	)
+'
+
 test_expect_success 'kill p4d' '
 	kill_p4d
 '
-- 
1.7.12.1.457.g468b3ef