[PATCH 0/4] git p4 submit fixes

[PATCH 0/4] git p4 submit fixes

[Pete Wyckoff]

These patches fix some issues associated with submit.  They add a
good chunk of basic testing for the submit path as well.
The last patch was sent to the list back in February, if it
looks familiar.

Pete Wyckoff (4):
  git p4: bring back files in deleted client directory
  git p4: test submit
  git p4: fix writable file after rename or copy
  git p4: submit files with wildcards

 git-p4.py                     |   96 +++++++++++++++++++++++++------------
 t/t9800-git-p4-basic.sh       |  106 +++++++++++++++++++++++++++++++++++++++++
 t/t9807-git-p4-submit.sh      |  101 ++++++++++++++++++++++++++++++++++++++-
 t/t9809-git-p4-client-view.sh |   40 ++++++++++++++--
 4 files changed, 308 insertions(+), 35 deletions(-)

-- 
1.7.10.366.g90ade

[PATCH 1/4] git p4: bring back files in deleted client directory

[Pete Wyckoff]

The code to auto-create the client directory, added in 0591cfa
(git-p4: ensure submit clientPath exists before chdir,
2011-12-09), works when the client directory never existed.

But if the directory is summarily removed without telling p4,
the sync operation will not bring back all the files.  Always
do "sync -f" if the client directory is newly created.

Reported-by: Gary Gibbons <ggibbons@perforce.com>
Signed-off-by: Pete Wyckoff <pw@padd.com>
---
 git-p4.py                |   12 +++++++++---
 t/t9807-git-p4-submit.sh |    7 +++++--
 2 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/git-p4.py b/git-p4.py
index f910d5a..a2eba5d 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -133,8 +133,8 @@ def p4_system(cmd):
 def p4_integrate(src, dest):
     p4_system(["integrate", "-Dt", src, dest])
 
-def p4_sync(path):
-    p4_system(["sync", path])
+def p4_sync(f, *options):
+    p4_system(["sync"] + list(options) + [f])
 
 def p4_add(f):
     p4_system(["add", f])
@@ -1279,12 +1279,18 @@ class P4Submit(Command, P4UserMap):
         self.oldWorkingDirectory = os.getcwd()
 
         # ensure the clientPath exists
+        new_client_dir = False
         if not os.path.exists(self.clientPath):
+            new_client_dir = True
             os.makedirs(self.clientPath)
 
         chdir(self.clientPath)
         print "Synchronizing p4 checkout..."
-        p4_sync("...")
+        if new_client_dir:
+            # old one was destroyed, and maybe nobody told p4
+            p4_sync("...", "-f")
+        else:
+            p4_sync("...")
         self.check()
 
         commits = []
diff --git a/t/t9807-git-p4-submit.sh b/t/t9807-git-p4-submit.sh
index 1541716..2d7dc27 100755
--- a/t/t9807-git-p4-submit.sh
+++ b/t/t9807-git-p4-submit.sh
@@ -28,6 +28,11 @@ test_expect_success 'submit with no client dir' '
 		rm -rf "$cli" &&
 		git config git-p4.skipSubmitEdit true &&
 		git p4 submit
+	) &&
+	(
+		cd "$cli" &&
+		test_path_is_file file1 &&
+		test_path_is_file file2
 	)
 '
 
@@ -44,7 +49,6 @@ test_expect_success 'submit --origin' '
 	) &&
 	(
 		cd "$cli" &&
-		p4 sync &&
 		test_path_is_missing "file3.t" &&
 		test_path_is_file "file4.t"
 	)
@@ -79,7 +83,6 @@ test_expect_success 'submit with master branch name from argv' '
 	) &&
 	(
 		cd "$cli" &&
-		p4 sync &&
 		test_path_is_file "file6.t" &&
 		test_path_is_missing "file7.t"
 	)
-- 
1.7.10.366.g90ade

Re: [PATCH 1/4] git p4: bring back files in deleted client directory

[Luke Diamand]

On 30/04/12 01:57, Pete Wyckoff wrote:
> The code to auto-create the client directory, added in 0591cfa
> (git-p4: ensure submit clientPath exists before chdir,
> 2011-12-09), works when the client directory never existed.
>
> But if the directory is summarily removed without telling p4,
> the sync operation will not bring back all the files.  Always
> do "sync -f" if the client directory is newly created.

I'm possibly missing something obvious here, but 
./t9807-git-p4-submit.sh fails with this change.

Rebasing the current branch onto remotes/p4/master
First, rewinding head to replay your work on top of it...
File file1 doesn't exist. file1
not ok - 3 submit with no client dir

Luke

Re: [PATCH 1/4] git p4: bring back files in deleted client directory

[Pete Wyckoff]

luke@diamand.org wrote on Mon, 30 Apr 2012 07:55 +0100:
> On 30/04/12 01:57, Pete Wyckoff wrote:
> >The code to auto-create the client directory, added in 0591cfa
> >(git-p4: ensure submit clientPath exists before chdir,
> >2011-12-09), works when the client directory never existed.
> >
> >But if the directory is summarily removed without telling p4,
> >the sync operation will not bring back all the files.  Always
> >do "sync -f" if the client directory is newly created.
> 
> I'm possibly missing something obvious here, but
> ./t9807-git-p4-submit.sh fails with this change.
> 
> Rebasing the current branch onto remotes/p4/master
> First, rewinding head to replay your work on top of it...
> File file1 doesn't exist. file1
> not ok - 3 submit with no client dir

I can't figure it out.  Will you help debug a bit?  Something
like this maybe.

diff --git a/t/t9807-git-p4-submit.sh b/t/t9807-git-p4-submit.sh
index f23b4c3..e98cc5e 100755
--- a/t/t9807-git-p4-submit.sh
+++ b/t/t9807-git-p4-submit.sh
@@ -31,6 +31,7 @@ test_expect_success 'submit with no client dir' '
 	) &&
 	(
 		cd "$cli" &&
+		pwd && ls -la && p4 sync && ls -la &&
 		test_path_is_file file1 &&
 		test_path_is_file file2
 	)

And if the other "submit modify" etc. tests don't work, it
could be fallout from this one.  Deleting "$cli" is perhaps
at fault:  permissions, ... ?

Or somehow I completely broke things.

I'll try with a newer version of p4, to see if there's a
dependency there.

Thanks for checking!

		-- Pete

Re: [PATCH 1/4] git p4: bring back files in deleted client directory

[Luke Diamand]

On 30/04/12 13:36, Pete Wyckoff wrote:
> luke@diamand.org wrote on Mon, 30 Apr 2012 07:55 +0100:
>>
>> Rebasing the current branch onto remotes/p4/master
>> First, rewinding head to replay your work on top of it...
>> File file1 doesn't exist. file1
>> not ok - 3 submit with no client dir
>
> I can't figure it out.  Will you help debug a bit?  Something
> like this maybe.

User error.

% cp git-p4.py git-p4

Then it works fine.

Is there a way to get lib-git-p4.sh to check this?

Thanks
Luke

Re: [PATCH 1/4] git p4: bring back files in deleted client directory

[Pete Wyckoff]

luke@diamand.org wrote on Mon, 30 Apr 2012 19:34 +0100:
> On 30/04/12 13:36, Pete Wyckoff wrote:
> >luke@diamand.org wrote on Mon, 30 Apr 2012 07:55 +0100:
> >>
> >>Rebasing the current branch onto remotes/p4/master
> >>First, rewinding head to replay your work on top of it...
> >>File file1 doesn't exist. file1
> >>not ok - 3 submit with no client dir
> >
> >I can't figure it out.  Will you help debug a bit?  Something
> >like this maybe.
> 
> User error.
> 
> % cp git-p4.py git-p4
> 
> Then it works fine.

Whew.  I caught myself forgetting to build a couple
of times too.

"make git-p4" is a bit more official.

> Is there a way to get lib-git-p4.sh to check this?

This does seem appealing, but I'm not sure if it will be
attractive to everybody else.

My thought pattern of what to put in lib-git-p4.sh
went like:

    test git-p4.py -nt git-p4 && {
	echo "You must make git-p4" >&2
	exit 1
    }

to

    test git-p4.py -nt git-p4 && {
	echo "Making git-p4" >&2
	make git-p4
    }

to

    make git-p4

to

    make

and that's when I realized that everybody else has
the same problem too.  Somehow they've gotten used
to rebuilding before rerunning the tests.  I suspect
that we should get used to it too.

That said, it would have saved me some head scratching
on a few occasions.

		-- Pete

[PATCH 2/4] git p4: test submit

[Pete Wyckoff]

Try each of the five diff patterns that might happen during submit.

Signed-off-by: Pete Wyckoff <pw@padd.com>
---
 t/t9807-git-p4-submit.sh |   92 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 92 insertions(+)

diff --git a/t/t9807-git-p4-submit.sh b/t/t9807-git-p4-submit.sh
index 2d7dc27..a2499ee 100755
--- a/t/t9807-git-p4-submit.sh
+++ b/t/t9807-git-p4-submit.sh
@@ -88,6 +88,98 @@ test_expect_success 'submit with master branch name from argv' '
 	)
 '
 
+#
+# Basic submit tests, the five handled cases
+#
+
+test_expect_success 'submit modify' '
+	test_when_finished cleanup_git &&
+	git p4 clone --dest="$git" //depot &&
+	(
+		cd "$git" &&
+		git config git-p4.skipSubmitEdit true &&
+		echo line >>file1 &&
+		git add file1 &&
+		git commit -m file1 &&
+		git p4 submit
+	) &&
+	(
+		cd "$cli" &&
+		test_path_is_file file1 &&
+		test_line_count = 2 file1
+	)
+'
+
+test_expect_success 'submit add' '
+	test_when_finished cleanup_git &&
+	git p4 clone --dest="$git" //depot &&
+	(
+		cd "$git" &&
+		git config git-p4.skipSubmitEdit true &&
+		echo file13 >file13 &&
+		git add file13 &&
+		git commit -m file13 &&
+		git p4 submit
+	) &&
+	(
+		cd "$cli" &&
+		test_path_is_file file13
+	)
+'
+
+test_expect_success 'submit delete' '
+	test_when_finished cleanup_git &&
+	git p4 clone --dest="$git" //depot &&
+	(
+		cd "$git" &&
+		git config git-p4.skipSubmitEdit true &&
+		git rm file4.t &&
+		git commit -m "delete file4.t" &&
+		git p4 submit
+	) &&
+	(
+		cd "$cli" &&
+		test_path_is_missing file4.t
+	)
+'
+
+test_expect_success 'submit copy' '
+	test_when_finished cleanup_git &&
+	git p4 clone --dest="$git" //depot &&
+	(
+		cd "$git" &&
+		git config git-p4.skipSubmitEdit true &&
+		git config git-p4.detectCopies true &&
+		git config git-p4.detectCopiesHarder true &&
+		cp file5.t file5.ta &&
+		git add file5.ta &&
+		git commit -m "copy to file5.ta" &&
+		git p4 submit
+	) &&
+	(
+		cd "$cli" &&
+		test_path_is_file file5.ta
+	)
+'
+
+test_expect_success 'submit rename' '
+	test_when_finished cleanup_git &&
+	git p4 clone --dest="$git" //depot &&
+	(
+		cd "$git" &&
+		git config git-p4.skipSubmitEdit true &&
+		git config git-p4.detectRenames true &&
+		git mv file6.t file6.ta &&
+		git commit -m "rename file6.t to file6.ta" &&
+		git p4 submit
+	) &&
+	(
+		cd "$cli" &&
+		test_path_is_missing file6.t &&
+		test_path_is_file file6.ta
+	)
+'
+
 test_expect_success 'kill p4d' '
 	kill_p4d
 '
-- 
1.7.10.366.g90ade

[PATCH 3/4] git p4: fix writable file after rename or copy

[Pete Wyckoff]

The way rename works is with a "p4 integrate", optionally
followed by a "p4 edit" if the change is not a 100% rename.
Contents are generated by applying a patch, not doing a file
system rename.  Copy is similar.

In this case, p4 does not fix the permissions back to read-only.
Make sure this happens by calling "p4 sync -f".

Signed-off-by: Pete Wyckoff <pw@padd.com>
---
 git-p4.py                     |   12 ++++++++++++
 t/t9807-git-p4-submit.sh      |    6 ++++--
 t/t9809-git-p4-client-view.sh |    7 ++++---
 3 files changed, 20 insertions(+), 5 deletions(-)

diff --git a/git-p4.py b/git-p4.py
index a2eba5d..888e3e7 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -1021,6 +1021,7 @@ class P4Submit(Command, P4UserMap):
         filesToAdd = set()
         filesToDelete = set()
         editedFiles = set()
+        pureRenameCopy = set()
         filesToChangeExecBit = {}
 
         for line in diff:
@@ -1044,10 +1045,13 @@ class P4Submit(Command, P4UserMap):
             elif modifier == "C":
                 src, dest = diff['src'], diff['dst']
                 p4_integrate(src, dest)
+                pureRenameCopy.add(dest)
                 if diff['src_sha1'] != diff['dst_sha1']:
                     p4_edit(dest)
+                    pureRenameCopy.discard(dest)
                 if isModeExecChanged(diff['src_mode'], diff['dst_mode']):
                     p4_edit(dest)
+                    pureRenameCopy.discard(dest)
                     filesToChangeExecBit[dest] = diff['dst_mode']
                 os.unlink(dest)
                 editedFiles.add(dest)
@@ -1056,6 +1060,8 @@ class P4Submit(Command, P4UserMap):
                 p4_integrate(src, dest)
                 if diff['src_sha1'] != diff['dst_sha1']:
                     p4_edit(dest)
+                else:
+                    pureRenameCopy.add(dest)
                 if isModeExecChanged(diff['src_mode'], diff['dst_mode']):
                     p4_edit(dest)
                     filesToChangeExecBit[dest] = diff['dst_mode']
@@ -1209,6 +1215,12 @@ class P4Submit(Command, P4UserMap):
                         # unmarshalled.
                         changelist = self.lastP4Changelist()
                         self.modifyChangelistUser(changelist, p4User)
+
+                # The rename/copy happened by applying a patch that created a
+                # new file.  This leaves it writable, which confuses p4.
+                for f in pureRenameCopy:
+                    p4_sync(f, "-f")
+
             else:
                 # skip this patch
                 print "Submission cancelled, undoing p4 changes."
diff --git a/t/t9807-git-p4-submit.sh b/t/t9807-git-p4-submit.sh
index a2499ee..f23b4c3 100755
--- a/t/t9807-git-p4-submit.sh
+++ b/t/t9807-git-p4-submit.sh
@@ -158,7 +158,8 @@ test_expect_success 'submit copy' '
 	) &&
 	(
 		cd "$cli" &&
-		test_path_is_file file5.ta
+		test_path_is_file file5.ta &&
+		test ! -w file5.ta
 	)
 '
 
@@ -176,7 +177,8 @@ test_expect_success 'submit rename' '
 	(
 		cd "$cli" &&
 		test_path_is_missing file6.t &&
-		test_path_is_file file6.ta
+		test_path_is_file file6.ta &&
+		test ! -w file6.ta
 	)
 '
 
diff --git a/t/t9809-git-p4-client-view.sh b/t/t9809-git-p4-client-view.sh
index 796b02c..43ed1fe 100755
--- a/t/t9809-git-p4-client-view.sh
+++ b/t/t9809-git-p4-client-view.sh
@@ -349,7 +349,8 @@ test_expect_success 'subdir clone, submit copy' '
 	) &&
 	(
 		cd "$cli" &&
-		test_path_is_file dir1/file11a
+		test_path_is_file dir1/file11a &&
+		test ! -w dir1/file11a
 	)
 '
 
@@ -368,14 +369,14 @@ test_expect_success 'subdir clone, submit rename' '
 	(
 		cd "$cli" &&
 		test_path_is_missing dir1/file13 &&
-		test_path_is_file dir1/file13a
+		test_path_is_file dir1/file13a &&
+		test ! -w dir1/file13a
 	)
 '
 
 test_expect_success 'reinit depot' '
 	(
 		cd "$cli" &&
-		p4 sync -f &&
 		rm files &&
 		p4 delete */* &&
 		p4 submit -d "delete all files" &&
-- 
1.7.10.366.g90ade

[PATCH 4/4] git p4: submit files with wildcards

[Pete Wyckoff]

There are four wildcard characters in p4.  Files with these
characters can be added to p4 repos using the "-f" option.  They
are stored in %xx notation, and when checked out, p4 converts
them back to normal.

When adding files with wildcards in git, the submit path must
be careful to use the encoded names in some places, and it
must use "-f" to add them.  All other p4 commands that operate
on the client directory expect encoded filenames as arguments.

Support for wildcards in the clone/sync path was added in
084f630 (git-p4: decode p4 wildcard characters, 2011-02-19),
but that change did not handle the submit path.

There was a problem with wildcards in the sync path too.  Commit
084f630 (git-p4: decode p4 wildcard characters, 2011-02-19)
handled files with p4 wildcards that were added or modified in
p4.  Do this for deleted files, and also in branch detection
checks, too.

Reported-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Pete Wyckoff <pw@padd.com>
---
 git-p4.py                     |   74 +++++++++++++++++-----------
 t/t9800-git-p4-basic.sh       |  106 +++++++++++++++++++++++++++++++++++++++++
 t/t9809-git-p4-client-view.sh |   33 +++++++++++++
 3 files changed, 185 insertions(+), 28 deletions(-)

diff --git a/git-p4.py b/git-p4.py
index 888e3e7..5afd501 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -131,25 +131,29 @@ def p4_system(cmd):
     subprocess.check_call(real_cmd, shell=expand)
 
 def p4_integrate(src, dest):
-    p4_system(["integrate", "-Dt", src, dest])
+    p4_system(["integrate", "-Dt", wildcard_encode(src), wildcard_encode(dest)])
 
 def p4_sync(f, *options):
-    p4_system(["sync"] + list(options) + [f])
+    p4_system(["sync"] + list(options) + [wildcard_encode(f)])
 
 def p4_add(f):
-    p4_system(["add", f])
+    # forcibly add file names with wildcards
+    if wildcard_present(f):
+        p4_system(["add", "-f", f])
+    else:
+        p4_system(["add", f])
 
 def p4_delete(f):
-    p4_system(["delete", f])
+    p4_system(["delete", wildcard_encode(f)])
 
 def p4_edit(f):
-    p4_system(["edit", f])
+    p4_system(["edit", wildcard_encode(f)])
 
 def p4_revert(f):
-    p4_system(["revert", f])
+    p4_system(["revert", wildcard_encode(f)])
 
-def p4_reopen(type, file):
-    p4_system(["reopen", "-t", type, file])
+def p4_reopen(type, f):
+    p4_system(["reopen", "-t", type, wildcard_encode(f)])
 
 #
 # Canonicalize the p4 type and return a tuple of the
@@ -246,7 +250,7 @@ def setP4ExecBit(file, mode):
 def getP4OpenedType(file):
     # Returns the perforce file type for the given file.
 
-    result = p4_read_pipe(["opened", file])
+    result = p4_read_pipe(["opened", wildcard_encode(file)])
     match = re.match(".*\((.+)\)\r?$", result)
     if match:
         return match.group(1)
@@ -636,6 +640,34 @@ def getClientRoot():
 
     return entry["Root"]
 
+#
+# P4 wildcards are not allowed in filenames.  P4 complains
+# if you simply add them, but you can force it with "-f", in
+# which case it translates them into %xx encoding internally.
+#
+def wildcard_decode(path):
+    # Search for and fix just these four characters.  Do % last so
+    # that fixing it does not inadvertently create new %-escapes.
+    # Cannot have * in a filename in windows; untested as to
+    # what p4 would do in such a case.
+    if not platform.system() == "Windows":
+        path = path.replace("%2A", "*")
+    path = path.replace("%23", "#") \
+               .replace("%40", "@") \
+               .replace("%25", "%")
+    return path
+
+def wildcard_encode(path):
+    # do % first to avoid double-encoding the %s introduced here
+    path = path.replace("%", "%25") \
+               .replace("*", "%2A") \
+               .replace("#", "%23") \
+               .replace("@", "%40")
+    return path
+
+def wildcard_present(path):
+    return path.translate(None, "*#@%") != path
+
 class Command:
     def __init__(self):
         self.usage = "usage: %prog [options]"
@@ -1170,7 +1202,8 @@ class P4Submit(Command, P4UserMap):
                 del(os.environ["P4DIFF"])
             diff = ""
             for editedFile in editedFiles:
-                diff += p4_read_pipe(['diff', '-du', editedFile])
+                diff += p4_read_pipe(['diff', '-du',
+                                      wildcard_encode(editedFile)])
 
             newdiff = ""
             for newFile in filesToAdd:
@@ -1605,23 +1638,6 @@ class P4Sync(Command, P4UserMap):
         if gitConfig("git-p4.syncFromOrigin") == "false":
             self.syncWithOrigin = False
 
-    #
-    # P4 wildcards are not allowed in filenames.  P4 complains
-    # if you simply add them, but you can force it with "-f", in
-    # which case it translates them into %xx encoding internally.
-    # Search for and fix just these four characters.  Do % last so
-    # that fixing it does not inadvertently create new %-escapes.
-    #
-    def wildcard_decode(self, path):
-        # Cannot have * in a filename in windows; untested as to
-        # what p4 would do in such a case.
-        if not self.isWindows:
-            path = path.replace("%2A", "*")
-        path = path.replace("%23", "#") \
-                   .replace("%40", "@") \
-                   .replace("%25", "%")
-        return path
-
     # Force a checkpoint in fast-import and wait for it to finish
     def checkpoint(self):
         self.gitStream.write("checkpoint\n\n")
@@ -1689,6 +1705,7 @@ class P4Sync(Command, P4UserMap):
             fnum = fnum + 1
 
             relPath = self.stripRepoPath(path, self.depotPaths)
+            relPath = wildcard_decode(relPath)
 
             for branch in self.knownBranches.keys():
 
@@ -1706,7 +1723,7 @@ class P4Sync(Command, P4UserMap):
 
     def streamOneP4File(self, file, contents):
         relPath = self.stripRepoPath(file['depotFile'], self.branchPrefixes)
-        relPath = self.wildcard_decode(relPath)
+        relPath = wildcard_decode(relPath)
         if verbose:
             sys.stderr.write("%s\n" % relPath)
 
@@ -1775,6 +1792,7 @@ class P4Sync(Command, P4UserMap):
 
     def streamOneP4Deletion(self, file):
         relPath = self.stripRepoPath(file['path'], self.branchPrefixes)
+        relPath = wildcard_decode(relPath)
         if verbose:
             sys.stderr.write("delete %s\n" % relPath)
         self.gitStream.write("D %s\n" % relPath)
diff --git a/t/t9800-git-p4-basic.sh b/t/t9800-git-p4-basic.sh
index 13be144..0223fcf 100755
--- a/t/t9800-git-p4-basic.sh
+++ b/t/t9800-git-p4-basic.sh
@@ -163,6 +163,112 @@ test_expect_success 'wildcard files git p4 clone' '
 	)
 '
 
+test_expect_success 'wildcard files submit back to p4, add' '
+	test_when_finished cleanup_git &&
+	git p4 clone --dest="$git" //depot &&
+	(
+		cd "$git" &&
+		echo git-wild-hash >git-wild#hash &&
+		echo git-wild-star >git-wild\*star &&
+		echo git-wild-at >git-wild@at &&
+		echo git-wild-percent >git-wild%percent &&
+		git add git-wild* &&
+		git commit -m "add some wildcard filenames" &&
+		git config git-p4.skipSubmitEdit true &&
+		git p4 submit
+	) &&
+	(
+		cd "$cli" &&
+		test_path_is_file git-wild#hash &&
+		test_path_is_file git-wild\*star &&
+		test_path_is_file git-wild@at &&
+		test_path_is_file git-wild%percent
+	)
+'
+
+test_expect_success 'wildcard files submit back to p4, modify' '
+	test_when_finished cleanup_git &&
+	git p4 clone --dest="$git" //depot &&
+	(
+		cd "$git" &&
+		echo new-line >>git-wild#hash &&
+		echo new-line >>git-wild\*star &&
+		echo new-line >>git-wild@at &&
+		echo new-line >>git-wild%percent &&
+		git add git-wild* &&
+		git commit -m "modify the wildcard files" &&
+		git config git-p4.skipSubmitEdit true &&
+		git p4 submit
+	) &&
+	(
+		cd "$cli" &&
+		test_line_count = 2 git-wild#hash &&
+		test_line_count = 2 git-wild\*star &&
+		test_line_count = 2 git-wild@at &&
+		test_line_count = 2 git-wild%percent
+	)
+'
+
+test_expect_success 'wildcard files submit back to p4, copy' '
+	test_when_finished cleanup_git &&
+	git p4 clone --dest="$git" //depot &&
+	(
+		cd "$git" &&
+		cp file2 git-wild-cp#hash &&
+		git add git-wild-cp#hash &&
+		cp git-wild\*star file-wild-3 &&
+		git add file-wild-3 &&
+		git commit -m "wildcard copies" &&
+		git config git-p4.detectCopies true &&
+		git config git-p4.detectCopiesHarder true &&
+		git config git-p4.skipSubmitEdit true &&
+		git p4 submit
+	) &&
+	(
+		cd "$cli" &&
+		test_path_is_file git-wild-cp#hash &&
+		test_path_is_file file-wild-3
+	)
+'
+
+test_expect_success 'wildcard files submit back to p4, rename' '
+	test_when_finished cleanup_git &&
+	git p4 clone --dest="$git" //depot &&
+	(
+		cd "$git" &&
+		git mv git-wild@at file-wild-4 &&
+		git mv file-wild-3 git-wild-cp%percent &&
+		git commit -m "wildcard renames" &&
+		git config git-p4.detectRenames true &&
+		git config git-p4.skipSubmitEdit true &&
+		git p4 submit
+	) &&
+	(
+		cd "$cli" &&
+		test_path_is_missing git-wild@at &&
+		test_path_is_file git-wild-cp%percent
+	)
+'
+
+test_expect_success 'wildcard files submit back to p4, delete' '
+	test_when_finished cleanup_git &&
+	git p4 clone --dest="$git" //depot &&
+	(
+		cd "$git" &&
+		git rm git-wild* &&
+		git commit -m "delete the wildcard files" &&
+		git config git-p4.skipSubmitEdit true &&
+		git p4 submit
+	) &&
+	(
+		cd "$cli" &&
+		test_path_is_missing git-wild#hash &&
+		test_path_is_missing git-wild\*star &&
+		test_path_is_missing git-wild@at &&
+		test_path_is_missing git-wild%percent
+	)
+'
+
 test_expect_success 'clone bare' '
 	git p4 clone --dest="$git" --bare //depot &&
 	test_when_finished cleanup_git &&
diff --git a/t/t9809-git-p4-client-view.sh b/t/t9809-git-p4-client-view.sh
index 43ed1fe..7d993ef 100755
--- a/t/t9809-git-p4-client-view.sh
+++ b/t/t9809-git-p4-client-view.sh
@@ -374,6 +374,39 @@ test_expect_success 'subdir clone, submit rename' '
 	)
 '
 
+# see t9800 for the non-client-spec case, and the rest of the wildcard tests
+test_expect_success 'wildcard files submit back to p4, client-spec case' '
+	client_view "//depot/... //client/..." &&
+	test_when_finished cleanup_git &&
+	git p4 clone --use-client-spec --dest="$git" //depot/dir1 &&
+	(
+		cd "$git" &&
+		echo git-wild-hash >dir1/git-wild#hash &&
+		echo git-wild-star >dir1/git-wild\*star &&
+		echo git-wild-at >dir1/git-wild@at &&
+		echo git-wild-percent >dir1/git-wild%percent &&
+		git add dir1/git-wild* &&
+		git commit -m "add some wildcard filenames" &&
+		git config git-p4.skipSubmitEditCheck true &&
+		git p4 submit
+	) &&
+	(
+		cd "$cli" &&
+		test_path_is_file dir1/git-wild#hash &&
+		test_path_is_file dir1/git-wild\*star &&
+		test_path_is_file dir1/git-wild@at &&
+		test_path_is_file dir1/git-wild%percent
+	) &&
+	(
+		# delete these carefully, cannot just do "p4 delete"
+		# on files with wildcards; but git-p4 knows how
+		cd "$git" &&
+		git rm dir1/git-wild* &&
+		git commit -m "clean up the wildcards" &&
+		git p4 submit
+	)
+'
+
 test_expect_success 'reinit depot' '
 	(
 		cd "$cli" &&
-- 
1.7.10.366.g90ade

Re: [PATCH 4/4] git p4: submit files with wildcards

[Luke Diamand]

On 30/04/12 01:57, Pete Wyckoff wrote:
> There are four wildcard characters in p4.  Files with these
> characters can be added to p4 repos using the "-f" option.  They
> are stored in %xx notation, and when checked out, p4 converts
> them back to normal.
>
> When adding files with wildcards in git, the submit path must
> be careful to use the encoded names in some places, and it
> must use "-f" to add them.  All other p4 commands that operate
> on the client directory expect encoded filenames as arguments.
>
> Support for wildcards in the clone/sync path was added in
> 084f630 (git-p4: decode p4 wildcard characters, 2011-02-19),
> but that change did not handle the submit path.
>
> There was a problem with wildcards in the sync path too.  Commit
> 084f630 (git-p4: decode p4 wildcard characters, 2011-02-19)
> handled files with p4 wildcards that were added or modified in
> p4.  Do this for deleted files, and also in branch detection
> checks, too.

Ack, thanks for fixing this. The other changes look fine as well.

>
> Reported-by: Luke Diamand<luke@diamand.org>
> Signed-off-by: Pete Wyckoff<pw@padd.com>
> ---
>   git-p4.py                     |   74 +++++++++++++++++-----------
>   t/t9800-git-p4-basic.sh       |  106 +++++++++++++++++++++++++++++++++++++++++
>   t/t9809-git-p4-client-view.sh |   33 +++++++++++++
>   3 files changed, 185 insertions(+), 28 deletions(-)
>
> diff --git a/git-p4.py b/git-p4.py
> index 888e3e7..5afd501 100755
> --- a/git-p4.py
> +++ b/git-p4.py
> @@ -131,25 +131,29 @@ def p4_system(cmd):
>       subprocess.check_call(real_cmd, shell=expand)
>
>   def p4_integrate(src, dest):
> -    p4_system(["integrate", "-Dt", src, dest])
> +    p4_system(["integrate", "-Dt", wildcard_encode(src), wildcard_encode(dest)])
>
>   def p4_sync(f, *options):
> -    p4_system(["sync"] + list(options) + [f])
> +    p4_system(["sync"] + list(options) + [wildcard_encode(f)])
>
>   def p4_add(f):
> -    p4_system(["add", f])
> +    # forcibly add file names with wildcards
> +    if wildcard_present(f):
> +        p4_system(["add", "-f", f])
> +    else:
> +        p4_system(["add", f])
>
>   def p4_delete(f):
> -    p4_system(["delete", f])
> +    p4_system(["delete", wildcard_encode(f)])
>
>   def p4_edit(f):
> -    p4_system(["edit", f])
> +    p4_system(["edit", wildcard_encode(f)])
>
>   def p4_revert(f):
> -    p4_system(["revert", f])
> +    p4_system(["revert", wildcard_encode(f)])
>
> -def p4_reopen(type, file):
> -    p4_system(["reopen", "-t", type, file])
> +def p4_reopen(type, f):
> +    p4_system(["reopen", "-t", type, wildcard_encode(f)])
>
>   #
>   # Canonicalize the p4 type and return a tuple of the
> @@ -246,7 +250,7 @@ def setP4ExecBit(file, mode):
>   def getP4OpenedType(file):
>       # Returns the perforce file type for the given file.
>
> -    result = p4_read_pipe(["opened", file])
> +    result = p4_read_pipe(["opened", wildcard_encode(file)])
>       match = re.match(".*\((.+)\)\r?$", result)
>       if match:
>           return match.group(1)
> @@ -636,6 +640,34 @@ def getClientRoot():
>
>       return entry["Root"]
>
> +#
> +# P4 wildcards are not allowed in filenames.  P4 complains
> +# if you simply add them, but you can force it with "-f", in
> +# which case it translates them into %xx encoding internally.
> +#
> +def wildcard_decode(path):
> +    # Search for and fix just these four characters.  Do % last so
> +    # that fixing it does not inadvertently create new %-escapes.
> +    # Cannot have * in a filename in windows; untested as to
> +    # what p4 would do in such a case.
> +    if not platform.system() == "Windows":
> +        path = path.replace("%2A", "*")
> +    path = path.replace("%23", "#") \
> +               .replace("%40", "@") \
> +               .replace("%25", "%")
> +    return path
> +
> +def wildcard_encode(path):
> +    # do % first to avoid double-encoding the %s introduced here
> +    path = path.replace("%", "%25") \
> +               .replace("*", "%2A") \
> +               .replace("#", "%23") \
> +               .replace("@", "%40")
> +    return path
> +
> +def wildcard_present(path):
> +    return path.translate(None, "*#@%") != path
> +
>   class Command:
>       def __init__(self):
>           self.usage = "usage: %prog [options]"
> @@ -1170,7 +1202,8 @@ class P4Submit(Command, P4UserMap):
>                   del(os.environ["P4DIFF"])
>               diff = ""
>               for editedFile in editedFiles:
> -                diff += p4_read_pipe(['diff', '-du', editedFile])
> +                diff += p4_read_pipe(['diff', '-du',
> +                                      wildcard_encode(editedFile)])
>
>               newdiff = ""
>               for newFile in filesToAdd:
> @@ -1605,23 +1638,6 @@ class P4Sync(Command, P4UserMap):
>           if gitConfig("git-p4.syncFromOrigin") == "false":
>               self.syncWithOrigin = False
>
> -    #
> -    # P4 wildcards are not allowed in filenames.  P4 complains
> -    # if you simply add them, but you can force it with "-f", in
> -    # which case it translates them into %xx encoding internally.
> -    # Search for and fix just these four characters.  Do % last so
> -    # that fixing it does not inadvertently create new %-escapes.
> -    #
> -    def wildcard_decode(self, path):
> -        # Cannot have * in a filename in windows; untested as to
> -        # what p4 would do in such a case.
> -        if not self.isWindows:
> -            path = path.replace("%2A", "*")
> -        path = path.replace("%23", "#") \
> -                   .replace("%40", "@") \
> -                   .replace("%25", "%")
> -        return path
> -
>       # Force a checkpoint in fast-import and wait for it to finish
>       def checkpoint(self):
>           self.gitStream.write("checkpoint\n\n")
> @@ -1689,6 +1705,7 @@ class P4Sync(Command, P4UserMap):
>               fnum = fnum + 1
>
>               relPath = self.stripRepoPath(path, self.depotPaths)
> +            relPath = wildcard_decode(relPath)
>
>               for branch in self.knownBranches.keys():
>
> @@ -1706,7 +1723,7 @@ class P4Sync(Command, P4UserMap):
>
>       def streamOneP4File(self, file, contents):
>           relPath = self.stripRepoPath(file['depotFile'], self.branchPrefixes)
> -        relPath = self.wildcard_decode(relPath)
> +        relPath = wildcard_decode(relPath)
>           if verbose:
>               sys.stderr.write("%s\n" % relPath)
>
> @@ -1775,6 +1792,7 @@ class P4Sync(Command, P4UserMap):
>
>       def streamOneP4Deletion(self, file):
>           relPath = self.stripRepoPath(file['path'], self.branchPrefixes)
> +        relPath = wildcard_decode(relPath)
>           if verbose:
>               sys.stderr.write("delete %s\n" % relPath)
>           self.gitStream.write("D %s\n" % relPath)
> diff --git a/t/t9800-git-p4-basic.sh b/t/t9800-git-p4-basic.sh
> index 13be144..0223fcf 100755
> --- a/t/t9800-git-p4-basic.sh
> +++ b/t/t9800-git-p4-basic.sh
> @@ -163,6 +163,112 @@ test_expect_success 'wildcard files git p4 clone' '
>   	)
>   '
>
> +test_expect_success 'wildcard files submit back to p4, add' '
> +	test_when_finished cleanup_git&&
> +	git p4 clone --dest="$git" //depot&&
> +	(
> +		cd "$git"&&
> +		echo git-wild-hash>git-wild#hash&&
> +		echo git-wild-star>git-wild\*star&&
> +		echo git-wild-at>git-wild@at&&
> +		echo git-wild-percent>git-wild%percent&&
> +		git add git-wild*&&
> +		git commit -m "add some wildcard filenames"&&
> +		git config git-p4.skipSubmitEdit true&&
> +		git p4 submit
> +	)&&
> +	(
> +		cd "$cli"&&
> +		test_path_is_file git-wild#hash&&
> +		test_path_is_file git-wild\*star&&
> +		test_path_is_file git-wild@at&&
> +		test_path_is_file git-wild%percent
> +	)
> +'
> +
> +test_expect_success 'wildcard files submit back to p4, modify' '
> +	test_when_finished cleanup_git&&
> +	git p4 clone --dest="$git" //depot&&
> +	(
> +		cd "$git"&&
> +		echo new-line>>git-wild#hash&&
> +		echo new-line>>git-wild\*star&&
> +		echo new-line>>git-wild@at&&
> +		echo new-line>>git-wild%percent&&
> +		git add git-wild*&&
> +		git commit -m "modify the wildcard files"&&
> +		git config git-p4.skipSubmitEdit true&&
> +		git p4 submit
> +	)&&
> +	(
> +		cd "$cli"&&
> +		test_line_count = 2 git-wild#hash&&
> +		test_line_count = 2 git-wild\*star&&
> +		test_line_count = 2 git-wild@at&&
> +		test_line_count = 2 git-wild%percent
> +	)
> +'
> +
> +test_expect_success 'wildcard files submit back to p4, copy' '
> +	test_when_finished cleanup_git&&
> +	git p4 clone --dest="$git" //depot&&
> +	(
> +		cd "$git"&&
> +		cp file2 git-wild-cp#hash&&
> +		git add git-wild-cp#hash&&
> +		cp git-wild\*star file-wild-3&&
> +		git add file-wild-3&&
> +		git commit -m "wildcard copies"&&
> +		git config git-p4.detectCopies true&&
> +		git config git-p4.detectCopiesHarder true&&
> +		git config git-p4.skipSubmitEdit true&&
> +		git p4 submit
> +	)&&
> +	(
> +		cd "$cli"&&
> +		test_path_is_file git-wild-cp#hash&&
> +		test_path_is_file file-wild-3
> +	)
> +'
> +
> +test_expect_success 'wildcard files submit back to p4, rename' '
> +	test_when_finished cleanup_git&&
> +	git p4 clone --dest="$git" //depot&&
> +	(
> +		cd "$git"&&
> +		git mv git-wild@at file-wild-4&&
> +		git mv file-wild-3 git-wild-cp%percent&&
> +		git commit -m "wildcard renames"&&
> +		git config git-p4.detectRenames true&&
> +		git config git-p4.skipSubmitEdit true&&
> +		git p4 submit
> +	)&&
> +	(
> +		cd "$cli"&&
> +		test_path_is_missing git-wild@at&&
> +		test_path_is_file git-wild-cp%percent
> +	)
> +'
> +
> +test_expect_success 'wildcard files submit back to p4, delete' '
> +	test_when_finished cleanup_git&&
> +	git p4 clone --dest="$git" //depot&&
> +	(
> +		cd "$git"&&
> +		git rm git-wild*&&
> +		git commit -m "delete the wildcard files"&&
> +		git config git-p4.skipSubmitEdit true&&
> +		git p4 submit
> +	)&&
> +	(
> +		cd "$cli"&&
> +		test_path_is_missing git-wild#hash&&
> +		test_path_is_missing git-wild\*star&&
> +		test_path_is_missing git-wild@at&&
> +		test_path_is_missing git-wild%percent
> +	)
> +'
> +
>   test_expect_success 'clone bare' '
>   	git p4 clone --dest="$git" --bare //depot&&
>   	test_when_finished cleanup_git&&
> diff --git a/t/t9809-git-p4-client-view.sh b/t/t9809-git-p4-client-view.sh
> index 43ed1fe..7d993ef 100755
> --- a/t/t9809-git-p4-client-view.sh
> +++ b/t/t9809-git-p4-client-view.sh
> @@ -374,6 +374,39 @@ test_expect_success 'subdir clone, submit rename' '
>   	)
>   '
>
> +# see t9800 for the non-client-spec case, and the rest of the wildcard tests
> +test_expect_success 'wildcard files submit back to p4, client-spec case' '
> +	client_view "//depot/... //client/..."&&
> +	test_when_finished cleanup_git&&
> +	git p4 clone --use-client-spec --dest="$git" //depot/dir1&&
> +	(
> +		cd "$git"&&
> +		echo git-wild-hash>dir1/git-wild#hash&&
> +		echo git-wild-star>dir1/git-wild\*star&&
> +		echo git-wild-at>dir1/git-wild@at&&
> +		echo git-wild-percent>dir1/git-wild%percent&&
> +		git add dir1/git-wild*&&
> +		git commit -m "add some wildcard filenames"&&
> +		git config git-p4.skipSubmitEditCheck true&&
> +		git p4 submit
> +	)&&
> +	(
> +		cd "$cli"&&
> +		test_path_is_file dir1/git-wild#hash&&
> +		test_path_is_file dir1/git-wild\*star&&
> +		test_path_is_file dir1/git-wild@at&&
> +		test_path_is_file dir1/git-wild%percent
> +	)&&
> +	(
> +		# delete these carefully, cannot just do "p4 delete"
> +		# on files with wildcards; but git-p4 knows how
> +		cd "$git"&&
> +		git rm dir1/git-wild*&&
> +		git commit -m "clean up the wildcards"&&
> +		git p4 submit
> +	)
> +'
> +
>   test_expect_success 'reinit depot' '
>   	(
>   		cd "$cli"&&