[PATCH v3 0/4] git-p4: Improve branch support

[PATCH v3 0/4] git-p4: Improve branch support

[Vitor Antunes]

Add missing test case from last version of this set of patches.

Vitor Antunes (4):
  git-p4: Correct branch base depot path detection
  git-p4: Allow filtering Perforce branches by user
  git-p4: Allow branch definition with git config
  git-p4: Add simple test case for branch import

 contrib/fast-import/git-p4     |   40 ++++++++++++++++++++++--
 contrib/fast-import/git-p4.txt |   13 ++++++++
 t/t9800-git-p4.sh              |   64 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 113 insertions(+), 4 deletions(-)

-- 
1.7.5.4

[PATCH v3 1/4] git-p4: Correct branch base depot path detection

[Vitor Antunes]

When branch detection is enabled each branch is named in git after their
relative depot path in Perforce. To do this the depot paths are compared against
each other to find their common base path. The current algorithm makes this
comparison on a character by character basis.
Assuming we have the following branches:

//depot/branches/featureA
//depot/branches/featureB

Then the base depot path would be //depot/branches/feature, which is an invalid
depot path.
The current patch fixes this by splitting the path into a list and comparing the
list entries, making it choose correctly //depot/branches as the base path.

Signed-off-by: Vitor Antunes <vitor.hda@gmail.com>
---
 contrib/fast-import/git-p4 |    8 +++++---
 1 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/contrib/fast-import/git-p4 b/contrib/fast-import/git-p4
index 29a5390..95246e9 100755
--- a/contrib/fast-import/git-p4
+++ b/contrib/fast-import/git-p4
@@ -1829,12 +1829,14 @@ class P4Sync(Command, P4UserMap):
                     else:
                         paths = []
                         for (prev, cur) in zip(self.previousDepotPaths, depotPaths):
-                            for i in range(0, min(len(cur), len(prev))):
-                                if cur[i] <> prev[i]:
+                            prev_list = prev.split("/")
+                            cur_list = cur.split("/")
+                            for i in range(0, min(len(cur_list), len(prev_list))):
+                                if cur_list[i] <> prev_list[i]:
                                     i = i - 1
                                     break
 
-                            paths.append (cur[:i + 1])
+                            paths.append ("/".join(cur_list[:i + 1]))
 
                         self.previousDepotPaths = paths
 
-- 
1.7.5.4

[PATCH v3 2/4] git-p4: Allow filtering Perforce branches by user

[Vitor Antunes]

All branches in the Perforce server are downloaded to allow branch detection. If
you have a centralized server on a remote location and there is a big number of
branches this operation can take some time.
This patch adds the configuration option git-p4.branchUser to allow filtering
the branch list by user. Although this limits the branch maintenance in Perforce
to be done by a single user, it might be an advantage when the number of
branches being used in a specific depot is very small when compared with the
branches available in the server.

Signed-off-by: Vitor Antunes <vitor.hda@gmail.com>
---
 contrib/fast-import/git-p4     |    8 +++++++-
 contrib/fast-import/git-p4.txt |    6 ++++++
 2 files changed, 13 insertions(+), 1 deletions(-)

diff --git a/contrib/fast-import/git-p4 b/contrib/fast-import/git-p4
index 95246e9..8b88f97 100755
--- a/contrib/fast-import/git-p4
+++ b/contrib/fast-import/git-p4
@@ -1455,7 +1455,13 @@ class P4Sync(Command, P4UserMap):
     def getBranchMapping(self):
         lostAndFoundBranches = set()
 
-        for info in p4CmdList("branches"):
+        user = gitConfig("git-p4.branchUser")
+        if len(user) > 0:
+            command = "branches -u %s" % user
+        else:
+            command = "branches"
+
+        for info in p4CmdList(command):
             details = p4Cmd("branch -o %s" % info["branch"])
             viewIdx = 0
             while details.has_key("View%s" % viewIdx):
diff --git a/contrib/fast-import/git-p4.txt b/contrib/fast-import/git-p4.txt
index 2ffbccc..97b66b9 100644
--- a/contrib/fast-import/git-p4.txt
+++ b/contrib/fast-import/git-p4.txt
@@ -257,6 +257,12 @@ Perforce server. Will enable --find-copies-harder git argument.
 
   git config [--global] git-p4.detectCopies true
 
+git-p4.branchUser
+
+Only use branch specifications defined by the selected username.
+
+  git config [--global] git-p4.branchUser username
+
 Implementation Details...
 =========================
 
-- 
1.7.5.4

[PATCH v3 3/4] git-p4: Allow branch definition with git config

[Vitor Antunes]

Perforce does not strictly require the usage of branch specifications to create
branches. In these cases the branch detection code of git-p4 will not be able to
import them.
This patch adds support for git-p4.branchList configuration option, allowing
branches to be defined in git config.

Signed-off-by: Vitor Antunes <vitor.hda@gmail.com>
---
 contrib/fast-import/git-p4     |   24 ++++++++++++++++++++++++
 contrib/fast-import/git-p4.txt |    7 +++++++
 2 files changed, 31 insertions(+), 0 deletions(-)

diff --git a/contrib/fast-import/git-p4 b/contrib/fast-import/git-p4
index 8b88f97..f622a38 100755
--- a/contrib/fast-import/git-p4
+++ b/contrib/fast-import/git-p4
@@ -342,6 +342,11 @@ def gitConfig(key, args = None): # set args to "--bool", for instance
         _gitConfig[key] = read_pipe(cmd, ignore_error=True).strip()
     return _gitConfig[key]
 
+def gitConfigList(key):
+    if not _gitConfig.has_key(key):
+        _gitConfig[key] = read_pipe("git config --get-all %s" % key, ignore_error=True).strip().split(os.linesep)
+    return _gitConfig[key]
+
 def p4BranchesInGit(branchesAreInRemotes = True):
     branches = {}
 
@@ -1490,6 +1495,25 @@ class P4Sync(Command, P4UserMap):
                     if source not in self.knownBranches:
                         lostAndFoundBranches.add(source)
 
+        # Perforce does not strictly require branches to be defined, so we also
+        # check git config for a branch list.
+        #
+        # Example of branch definition in git config file:
+        # [git-p4]
+        #   branchList=main:branchA
+        #   branchList=main:branchB
+        #   branchList=branchA:branchC
+        configBranches = gitConfigList("git-p4.branchList")
+        for branch in configBranches:
+            if branch:
+                (source, destination) = branch.split(":")
+                self.knownBranches[destination] = source
+
+                lostAndFoundBranches.discard(destination)
+
+                if source not in self.knownBranches:
+                    lostAndFoundBranches.add(source)
+
 
         for branch in lostAndFoundBranches:
             self.knownBranches[branch] = branch
diff --git a/contrib/fast-import/git-p4.txt b/contrib/fast-import/git-p4.txt
index 97b66b9..52003ae 100644
--- a/contrib/fast-import/git-p4.txt
+++ b/contrib/fast-import/git-p4.txt
@@ -263,6 +263,13 @@ Only use branch specifications defined by the selected username.
 
   git config [--global] git-p4.branchUser username
 
+git-p4.branchList
+
+List of branches to be imported when branch detection is enabled.
+
+  git config [--global] git-p4.branchList main:branchA
+  git config [--global] --add git-p4.branchList main:branchB
+
 Implementation Details...
 =========================
 
-- 
1.7.5.4

[PATCH v3 4/4] git-p4: Add simple test case for branch import

[Vitor Antunes]

Create a basic branch structure in P4 and clone it with git-p4.
Also, make an update on P4 side and check if git-p4 imports it correctly.
The branch structure is created in such a way that git-p4 will fail to import
updates if patch "git-p4: Correct branch base depot path detection" is not
applied.

Signed-off-by: Vitor Antunes <vitor.hda@gmail.com>
---
 t/t9800-git-p4.sh |   64 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 64 insertions(+), 0 deletions(-)

diff --git a/t/t9800-git-p4.sh b/t/t9800-git-p4.sh
index a4f3d66..777ead8 100755
--- a/t/t9800-git-p4.sh
+++ b/t/t9800-git-p4.sh
@@ -390,6 +390,70 @@ test_expect_success 'detect copies' '
 	rm -rf "$git" && mkdir "$git"
 '
 
+# Create a simple branch structure in P4 depot to check if it is correctly
+# cloned.
+test_expect_success 'add simple p4 branches' '
+	cd "$cli" &&
+	mkdir branch1 &&
+	cd branch1 &&
+	echo file1 >file1 &&
+	echo file2 >file2 &&
+	p4 add file* &&
+	p4 submit -d "branch1" &&
+	p4 integrate //depot/branch1/... //depot/branch2/... &&
+	p4 submit -d "branch2" &&
+	echo file3 >file3 &&
+	p4 add file3 &&
+	p4 submit -d "add file3 in branch1" &&
+	p4 open file2 &&
+	echo update >>file2 &&
+	p4 submit -d "update file2 in branch1" &&
+	p4 integrate //depot/branch1/... //depot/branch3/... &&
+	p4 submit -d "branch3" &&
+	cd "$TRASH_DIRECTORY"
+'
+
+# Configure branches through git-config and clone them.
+# All files are tested to make sure branches were cloned correctly.
+# Finally, make an update to branch1 on P4 side to check if it is imported
+# correctly by git-p4.
+test_expect_success 'git-p4 clone simple branches' '
+	git init "$git" &&
+	cd "$git" &&
+	git config git-p4.branchList branch1:branch2 &&
+	git config --add git-p4.branchList branch1:branch3 &&
+	cd "$TRASH_DIRECTORY" &&
+	"$GITP4" clone --dest="$git" --detect-branches //depot@all &&
+	cd "$git" &&
+	git log --all --graph --decorate --stat &&
+	git reset --hard p4/depot/branch1 &&
+	test -f file1 &&
+	test -f file2 &&
+	test -f file3 &&
+	grep -q update file2 &&
+	git reset --hard p4/depot/branch2 &&
+	test -f file1 &&
+	test -f file2 &&
+	test \! -z file3 &&
+	! grep -q update file2 &&
+	git reset --hard p4/depot/branch3 &&
+	test -f file1 &&
+	test -f file2 &&
+	test -f file3 &&
+	grep -q update file2 &&
+	cd "$cli" &&
+	cd branch1 &&
+	p4 edit file2 &&
+	echo file2_ >> file2 &&
+	p4 submit -d "update file2 in branch3" &&
+	cd "$git" &&
+	git reset --hard p4/depot/branch1 &&
+	"$GITP4" rebase &&
+	grep -q file2_ file2 &&
+	cd "$TRASH_DIRECTORY" &&
+	rm -rf "$git" && mkdir "$git"
+'
+
 test_expect_success 'shutdown' '
 	pid=`pgrep -f p4d` &&
 	test -n "$pid" &&
-- 
1.7.5.4

Re: [PATCH v3 4/4] git-p4: Add simple test case for branch import

[Pete Wyckoff]

vitor.hda@gmail.com wrote on Fri, 19 Aug 2011 00:44 +0100:
> Create a basic branch structure in P4 and clone it with git-p4.
> Also, make an update on P4 side and check if git-p4 imports it correctly.
> The branch structure is created in such a way that git-p4 will fail to import
> updates if patch "git-p4: Correct branch base depot path detection" is not
> applied.

Here are some minor style edits to this test.

		-- Pete

-----------------8<-----------------------

>From 958effaad24d129ad472b48f3a691c7e7de2e918 Mon Sep 17 00:00:00 2001
From: Pete Wyckoff <pw@padd.com>
Date: Sat, 20 Aug 2011 14:24:31 -0400
Subject: [PATCH 2/4] git-p4: simple branch tests edits

More review comments.

Signed-off-by: Pete Wyckoff <pw@padd.com>
---
 t/t9800-git-p4.sh |   17 +++++++----------
 1 files changed, 7 insertions(+), 10 deletions(-)

diff --git a/t/t9800-git-p4.sh b/t/t9800-git-p4.sh
index bbf1485..5bcb8b4 100755
--- a/t/t9800-git-p4.sh
+++ b/t/t9800-git-p4.sh
@@ -412,7 +412,7 @@ test_expect_success 'add simple p4 branches' '
 	cd branch1 &&
 	echo file1 >file1 &&
 	echo file2 >file2 &&
-	p4 add file* &&
+	p4 add file1 file2 &&
 	p4 submit -d "branch1" &&
 	p4 integrate //depot/branch1/... //depot/branch2/... &&
 	p4 submit -d "branch2" &&
@@ -432,13 +432,12 @@ test_expect_success 'add simple p4 branches' '
 # Finally, make an update to branch1 on P4 side to check if it is imported
 # correctly by git-p4.
 test_expect_success 'git-p4 clone simple branches' '
-	git init "$git" &&
+	test_when_finished cleanup_git &&
+	test_create_repo "$git" &&
 	cd "$git" &&
 	git config git-p4.branchList branch1:branch2 &&
 	git config --add git-p4.branchList branch1:branch3 &&
-	cd "$TRASH_DIRECTORY" &&
-	"$GITP4" clone --dest="$git" --detect-branches //depot@all &&
-	cd "$git" &&
+	"$GITP4" clone --dest=. --detect-branches //depot@all &&
 	git log --all --graph --decorate --stat &&
 	git reset --hard p4/depot/branch1 &&
 	test -f file1 &&
@@ -448,7 +447,7 @@ test_expect_success 'git-p4 clone simple branches' '
 	git reset --hard p4/depot/branch2 &&
 	test -f file1 &&
 	test -f file2 &&
-	test \! -z file3 &&
+	test ! -f file3 &&
 	! grep -q update file2 &&
 	git reset --hard p4/depot/branch3 &&
 	test -f file1 &&
@@ -458,14 +457,12 @@ test_expect_success 'git-p4 clone simple branches' '
 	cd "$cli" &&
 	cd branch1 &&
 	p4 edit file2 &&
-	echo file2_ >> file2 &&
+	echo file2_ >>file2 &&
 	p4 submit -d "update file2 in branch3" &&
 	cd "$git" &&
 	git reset --hard p4/depot/branch1 &&
 	"$GITP4" rebase &&
-	grep -q file2_ file2 &&
-	cd "$TRASH_DIRECTORY" &&
-	rm -rf "$git" && mkdir "$git"
+	grep -q file2_ file2
 '
 
 # Create a complex branch structure in P4 depot to check if they are correctly
-- 
1.7.5.4

Re: [PATCH v3 0/4] git-p4: Improve branch support

[Pete Wyckoff]

vitor.hda@gmail.com wrote on Fri, 19 Aug 2011 00:44 +0100:
> Add missing test case from last version of this set of patches.
> 
> Vitor Antunes (4):
>   git-p4: Correct branch base depot path detection
>   git-p4: Allow filtering Perforce branches by user
>   git-p4: Allow branch definition with git config
>   git-p4: Add simple test case for branch import
> 
>  contrib/fast-import/git-p4     |   40 ++++++++++++++++++++++--
>  contrib/fast-import/git-p4.txt |   13 ++++++++
>  t/t9800-git-p4.sh              |   64 ++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 113 insertions(+), 4 deletions(-)

Patches 1 and 2 are great.  We should have gotten those in way
back when you first submitted them.  I happily ack those.

I'm still a bit hung up on #3, mainly because I don't get branch
support.  Let me play around with your test.  Having this
playbook of how it is supposed to work will help to educate me.

		-- Pete

Re: [PATCH v3 0/4] git-p4: Improve branch support

[Vitor Antunes]

On Fri, Aug 19, 2011 at 12:53 PM, Pete Wyckoff <pw@padd.com> wrote:
> Patches 1 and 2 are great.  We should have gotten those in way
> back when you first submitted them.  I happily ack those.
>
> I'm still a bit hung up on #3, mainly because I don't get branch
> support.  Let me play around with your test.  Having this
> playbook of how it is supposed to work will help to educate me.

(I am probably going in too much detail here. Please skip any
redundant information.)

In general, you can see branches in P4 as you do in SVN. They are
simple copies from one directory to another. For example:

    p4 integrate //depot/big/directory/path/myproj/version_A \
                 //depot/big/directory/path/myproj/version_B

Now the history of "version_B" is tied to "version_A". If you want to
integrate (merge, in this case) new updates from one of the
directories to the other you can just rerun the above command
(version_A and version_B order may change depending on the direction
of the merge).

P4 also allows you to define "branch specs" that you can use as a
short hand instead of having to type in the full directory paths. So,
assuming that you have a branch named "myproj_B" with the following
view defined:

    //depot/big/directory/path/myproj/version_A \
        //depot/big/directory/path/myproj/version_B

When you want to integrate it again you can simply type:

    p4 integrate -b myproj_B

or, if you want to integrate from version_B into version_A instead:

    p4 integrate -b myproj_B -r

git-p4 is using these branch specs to identify branches. Without this
extra information it is nearly impossible for it to identify branches
because an integrate can be done of a single file.

But since P4 does not strictly require branch specs and because most
of the time integrations are done through P4V, then most of the time
no branch specs are created. Now, if this happens git-p4 will not be
able to detect branches! And creating branches in P4 just for git-p4
does not make much sense, right? This is the main reason behind the
third patch... :)

Sorry for the long email.

-- 
Vitor Antunes

Re: [PATCH v3 0/4] git-p4: Improve branch support

[Pete Wyckoff]

vitor.hda@gmail.com wrote on Fri, 19 Aug 2011 15:30 +0100:
> On Fri, Aug 19, 2011 at 12:53 PM, Pete Wyckoff <pw@padd.com> wrote:
> > Patches 1 and 2 are great.  We should have gotten those in way
> > back when you first submitted them.  I happily ack those.
> >
> > I'm still a bit hung up on #3, mainly because I don't get branch
> > support.  Let me play around with your test.  Having this
> > playbook of how it is supposed to work will help to educate me.
> 
> (I am probably going in too much detail here. Please skip any
> redundant information.)
> 
> In general, you can see branches in P4 as you do in SVN. They are
> simple copies from one directory to another. For example:
> 
>     p4 integrate //depot/big/directory/path/myproj/version_A \
>                  //depot/big/directory/path/myproj/version_B
> 
> Now the history of "version_B" is tied to "version_A". If you want to
> integrate (merge, in this case) new updates from one of the
> directories to the other you can just rerun the above command
> (version_A and version_B order may change depending on the direction
> of the merge).
> 
> P4 also allows you to define "branch specs" that you can use as a
> short hand instead of having to type in the full directory paths. So,
> assuming that you have a branch named "myproj_B" with the following
> view defined:
> 
>     //depot/big/directory/path/myproj/version_A \
>         //depot/big/directory/path/myproj/version_B
> 
> When you want to integrate it again you can simply type:
> 
>     p4 integrate -b myproj_B
> 
> or, if you want to integrate from version_B into version_A instead:
> 
>     p4 integrate -b myproj_B -r
> 
> git-p4 is using these branch specs to identify branches. Without this
> extra information it is nearly impossible for it to identify branches
> because an integrate can be done of a single file.
> 
> But since P4 does not strictly require branch specs and because most
> of the time integrations are done through P4V, then most of the time
> no branch specs are created. Now, if this happens git-p4 will not be
> able to detect branches! And creating branches in P4 just for git-p4
> does not make much sense, right? This is the main reason behind the
> third patch... :)

Thank you for these useful comments.  I have taken a first stab
at documenting the bevy of confusing branch flags and settings
in git-p4.  Please correct my errors and add other information
that is missing.

In particular, a clear case of how to set up the branches for
the branchList example would be good.  My minimal test did not
exactly work.


		-- Pete

--------------8<----------------------

From 36608a6e195041e8738bab24b84fadd0bf386865 Mon Sep 17 00:00:00 2001
From: Pete Wyckoff <pw@padd.com>
Date: Sat, 20 Aug 2011 15:04:09 -0400
Subject: [PATCH 4/4] git-p4: branch detection documentation start

Initial stab at documentation for --detect-branches and the
new branchList config setting.

Signed-off-by: Pete Wyckoff <pw@padd.com>
---
 contrib/fast-import/git-p4.txt |   48 +++++++++++++++++++++++++++++++++++++++-
 1 files changed, 47 insertions(+), 1 deletions(-)

diff --git a/contrib/fast-import/git-p4.txt b/contrib/fast-import/git-p4.txt
index 52003ae..6d21557 100644
--- a/contrib/fast-import/git-p4.txt
+++ b/contrib/fast-import/git-p4.txt
@@ -265,11 +265,57 @@ Only use branch specifications defined by the selected username.
 
 git-p4.branchList
 
-List of branches to be imported when branch detection is enabled.
+List of branches to be imported when branch detection is enabled.  This
+example shows that branchA and branchB are both created off branch main.
 
   git config [--global] git-p4.branchList main:branchA
   git config [--global] --add git-p4.branchList main:branchB
 
+Branch Detection
+================
+
+There are a few options that try to automatically create branches in git
+corresponding to branches in p4.
+
+Unfortunately, the concept of branch in p4 is quite different:  a p4
+branch is a mapping from one area of the repository to another.  It is
+used as input to "p4 integrate" to specify the source and destination of
+an integration.
+
+If you have a repository where all the branches are below a single
+directory, you can use "--detect-branches" when cloning or syncing to
+have git-p4 automatically look up the p4 branches and use these to
+generate branches in git.  Example:
+
+    P4 repository structure:
+
+	//depot/main/...
+	//depot/branch1/...
+    
+    View section from "p4 branch -o branch1"
+
+	//depot/main/... //depot/branch1/...
+    
+    Git clone command:
+
+	git-p4 clone --detect-branches //depot@all
+
+    Produces a separate branch in remotes/p4 for
+    //depot/main (called "master"), and one for //depot/branch1
+    (called "depot/branch1").
+
+However, it is not necessary to create branches in p4 to be able to
+use them like branches.  It is quite difficult to detect this
+automatically, but a git configuration setting "git-p4.branchList" can be
+used to explicitly identify branch relationships.  It is a list of
+"source":"destination" pairs, like a simple p4 branch specification,
+where the "source" and "destination" are the path elements in the
+p4 repository.  The example above relied on the presence of the p4
+branch.  Without that, you can get the same results by setting:
+
+  git config git-p4.branchList main:branch1
+
+
 Implementation Details...
 =========================
 
-- 
1.7.5.4

Re: [PATCH v3 0/4] git-p4: Improve branch support

[Vitor Antunes]

On Sat, Aug 20, 2011 at 8:14 PM, Pete Wyckoff <pw@padd.com> wrote:
> In particular, a clear case of how to set up the branches for
> the branchList example would be good.  My minimal test did not
> exactly work.

Could you send me the list of steps you followed that resulted in git-p4
failing to detect the branches?

Thanks,
-- 
Vitor Antunes

Re: [PATCH v3 0/4] git-p4: Improve branch support

[Junio C Hamano]

Even though this is labeled as v3, I do not see a corresponding v2 in
recent mail log. Is this an unrelated series that depends on v2 of your
"p4 rename/copy" topic?

Re: [PATCH v3 0/4] git-p4: Improve branch support

[Vitor Antunes]

On Fri, Aug 19, 2011 at 7:02 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Even though this is labeled as v3, I do not see a corresponding v2 in
> recent mail log. Is this an unrelated series that depends on v2 of your
> "p4 rename/copy" topic?

You have to search in older archives... :)

http://article.gmane.org/gmane.comp.version-control.git/167998

-- 
Vitor Antunes