P4 Sync problem

P4 Sync problem

[Dmitry Smirnov]

Hi,

I'm trying to import Perfoce client into Git repository.
I had configured git-p4.clent=MYCLIENT and git-p4.useclientspec=true.

When runnign git p4 sync --verbose I got the follwing:

c:\p4\views\Git\p4client>git p4 sync --verbose
Reading pipe: git config git-p4.useclientspec
Reading pipe: git config git-p4.user
Reading pipe: git config git-p4.password
Reading pipe: git config git-p4.port
Reading pipe: git config git-p4.host
Reading pipe: git config git-p4.client
p4 -c MYCLIENT -G client -o
Opening pipe: p4 -c MYCLIENT -G client -o
Reading pipe: git rev-parse --symbolic  --remotes
branches: []
Getting p4 changes for ...
Traceback (most recent call last):
  File "/usr/sbin/git-core//git-p4", line 1929, in ?
    main()
  File "/usr/sbin/git-core//git-p4", line 1924, in main
    if not cmd.run(args):
  File "/usr/sbin/git-core//git-p4", line 1676, in run
    changes = p4ChangesForPaths(self.depotPaths, self.changeRange)
  File "/usr/sbin/git-core//git-p4", line 442, in p4ChangesForPaths
    assert depotPaths
AssertionError

I would appreciate if someone help to solve this problem.

Dmitry

Re: P4 Sync problem

[Tor Arvid Lund]

On Tue, Nov 10, 2009 at 2:56 PM, Dmitry Smirnov <divis1969@gmail.com> wrote:
> Hi,
>
> I'm trying to import Perfoce client into Git repository.
> I had configured git-p4.clent=MYCLIENT and git-p4.useclientspec=true.
>
> When runnign git p4 sync --verbose I got the follwing:
>
> c:\p4\views\Git\p4client>git p4 sync --verbose
<snip>
> Traceback (most recent call last):
>  File "/usr/sbin/git-core//git-p4", line 1929, in ?
>    main()
>  File "/usr/sbin/git-core//git-p4", line 1924, in main
>    if not cmd.run(args):
>  File "/usr/sbin/git-core//git-p4", line 1676, in run
>    changes = p4ChangesForPaths(self.depotPaths, self.changeRange)
>  File "/usr/sbin/git-core//git-p4", line 442, in p4ChangesForPaths
>    assert depotPaths
> AssertionError

Hi. So - I think the problem is that git-p4 doesn't understand what it
is you want to sync. The git-p4.useclientspec flag was created for the
purpose where your perforce depot may look like this:

//depot/project1
//depot/project1/source_code
//depot/project1/documentation
//depot/project1/some_large_collection_of_binaries

Then - if I set up my client spec like:

//depot/...
-//depot/project1/some_large_collection_of_binaries

... and do

git p4 sync //depot/project1@all

... it should get all project1 files except the
"some_large_collection_of_binaries" subdirectory (provided that you
have set the git-p4.client and git-p4.useclientspec).

-Tor Arvid-

Re: P4 Sync problem

[Dmitry Smirnov]

Tor Arvid Lund <torarvid <at> gmail.com> writes:

> Hi. So - I think the problem is that git-p4 doesn't understand what it
> is you want to sync. 

I believe it will never understand that until self.depotPaths 
will be initialized.
It looks like it is intitilized from argument list only (or from already saved 
settings). See P4Sync.run(). 
But self.getClientSpec() which is called some time before, does not init neither 
self.depotPaths nor args.

I had fixed this with the following code (new lines marked with +):
        if self.useClientSpec or gitConfig("git-p4.useclientspec") == "true":
+            if self.verbose:
+                print "Get client spec"
            self.getClientSpec()
+            if self.verbose:
+                print "Client Spec Dirs: %s" % self.clientSpecDirs
+            if len(args) == 0:
+                for item in self.clientSpecDirs:
+                      k,v = item
+                      args.append(k)

Unfortunately, this fails on another stage:

>git p4 sync --verbose
Reading pipe: git config git-p4.useclientspec
Get client spec
Reading pipe: git config git-p4.user
Reading pipe: git config git-p4.password
Reading pipe: git config git-p4.port
Reading pipe: git config git-p4.host
Reading pipe: git config git-p4.client
p4 -c MYCLIENT-G client -o
Opening pipe: p4 -c MYCLIENT-G client -o

Client Spec Dirs:[ <my paths here> ]
Doing initial import of  <my paths here> from revision #head into 
refs/remotes/p4/master
p4 -c MYCLIENT-G files <my paths here appended with ...#head >
Opening pipe: p4 -c MYCLIENT-G files <my paths here appended with ...#head >
p4 returned an error: //MYCLIENT/path/file.c/...#head> - file(s) not in client 
view.

My client spec contains some line to the file:
-//depot/path/... //MYCLIENT/null/...
//depot/path/file.cs //MYCLIENT/path/file.cs

This means that I wish to sync only file.cs into my client and get rid of other 
files in this directory.
BTW, note that file extension is truncated in the log


Dmitry

Re: P4 Sync problem

[Tor Arvid Lund]

On Wed, Nov 11, 2009 at 10:43 AM, Dmitry Smirnov <divis1969@gmail.com> wrote:
> Tor Arvid Lund <torarvid <at> gmail.com> writes:
>
>> Hi. So - I think the problem is that git-p4 doesn't understand what it
>> is you want to sync.
>
> I believe it will never understand that until self.depotPaths
> will be initialized.

Correct.

> It looks like it is intitilized from argument list only (or from already saved
> settings). See P4Sync.run().

Also correct, me thinks.

> But self.getClientSpec() which is called some time before, does not init neither
> self.depotPaths nor args.

Correct again. The useclientspec flag was created (by me) as a simple
way to filter out files that I did not want to download (the reason
being my company tends to mix binaries with source files causing
unnecessary long sync times). So it was never used to tell git-p4
_what_ to sync, but rather what to _not_ sync.

So - in your case, you could have tried to just say "git p4 sync
//depot@all", and if your clientspec contained, say, only
"//depot/path/projectX", then projectX should be the only thing that
got downloaded. Keep in mind, though, that git-p4 would still ask the
p4 server for _all_ of its changelists, loop through them, and check
each file in each changelist against the clientSpecDirs array...

> I had fixed this with the following code (new lines marked with +):
>        if self.useClientSpec or gitConfig("git-p4.useclientspec") == "true":
> +            if self.verbose:
> +                print "Get client spec"
>            self.getClientSpec()
> +            if self.verbose:
> +                print "Client Spec Dirs: %s" % self.clientSpecDirs
> +            if len(args) == 0:
> +                for item in self.clientSpecDirs:
> +                      k,v = item
> +                      args.append(k)

Well, I see what you're trying to do, but I would not want to see that
patch in the official script, because some (most?) people (myself, at
least) use git-p4 to clone single projects out of a perforce depot
that may contain many projects. I do this myself by doing:

git p4 clone //depot/path/to/projectX@all

I usually use one clientspec in perforce, and I do not want to change
that... With your patch, I would be in trouble since my clientspec
contains "//depot/..." (followed by a lot of lines starting with '-')

> Unfortunately, this fails on another stage:
>
>>git p4 sync --verbose
<snip>
> Opening pipe: p4 -c MYCLIENT-G files <my paths here appended with ...#head >
> p4 returned an error: //MYCLIENT/path/file.c/...#head> - file(s) not in client
> view.
>
> My client spec contains some line to the file:
> -//depot/path/... //MYCLIENT/null/...
> //depot/path/file.cs //MYCLIENT/path/file.cs
>
> This means that I wish to sync only file.cs into my client and get rid of other
> files in this directory.
> BTW, note that file extension is truncated in the log

Yep - this is a bug (or lack of a feature, if you will...). The
clientspec functionality in git-p4 supports directories only. If you
look at the getClientSpec function, you see that it looks for "...".
It's probably an easy fix if you want to support having single files
in the client spec. I didn't (and don't) need it, so it didn't occur
to me at the time...

If you want to fix it, you might want to rename clientSpecDirs to
clientSpecEntries or something like that.

Btw... Am I understanding correctly what it is you wish to accomplish?
I'm guessing that you have a perforce server with a client spec set
up, and you want to sync everything on the entire server according to
that client spec?

-Tor Arvid-

Re: P4 Sync problem

[Dmitry Smirnov]

Tor Arvid Lund <torarvid <at> gmail.com> writes:


> Well, I see what you're trying to do, but I would not want to see that
> patch in the official script, because some (most?) people (myself, at
> least) use git-p4 to clone single projects out of a perforce depot
> that may contain many projects. I do this myself by doing:
> 
> git p4 clone //depot/path/to/projectX <at> all
> 
> I usually use one clientspec in perforce, and I do not want to change
> that... With your patch, I would be in trouble since my clientspec
> contains "//depot/..." (followed by a lot of lines starting with '-')

Well, does this mean that if you try to sync the client in perforce (visual or 
command line), you will sync all the projects?

In that case, git p4 will require significant effort to satisfy both of us :-) 
Unfortunatly, it seems I'm in minory group of git-p4 users...

i would propose to use both command-line arguments and a client spec 
to create a correct filter of what should be synced/cloned.
BTW, it looks this script does not honor neither the order of paths 
in the spec (which can be important) nor mapping of the files to a local tree.
 

> If you want to fix it, you might want to rename clientSpecDirs to
> clientSpecEntries or something like that.

For now, I just commented out two lines in the run() procedure:
            #if not p.endswith("/"):
            #    p += "/"

> Btw... Am I understanding correctly what it is you wish to accomplish?
> I'm guessing that you have a perforce server with a client spec set
> up, and you want to sync everything on the entire server according to
> that client spec?

yes. Client spec completely defines the project layout for me. 
It contains paths to some components that are mapped to the 
client working tree.
Just if your CS contain 

//depot/path/to/projectX/... //CLIENT/...

Re: P4 Sync problem

[Tor Arvid Lund]

On Wed, Nov 11, 2009 at 1:39 PM, Dmitry Smirnov <divis1969@gmail.com> wrote:
> Tor Arvid Lund <torarvid <at> gmail.com> writes:
>
>
>> Well, I see what you're trying to do, but I would not want to see that
>> patch in the official script, because some (most?) people (myself, at
>> least) use git-p4 to clone single projects out of a perforce depot
>> that may contain many projects. I do this myself by doing:
>>
>> git p4 clone //depot/path/to/projectX <at> all
>>
>> I usually use one clientspec in perforce, and I do not want to change
>> that... With your patch, I would be in trouble since my clientspec
>> contains "//depot/..." (followed by a lot of lines starting with '-')
>
> Well, does this mean that if you try to sync the client in perforce (visual or
> command line), you will sync all the projects?

Yes, but I generally try to not use perforce, but git instead :-)

> In that case, git p4 will require significant effort to satisfy both of us :-)
> Unfortunatly, it seems I'm in minory group of git-p4 users...

I don't know, but if I were to guess, then yes, you probably are... If
you have a complex perforce client spec setup, then there may of
course be problems that git-p4 might not solve for you. Since nobody
has volunteered to implement the features you describe yet, I believe
that most of us git-p4 users have fairly simple client spec setups.

For me, most projects in p4 are such that I can give one root
directory to "git p4 sync", and it works for me. I of course have
several git projects that sync from the same p4 server (only with
different root dirs). In cases where you have dependencies between
such projects, you should maybe read about git submodules - or maybe
googles "repo" script (search for "google repo git"). I don't know
much about any of these, other than 'they exist, and seemingly try to
solve such issues' :-/

> i would propose to use both command-line arguments and a client spec
> to create a correct filter of what should be synced/cloned.
> BTW, it looks this script does not honor neither the order of paths
> in the spec (which can be important) nor mapping of the files to a local tree.

When you have a client spec like:

//depot/A/...
-//depot/A/B/...
//depot/A/B/C/...

... git-p4 sorts these paths by length. For a given filename, it finds
the longest path that matches that files directory, and if that path
starts with a '-', the file is not synced (for a file
"//depot/A/B/myfile.c" it gets a match on "-//depot/A/B/...", and
myfile.c is not synced, but the file "//depot/A/B/C/myotherfile.c" it
matches "//depot/A/B/C/...")

Do you have an example that shows how it might fail?

And no, git-p4 does not care about the local mappings, it reads only
the server part.

-Tor Arvid-

>> If you want to fix it, you might want to rename clientSpecDirs to
>> clientSpecEntries or something like that.
>
> For now, I just commented out two lines in the run() procedure:
>            #if not p.endswith("/"):
>            #    p += "/"
>
>> Btw... Am I understanding correctly what it is you wish to accomplish?
>> I'm guessing that you have a perforce server with a client spec set
>> up, and you want to sync everything on the entire server according to
>> that client spec?
>
> yes. Client spec completely defines the project layout for me.
> It contains paths to some components that are mapped to the
> client working tree.
> Just if your CS contain
>
> //depot/path/to/projectX/... //CLIENT/...
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: P4 Sync problem

[Dmitry Smirnov]

Tor Arvid Lund <torarvid <at> gmail.com> writes:

> //depot/A/...
> -//depot/A/B/...
> //depot/A/B/C/...
> 
> ... git-p4 sorts these paths by length. For a given filename, it finds
> the longest path that matches that files directory, and if that path
> starts with a '-', the file is not synced (for a file
> "//depot/A/B/myfile.c" it gets a match on "-//depot/A/B/...", and
> myfile.c is not synced, but the file "//depot/A/B/C/myotherfile.c" it
> matches "//depot/A/B/C/...")
> 
> Do you have an example that shows how it might fail?

According to the P4 User's Guide 
(http://www.perforce.com/perforce/doc.091/manuals/p4guide/02_config.html#1066090) 
Perforce prefer later mapping if there is a conflict.

So, if you switch paths:
-//depot/A/B/...
//depot/A/...
perforce will sync all the the files. Of course, this is not a very 
useful case.  
I suppose the order is not important until mapping are implemented...

Re: P4 Sync problem

[masonk]

Hi, I know this is an old thread, but I think it might be relevant.  (At any
rate, at least you know I've been looking).

I'm trying to use git-p4 with a very large P4 repository, of which my client
view is a small fraction.  
My client view is within the administrator-set file limit of P4 files. 
E.G., p4 sync //depot/... works.

When I call git-p4 sync or clone, however, it bumps into the limit.  Even if
I am using client spec and detecting branches.  If p4 were successfully
detecting my branches -or- using my client spec, I would not be running into
the limit.  But:

git p4 sync //depot@all --use-client-spec --detect-branches --verbose
Syncing with origin first by calling git fetch origin
executing git fetch origin
Reading pipe: git config git-p4.user
Reading pipe: git config git-p4.password
Reading pipe: git config git-p4.port
Reading pipe: git config git-p4.host
Reading pipe: git config git-p4.client
p4 -u mkramer -c mkramer -G client -o
Opening pipe: p4 -u mkramer -c mkramer -G client -o
Reading pipe: git rev-parse --symbolic  --remotes
Reading pipe: git rev-parse p4/master
p4-git branches: []
initial parents: {}
Getting p4 changes for //depot/...
p4 -u mkramer -c mkramer changes //depot/...
Reading pipe: p4 -u mkramer -c mkramer changes //depot/...
Request too large (over 110000); see 'p4 help maxresults'.
Traceback (most recent call last):
  File "/home/mkramer/git/bin/git-p4", line 1924, in <module>
    main()
  File "/home/mkramer/git/bin/git-p4", line 1919, in main
    if not cmd.run(args):
  File "/home/mkramer/git/bin/git-p4", line 1671, in run
    changes = p4ChangesForPaths(self.depotPaths, self.changeRange)
  File "/home/mkramer/git/bin/git-p4", line 444, in p4ChangesForPaths
    for p in depotPaths]))
  File "/home/mkramer/git/bin/git-p4", line 108, in p4_read_pipe_lines
    return read_pipe_lines(real_cmd)
  File "/home/mkramer/git/bin/git-p4", line 101, in read_pipe_lines
    die('Command failed: %s' % c)
  File "/home/mkramer/git/bin/git-p4", line 59, in die
    raise Exception(msg)
Exception: Command failed: p4 -u mkramer -c mkramer changes //depot/...


Is this a limitation in the current --use-client-spec or --detect-branches
flags, or am I doing it wrong?
-- 
View this message in context: http://git.661346.n2.nabble.com/P4-Sync-problem-tp3979793p5326585.html
Sent from the git mailing list archive at Nabble.com.

Re: P4 Sync problem

[Pete Wyckoff]

divis1969@gmail.com wrote on Wed, 11 Nov 2009 12:39 +0000:
> i would propose to use both command-line arguments and a client spec 
> to create a correct filter of what should be synced/cloned.
> BTW, it looks this script does not honor neither the order of paths 
> in the spec (which can be important) nor mapping of the files to a local tree.

Here's a hack that I've been using to at least read the repo
locations out of the client spec.  In the useclientspec case, it
takes all the info on paths from an existing p4 client spec.  No
command line argument to git p4 sync.  Hack out the test for P4ENV;
that is site specific for me.

We've moved away from using client specs with more than 1 line, so
I'm not interested in this patch anymore.  Also, the long term
solution will probably be based on the foreign remote work that
Daniel and others are doing, so don't invest too much time in fixing
this up.  That said, if there is a clean way to support both your
client specs and how Tor and others use them, it is worth putting
in now.

Good luck.

		-- Pete

>From 55e8f6323894031119c755f2c3b3214c1c74b824 Mon Sep 17 00:00:00 2001
From: Pete Wyckoff <pw@padd.com>
Date: Wed, 26 Nov 2008 12:28:09 -0500
Subject: [PATCH] honor git client spec

Destination directories for parts of the depot are specified in the
client spec.  Use them as given.  Also read the entire client spec to
figure out what to do.
---
 contrib/fast-import/git-p4 |  108 +++++++++++++++++++++++++++++++++++---------
 1 files changed, 86 insertions(+), 22 deletions(-)

diff --git a/contrib/fast-import/git-p4 b/contrib/fast-import/git-p4
index 1cecae2..28fa95f 100755
--- a/contrib/fast-import/git-p4
+++ b/contrib/fast-import/git-p4
@@ -456,6 +456,26 @@ def p4ChangesForPaths(depotPaths, changeRange):
     changelist.sort()
     return changelist
 
+#
+# Sort by number of slashes first:  more specific at the top.  Then
+# sort by alpha within a given number of path components.
+#
+def clientSortFunc(a, b):
+    asrc = a[0]
+    bsrc = b[0]
+    asrclen = asrc.count("/")
+    bsrclen = bsrc.count("/")
+    if asrclen > bsrclen:
+	return -1
+    elif asrclen < bsrclen:
+	return 1
+    elif asrc > bsrc:
+	return 1
+    elif asrc < bsrc:
+	return -1
+    else:
+	return 0
+
 class Command:
     def __init__(self):
         self.usage = "usage: %prog [options]"
@@ -915,6 +935,11 @@ class P4Sync(Command):
         return files
 
     def stripRepoPath(self, path, prefixes):
+	if self.clientSpecDirs:
+	    for val in self.clientSpecDirs:
+		if path.startswith(val[0]):
+		    return val[1] + path[len(val[0]):]
+
         if self.keepRepoPath:
             prefixes = [re.sub("^(//[^/]+/).*", r'\1', prefixes[0])]
 
@@ -1038,7 +1063,7 @@ class P4Sync(Command):
             includeFile = True
             for val in self.clientSpecDirs:
                 if f['path'].startswith(val[0]):
-                    if val[1] <= 0:
+                    if val[1] == '-':
                         includeFile = False
                     break
 
@@ -1115,6 +1140,7 @@ class P4Sync(Command):
             self.gitStream.write("from %s\n" % parent)
 
         self.streamP4Files(new_files)
+
         self.gitStream.write("\n")
 
         change = int(details["change"])
@@ -1476,24 +1502,54 @@ class P4Sync(Command):
 
 
     def getClientSpec(self):
-        specList = p4CmdList( "client -o" )
+        specList = p4CmdList("client -o")
         temp = {}
+	client = ""
+        for entry in specList:
+            for k,v in entry.iteritems():
+		if k.startswith("Client"):
+		    client = v
+		    print "client is", client
+	if not client:
+	    sys.stderr.write("no client found\n")
+	    sys.exit(1)
+	client = "//" + client + "/"
         for entry in specList:
             for k,v in entry.iteritems():
                 if k.startswith("View"):
-                    if v.startswith('"'):
-                        start = 1
-                    else:
-                        start = 0
-                    index = v.find("...")
-                    v = v[start:index]
-                    if v.startswith("-"):
-                        v = v[1:]
-                        temp[v] = -len(v)
-                    else:
-                        temp[v] = len(v)
-        self.clientSpecDirs = temp.items()
-        self.clientSpecDirs.sort( lambda x, y: abs( y[1] ) - abs( x[1] ) )
+		    if v.startswith('"'):
+			v = v[1:]
+		    if v.endswith('"'):
+			v = v[:-1]
+		    d = v.split(" ");
+		    if len(d) != 2:
+			sys.stderr.write( \
+			    "expecting two fields in view, got: %s\n" % v)
+			sys.exit(1)
+		    if not d[0].endswith("..."):
+			sys.stderr.write(\
+			    "expecting trailing ..., got: %s\n" % d[0])
+			sys.exit(1)
+		    d[0] = d[0][:-3]
+		    if not d[1].endswith("..."):
+			sys.stderr.write(\
+			    "expecting trailing ..., got: %s\n" % d[1])
+			sys.exit(1)
+		    d[1] = d[1][:-3]
+		    if not d[1].startswith(client):
+			sys.stderr.write(\
+			    "expecting dest to start with %s, got: %s\n" % \
+			    (client, d[1]))
+			sys.exit(1)
+		    d[1] = d[1][len(client):]
+		    # negated items do not appear in tree
+		    if d[0].startswith("-"):
+			d[0] = d[0][1:]
+			d[1] = ""
+		    temp[d[0]] = d[1]
+
+	self.clientSpecDirs = temp.items()
+	self.clientSpecDirs.sort(clientSortFunc)
 
     def run(self, args):
         self.depotPaths = []
@@ -1755,7 +1811,7 @@ class P4Clone(P4Sync):
     def __init__(self):
         P4Sync.__init__(self)
         self.description = "Creates a new git repository and imports from Perforce into it"
-        self.usage = "usage: %prog [options] //depot/path[@revRange]"
+        self.usage = "usage: %prog [options] [//depot/path[@revRange]]"
         self.options += [
             optparse.make_option("--destination", dest="cloneDestination",
                                  action='store', default=None,
@@ -1783,18 +1839,26 @@ class P4Clone(P4Sync):
         return os.path.split(depotDir)[1]
 
     def run(self, args):
-        if len(args) < 1:
-            return False
-
         if self.keepRepoPath and not self.cloneDestination:
             sys.stderr.write("Must specify destination for --keep-path\n")
             sys.exit(1)
 
         depotPaths = args
 
-        if not self.cloneDestination and len(depotPaths) > 1:
-            self.cloneDestination = depotPaths[-1]
-            depotPaths = depotPaths[:-1]
+	if gitConfig("git-p4.useclientspec") == "true":
+	    if not os.path.exists("P4ENV"):
+		sys.stderr.write("Must copy P4ENV file from a valid client\n")
+		sys.exit(1)
+	    self.getClientSpec()
+	    if not depotPaths:
+		depotPaths = [p[0] for p in self.clientSpecDirs]
+	else:
+	    if not depotPaths:
+		sys.stderr.write("Must specify depot path if no client spec\n")
+		sys.exit(1)
+
+        if not self.cloneDestination:
+		self.cloneDestination = "."
 
         self.cloneExclude = ["/"+p for p in self.cloneExclude]
         for p in depotPaths:
-- 
1.6.2.5