Yet another git perforce integration

Yet another git perforce integration

[Simon Hausmann]

[-- Attachment #1: Type: text/plain, Size: 2430 bytes --]

Hi,

Integrations of Git with Perforce seem to be fashion these days, so I'd like 
to chip in and announce another one :)

I've written a Python script called git-p4 that allows (incremental) imports 
from Perforce as well as submitting changes back from git into a Perforce 
depot.

It is currently part of the fast-export repository that Chris Lee started at

        http://repo.or.cz/w/fast-export.git

I've been using it now for about a month for my everyday work (which requires 
Perforce), so it seems to work reasonably stable at least in my setup :)

The usage is pretty simple:

        git-p4 clone //depot/path/my/project

Syncing:

        git-p4 sync

or just

        git-p4 rebase

to sync and automatically rebase into the current branch (much like git-svn 
rebase :)

        git-p4 submit

is used to submit changes back into Perforce (use at your own risk! :)

The import itself uses git-fast-import, so it's very fast. There is also no 
need to have the imported Perforce projects in the Perforce client view as it 
just uses "p4 changes //depot/path" and "p4 
print //depot/path/file#revision", which I find very convenient and still 
fast enough for use (since of course only changed files are printed).

It also doesn't require any additional meta-data. Instead every import commit 
has a line added to the log message that contains the Perforce path the 
changeset comes from as well as the change number. git-p4 
sync/rebase "parses" this on the last commit in the "p4" git branch to find 
out where to continue importing for incremental imports.


What's still missing is a bit of cleanup. For example I'd like to put the 
import branch into refs/remotes instead of refs/heads, but I've had some 
problems with fast-import when trying that. Also the support for Perforce 
branches isn't quite working yet. I'd ideally like to for example map a 
structure like

//depot/project/main
//depot/project/branch1
//depot/project/branch2

into

remotes/p4/main
remotes/p4/branch1
remotes/p4/branch2

in git and still support incremental commits. But that isn't working yet ;(

Also I've never tried it on Windows and I expect problems as the script uses 
pipes, calls "patch", etc.

Nevertheless I hope that this tool may be interesting for others as well.
Maybe there's a possibility of including it in git/contrib/fast-import?


Simon

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Yet another git perforce integration

[Alex Riesen]

On 5/8/07, Simon Hausmann <simon@lst.de> wrote:
>
> git-p4 submit
>
> is used to submit changes back into Perforce (use at your own risk! :)

right, because you have to know where (and when) your perforce repo is,
and then where and when your copy (in Git) is, copy the p4 changes,
rebase your changes on top of the just copied changes (p4 history
is linear), try to submit, watch out for fails, because someone else could
have submitted something, possibly resync and repeat. Cursing and
swearing notwithstanding.
I simplified the operation  by promising to hit everyone who submits
anything in my domain (and was supported by the local P4 sysadms,
so it worked).

> The import itself uses git-fast-import, so it's very fast. There is also no
> need to have the imported Perforce projects in the Perforce client view as it
> just uses "p4 changes //depot/path" and "p4
> print //depot/path/file#revision", which I find very convenient and still
> fast enough for use (since of course only changed files are printed).

I confirm that, but could not use the same approach in git-p4-import
I posted earlier: I don't control the sync operation, and the stupid prog
that does the sync does not allow in-client operation (p4 sync -n).
I just have to work somehow with the junk it left in working directory

> It also doesn't require any additional meta-data. Instead every import commit
> has a line added to the log message that contains the Perforce path the
> changeset comes from as well as the change number. git-p4
> sync/rebase "parses" this on the last commit in the "p4" git branch to find
> out where to continue importing for incremental imports.

How do you handle that patchwork of mappings synced to diverse revisions
that P4 clients tend to become? Don't you have to save change number or
revision for _each_ file?

> What's still missing is a bit of cleanup. For example I'd like to put the
> import branch into refs/remotes instead of refs/heads, but I've had some
> problems with fast-import when trying that. Also the support for Perforce
> branches isn't quite working yet.

AFACS, it is impossible: Perforce does not have branches (in Git's meaning
of the word). It only has directories. Integration (it is something like
"in-repo-copy") metadata are just duct-taped on it (that stupid branchspec).

> Also I've never tried it on Windows and I expect problems as the script uses
> pipes, calls "patch", etc.

...and case-sensitivity. BTW, how does your script handle filenames with
special characters and spaces in them?

Re: Yet another git perforce integration

[Simon Hausmann]

[-- Attachment #1: Type: text/plain, Size: 4397 bytes --]

On Tuesday 08 May 2007 11:47:31 Alex Riesen wrote:
> On 5/8/07, Simon Hausmann <simon@lst.de> wrote:
> > git-p4 submit
> >
> > is used to submit changes back into Perforce (use at your own risk! :)
>
> right, because you have to know where (and when) your perforce repo is,
> and then where and when your copy (in Git) is, copy the p4 changes,
> rebase your changes on top of the just copied changes (p4 history
> is linear), try to submit, watch out for fails, because someone else could
> have submitted something, possibly resync and repeat. Cursing and
> swearing notwithstanding.

Yeah, it's a bit tricky. I'm using "p4 where" to locate the p4 checkout and 
submitting back to perforce is an interactive process for me. The user is 
presented with the entire diff for each change as well as the log message in 
an editor and can also abort/continue in-between. In real-life usage I've 
found it to be fairly smooth though (no or only few conflits), but then I 
guess it depends also on the number of people working on the depot as a whole 
at the same time :)

> > It also doesn't require any additional meta-data. Instead every import
> > commit has a line added to the log message that contains the Perforce
> > path the changeset comes from as well as the change number. git-p4
> > sync/rebase "parses" this on the last commit in the "p4" git branch to
> > find out where to continue importing for incremental imports.
>
> How do you handle that patchwork of mappings synced to diverse revisions
> that P4 clients tend to become? Don't you have to save change number or
> revision for _each_ file?

I'm not sure I understand the question. I don't really use the p4 client view 
at all. I know the last imported p4 change number from the top commit in the 
git "p4" branch. Then I use "p4 changes //depot/path/@lastchangenum,#head" to 
get the list of changes that need to be imported. For each change I look at 
the list of affected files using "p4 describe" and feed each file using "p4 
print" into git-fast-import. None of these import operations require anything 
from the client view/mapping.

Only when submitting changes back to Perforce the corresponding depot path 
must be checked out /somewhere/ (!= git repository), so that "p4 
where //depot/path" reports it. But then I just offer the user to run "p4 
sync" before applying the changes from git.

I guess the feasibility may very well depend a bit on the development model 
used with p4. I don't know if there are many models, but for example we have 
a fairly linear model where it's normal to be at the #head revision, don't 
use a whole lot of branches in Perforce. So p4 users are used to running "p4 
resolve" and I personally got used to git conflict resolving through "git 
rebase" :)

> > What's still missing is a bit of cleanup. For example I'd like to put the
> > import branch into refs/remotes instead of refs/heads, but I've had some
> > problems with fast-import when trying that. Also the support for Perforce
> > branches isn't quite working yet.
>
> AFACS, it is impossible: Perforce does not have branches (in Git's meaning
> of the word). It only has directories. Integration (it is something like
> "in-repo-copy") metadata are just duct-taped on it (that stupid
> branchspec).

True, it probably depends a bit on the depot organization. For example we have
//depot/qt/main and then branches like //depot/qt/4.3 or research branches 
like //depot/qt/research/somecoolfeature . That's the kind of structure I'd 
like to map to git, and merges /could/ be detected from integrations if after 
a commit that has integrations between these "known" branches all revisions 
of all files are integrated. But that's obviously a very slow operation and I 
currently don't think it's worth it. But of course detecting the initial 
branch point is easy and fast and that is worth it (to record the branch 
creation, there's no merge of course).

> > Also I've never tried it on Windows and I expect problems as the script
> > uses pipes, calls "patch", etc.
>
> ...and case-sensitivity. BTW, how does your script handle filenames with
> special characters and spaces in them?

I don't really treat them special. I get the file name from the output of "p4 
describe" (using the Guido option :) and pass it on to git-fast-import.

Simon

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Yet another git perforce integration

[Alex Riesen]

On 5/8/07, Simon Hausmann <simon@lst.de> wrote:
> > > It also doesn't require any additional meta-data. Instead every import
> > > commit has a line added to the log message that contains the Perforce
> > > path the changeset comes from as well as the change number. git-p4
> > > sync/rebase "parses" this on the last commit in the "p4" git branch to
> > > find out where to continue importing for incremental imports.
> >
> > How do you handle that patchwork of mappings synced to diverse revisions
> > that P4 clients tend to become? Don't you have to save change number or
> > revision for _each_ file?
>
> I'm not sure I understand the question. I don't really use the p4 client view
> at all. ...

Ah, you're _that_ lucky...

> > > problems with fast-import when trying that. Also the support for Perforce
> > > branches isn't quite working yet.
> >
> > AFACS, it is impossible: Perforce does not have branches (in Git's meaning
> > of the word). It only has directories. Integration (it is something like
> > "in-repo-copy") metadata are just duct-taped on it (that stupid
> > branchspec).
>
> True, it probably depends a bit on the depot organization. For example we have
> //depot/qt/main and then branches like //depot/qt/4.3 or research branches
> like //depot/qt/research/somecoolfeature . That's the kind of structure I'd
> like to map to git, ...

Recreating it anywhere, not just in Git, would be stupid. As it is in Perforce.

> > > Also I've never tried it on Windows and I expect problems as the script
> > > uses pipes, calls "patch", etc.
> >
> > ...and case-sensitivity. BTW, how does your script handle filenames with
> > special characters and spaces in them?
>
> I don't really treat them special. I get the file name from the output of "p4
> describe" (using the Guido option :) and pass it on to git-fast-import.
>

You seem to assume that the names never contain double quotes and
backslashes. For example in the names of directories.