Importing Bzr revisions

Importing Bzr revisions

[David Reitter]

[-- Attachment #1: Type: text/plain, Size: 243 bytes --]

Suppose I have a bzr branch that has been converted (somehow) to a git  
branch, is it then possible to merge new revisions from the bzr branch  
into the git one?
How would I go about doing that?

Thanks
- David

PS.: please cc me in replies.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2193 bytes --]

Re: Importing Bzr revisions

[Junio C Hamano]

David Reitter <david.reitter@gmail.com> writes:

> Suppose I have a bzr branch that has been converted (somehow) to a git
> branch, is it then possible to merge new revisions from the bzr branch
> into the git one?

It entirely depends on how that "somehow" goes.

If that "somehow" procedure performs a reliably reproducible conversion
(i.e. not only it will produce the identical git history when you feed the
same bzr history to the procedure twice, but it will produce the identical
git history followed by new history if you feed the bzr history after new
commits are added to the bzr history), you should be able to re-convert
the updated bzr history to git and merge the result with the result of the
earlier conversion.  The re-conversion process may not even have to be a
whole re-conversion; it could be incremental.  But that is entirely up to
the quality of the conversion "somehow" procedure implements.

Re: Importing Bzr revisions

[David Reitter]

[-- Attachment #1: Type: text/plain, Size: 1130 bytes --]

On Mar 23, 2009, at 4:06 AM, Junio C Hamano wrote:

> David Reitter <david.reitter@gmail.com> writes:
>
>> Suppose I have a bzr branch that has been converted (somehow) to a  
>> git
>> branch, is it then possible to merge new revisions from the bzr  
>> branch
>> into the git one?
>
> It entirely depends on how that "somehow" goes.
>
> If that "somehow" procedure performs a reliably reproducible  
> conversion
> (i.e. not only it will produce the identical git history when you  
> feed the

The question is then:  How would one reliably convert a Bzr branch to  
git?  One branch is enough for me, but we're talking >100k revisions,  
so if the procedure involves Bzr tools, it will take longer then is  
practicable.

Also, my impression is that not many people are doing this, because  
the fast-export/import combination hasn't seen a lot of development  
activity.  I'm a bit wary to set up my project in git when I don't  
just need to convert once, but in future convert and merge pretty much  
every day...  (The upstream project is going to use Bzr, while I am  
trying to see if Git is an option for me downstream.)

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2193 bytes --]

Re: Importing Bzr revisions

[David Reitter]

[-- Attachment #1: Type: text/plain, Size: 1267 bytes --]

On Mar 23, 2009, at 4:06 AM, Junio C Hamano wrote:
>
>> Suppose I have a bzr branch that has been converted (somehow) to a  
>> git
>> branch, is it then possible to merge new revisions from the bzr  
>> branch
>> into the git one?
>
> It entirely depends on how that "somehow" goes.
>
> If that "somehow" procedure performs a reliably reproducible  
> conversion
> (i.e. not only it will produce the identical git history when you  
> feed the
> same bzr history to the procedure twice, but it will produce the  
> identical
> git history followed by new history if you feed the bzr history  
> after new
> commits are added to the bzr history), you should be able to re- 
> convert

I'm just experimenting with "bzr fast-export", which converts to git,  
and it seems to take about 4 minutes for 1000 revisions on our  
(modern) server.  That would be around 7 hours for my emacs  
repository; I can't do that daily.

I wonder if there's a way for (bzr) fast-export / (git) fast-import to  
work incrementally, i.e. for selected or most recent revisions.

Or, one could do something like  bzr diff -r $REV.. $BBRANCH |  (cd  
$GBRANCH; patch -p0; git commit), plus preserving authors and log  
messages.   Is this roughly what the fast-export format does anyways?


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2193 bytes --]

Re: Importing Bzr revisions

[Shawn O. Pearce]

David Reitter <david.reitter@gmail.com> wrote:
>
> I'm just experimenting with "bzr fast-export", which converts to git,  
> and it seems to take about 4 minutes for 1000 revisions on our (modern) 
> server.  That would be around 7 hours for my emacs repository; I can't do 
> that daily.

No, you'd want to incrementally do that...

> I wonder if there's a way for (bzr) fast-export / (git) fast-import to  
> work incrementally, i.e. for selected or most recent revisions.

fast-import supports incremental use; git-p4 does it from Perforce.
The trick is the application writing the stream (bzr fast-export
in this case) needs to do something to pick up the prior revisions.
It might do that by using the same mark numbers, and requiring you
to use --import-marks and --export-marks on the git side to save the
mark database between runs.  I don't know, I haven't looked at it.

> Or, one could do something like  bzr diff -r $REV.. $BBRANCH |  (cd  
> $GBRANCH; patch -p0; git commit), plus preserving authors and log  
> messages.   Is this roughly what the fast-export format does anyways?

Eh, sort of.

The fast-import format works on whole files, not patches.  So we
have to get the entire file from bzr each time it is modified.
If the file is a small source file, you almost can't tell the
difference in performance.  If its a huge binary that changes often,
it hurts to keep dumping the entire thing over the stream.

But at the commit level, yes, it preserves authorship and log
messages, assuming the bzr fast-export program incldued that data.
And I'm rather certain it does.

-- 
Shawn.

Re: Importing Bzr revisions

[Teemu Likonen]

[I'll Cc to Bazaar list too because this subject is half-Git and
half-Bazaar and we may need help from both sides.]


On 2009-03-23 09:27 (-0400), David Reitter wrote:

> I'm just experimenting with "bzr fast-export", which converts to git,
> and it seems to take about 4 minutes for 1000 revisions on our
> (modern) server. That would be around 7 hours for my emacs repository;
> I can't do that daily.
>
> I wonder if there's a way for (bzr) fast-export / (git) fast-import to
> work incrementally, i.e. for selected or most recent revisions.

They can or should work incrementally, and actually I have succesfully
done that. The idea is to use --import-marks and --export-marks options
with "git fast-import" and --marks option with "bzr fast-export.

I noticed some problems with newer versions of "bzr fast-export", though
(since it was converted to a proper Bzr command). It seems to corrupt
the marks file when doing the first incremental export after the initial
export. At least the revisions are not in right order in the marks file
anymore. "git fast-import" can't continue to import from the same
revision where it left last time and it seems to create alternative
history -- or something.

Really I don't know if this is a bug in Bzr or in Git and haven't
figured out how to file a useful bug report.

Re: Importing Bzr revisions

[Pieter de Bie]

On Mar 23, 2009, at 1:27 PM, David Reitter wrote:

> On Mar 23, 2009, at 4:06 AM, Junio C Hamano wrote:
>>
>>> Suppose I have a bzr branch that has been converted (somehow) to a  
>>> git
>>> branch, is it then possible to merge new revisions from the bzr  
>>> branch
>>> into the git one?
>>
>> It entirely depends on how that "somehow" goes.
>>
>> If that "somehow" procedure performs a reliably reproducible  
>> conversion
>> (i.e. not only it will produce the identical git history when you  
>> feed the
>> same bzr history to the procedure twice, but it will produce the  
>> identical
>> git history followed by new history if you feed the bzr history  
>> after new
>> commits are added to the bzr history), you should be able to re- 
>> convert
>
> I'm just experimenting with "bzr fast-export", which converts to  
> git, and it seems to take about 4 minutes for 1000 revisions on our  
> (modern) server.  That would be around 7 hours for my emacs  
> repository; I can't do that daily.
>
> I wonder if there's a way for (bzr) fast-export / (git) fast-import  
> to work incrementally, i.e. for selected or most recent revisions.
>
> Or, one could do something like  bzr diff -r $REV.. $BBRANCH |  (cd  
> $GBRANCH; patch -p0; git commit), plus preserving authors and log  
> messages.   Is this roughly what the fast-export format does anyways?


You might want to take a look at git-bzr (http://github.com/pieter/git-bzr/tree/master 
) it allows incremental bidirectional interaction between git and bzr  
using the fast-export/import, so it might just work in your case.  
There are some issues with it, so you might want to check the  
'network' part on github and use one of the other variants.

That said, it's a 100 line script that hasn't been used much, so good  
luck :)

- Pieter

Re: Importing Bzr revisions

[David Reitter]

[-- Attachment #1: Type: text/plain, Size: 961 bytes --]

On Mar 23, 2009, at 10:18 AM, Pieter de Bie wrote:
>
> You might want to take a look at git-bzr (http://github.com/pieter/git-bzr/tree/master 
> ) it allows incremental bidirectional interaction between git and  
> bzr using the fast-export/import, so it might just work in your  
> case. There are some issues with it, so you might want to check the  
> 'network' part on github and use one of the other variants.
>
> That said, it's a 100 line script that hasn't been used much, so  
> good luck :)

Thanks, this appears to be useful.

AFter reading your script, I tried incremental exports on the bzr  
side: It still supports --import-marks and this seems to work quickly  
enough for my small test project.

Could you update the documentation as to how to install the git-bzr  
script?
(I searched for "plugin" in the git user manual, and google, and  
didn't find anything.  I haven't used git much, so I'm a bit naive as  
regards to such a question.)



[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2193 bytes --]

Re: Importing Bzr revisions

[Pieter de Bie]

On 23 mrt 2009, at 15:18, David Reitter wrote:

> Could you update the documentation as to how to install the git-bzr  
> script?
> (I searched for "plugin" in the git user manual, and google, and  
> didn't find anything.  I haven't used git much, so I'm a bit naive  
> as regards to such a question.)

There's nothing to install, just call the script.. if you run it  
without commands it should give some usage information. The README in  
the repository should tell the rest. If you'd like to call it using  
'git bzr' rather than 'git-bzr', you have to put it somewhere in your  
PATH

Re: Importing Bzr revisions

[David Reitter]

[-- Attachment #1: Type: text/plain, Size: 1024 bytes --]

On Mar 23, 2009, at 4:15 PM, Pieter de Bie wrote:
>> Could you update the documentation as to how to install the git-bzr  
>> script?
>> (I searched for "plugin" in the git user manual, and google, and  
>> didn't find anything.  I haven't used git much, so I'm a bit naive  
>> as regards to such a question.)
>
> There's nothing to install, just call the script.. if you run it  
> without commands it should give some usage information. The README  
> in the repository should tell the rest. If you'd like to call it  
> using 'git bzr' rather than 'git-bzr', you have to put it somewhere  
> in your PATH

Sure, okay, thanks.  (In my bzr mindset I didn't realize that "git  
bzr" just elegantly translated to "git-bzr".)

I'm currently attempting to convert those 100k revisions from the  
emacs repository, averaging 5.5min/1000 revisions.  Judging from CPU  
load (if that's a valid thing to do), I see a 9:1 distribution of the  
workload (bzr, reading, vs. git, writing).

Thanks for helping me out with this so far.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2193 bytes --]

Re: Importing Bzr revisions

[David Reitter]

[-- Attachment #1: Type: text/plain, Size: 4633 bytes --]

On Mar 23, 2009, at 10:07 AM, Teemu Likonen wrote:

>> I'm just experimenting with "bzr fast-export", which converts to git,
>> and it seems to take about 4 minutes for 1000 revisions on our
>> (modern) server. That would be around 7 hours for my emacs  
>> repository;
>> I can't do that daily.
>>
>> I wonder if there's a way for (bzr) fast-export / (git) fast-import  
>> to
>> work incrementally, i.e. for selected or most recent revisions.
>
> They can or should work incrementally, and actually I have succesfully
> done that. The idea is to use --import-marks and --export-marks  
> options
> with "git fast-import" and --marks option with "bzr fast-export.
>
> I noticed some problems with newer versions of "bzr fast-export",  
> though
> (since it was converted to a proper Bzr command). It seems to corrupt
> the marks file when doing the first incremental export after the  
> initial
> export. At least the revisions are not in right order in the marks  
> file
> anymore. "git fast-import" can't continue to import from the same
> revision where it left last time and it seems to create alternative
> history -- or something.
>
> Really I don't know if this is a bug in Bzr or in Git and haven't
> figured out how to file a useful bug report.



I'm experiencing pretty much the same problem.

Looking at the code (marks_file.py) I don't see why the order would  
matter (even though it would be nicer if the order was consistent).  I  
actually changed this so that it's always sorted, just to help me debug.

Now, I'm getting these errors back from git:

fatal: mark :96985 not declared
fast-import: dumping crash report to .git/fast_import_crash_74262
bzr: broken pipe

I couldn't reproduce this with a simple repository.

However, if one inspects the output of bzr fast-export, one finds  
stuff like this:

commit refs/heads/master
mark :96984
committer <dann> 1237847130 +0000
data 205
....
from :96985
M 644 inline lisp/ChangeLog
data 741178
--- ... --- ... --- ... ---
commit refs/heads/master
mark :96985
committer <jhd> 1237849747 +0000
...
from :96984
M 644 inline src/gtkutil.c
data 135796

I'm not sure about the structure of these files, but my educated guess  
would be that this is a circular reference.
Strange.  One would think that this should never happen.

That said, I get the same errors in other cases as well without  
circular reference.

As an experiment, I deleted the last 1000 or so revisions from the bzr  
marks file, so that they would be output again.
A couple of minutes and 430MB in output later, I imported this on the  
git side, which, after a few seconds, came back with this:
---------------------------------------------------------------------
Alloc'd objects:     105000
Total objects:            5 (      5063 duplicates                  )
       blobs  :            0 (      1549 duplicates          0 deltas)
       trees  :            2 (      2505 duplicates          0 deltas)
       commits:            3 (      1009 duplicates          0 deltas)
       tags   :            0 (         0 duplicates          0 deltas)
Total branches:           1 (         1 loads     )
       marks:        1048576 (     97012 unique    )
       atoms:           1937
Memory total:          5380 KiB
        pools:          2098 KiB
      objects:          3281 KiB
---------------------------------------------------------------------
pack_report: getpagesize()            =       4096
pack_report: core.packedGitWindowSize =   33554432
pack_report: core.packedGitLimit      =  268435456
pack_report: pack_used_ctr            =     104764
pack_report: pack_mmap_calls          =          6
pack_report: pack_open_windows        =          4 /          4
pack_report: pack_mapped              =  100666262 /  100666262
---------------------------------------------------------------------

So, it seems like there were 3 commits that were missing from previous  
transfers.  The final lines of the marks files for bzr and git seem  
coherent.  Hard to identify the culprit.

Further changes resulted in good conversion so far.  Deleting a good  
number of the most recent marks entries seems to be the right thing to  
recover.

A "bzr uncommit" seemed not to make its way to the git side.  No  
good.   I wonder if that is going to create a lasting inconsistency. I  
did get this subsequently:
warning: Not updating refs/heads/master (new tip  
6c81ccc916026d020eadeb0ad6e5b12c18aeccd3 does not contain  
3222cfee5bc00412b6f5e52a420f93564f586ee9)
This "not contained" revision resolved to the uncommitted one - that  
makes sense.
But what consequences does the warning have...?




[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2193 bytes --]