[PATCH v2 00/18] remote-bzr: massive changes

[PATCH v2 00/18] remote-bzr: massive changes

[Felipe Contreras]

Hi,

This is the same as last series, except that I dropped a couple of conflicting
patches, and added one test.


After being contacted by the emacs developers and others who are stuck with
Bazaar, which at this point seems to be utterly abandoned, I realized the
current implementation is too crude.

Bazaar branches are very simplistic, and our support for them is the same; we
need to create one remote per branch. This works nicely if you work on small
projects with few branches, but doesn't scale.

Big projects like emacs take a lot of space, and creating a remote per branch
is unrealistic, because each remote will have the whole Bazaar repository
copied, wasting space, and time each time a remote (for a bzr branch) is set
up.

Moreoever, a developer needs to constantly reset the master branch to the
commit he/she wants to push before pushing, since the transport-helper
infraestructure doesn't support pushing with refspecs (xwidget:master).

That is of course, if pushing actually worked (which in many cases doesn't).

In short, our support for real-world projects suck.

These patches fix all the issues I encountrered.

 1) First of all, there are several improvements for pushing. Before, we failed
    when trying to push a merge, now, even tricky merges work.

 2) Secondly, bzr branches are tied to a transport, so they time out if not
    used for a period of time, and importing/exporting huge chunks of a
    repository do take some time. So now they are only opened when they are
    about to be used.

 3) Then the big one; now bzr repositories are supported. They are very simple:
    basically an object store with no notion of branches, so to find the
    branches we need to traverse a directory (sometimes) remote, to find them.
    This is how Bazaar does it, ableit very slowly. Naturally, a lot of code
    had to be changed to support more than one branch.

 4) In addition, now remotes share all the objects, so adding a new remote
    doesn't imply fetchng a bunch of duplicated objects. They are just re-used
    automagically.

 5) Since the bzr objects are shared now, it only makes sense to share the
    remote-bzr marks, so we don't have to fast-import them again.

 6) The code was also reorganized to keep referenced as few objects as
    possible, since Bazaar seems to be need *a ton* of memory for them.

Finally, after all these changes I was finally able to clone the whole emacs
repository, all 130685 commits, and 56 branches without running out of memory
in my modest laptop.

Since the purpose of remote-bzr is to actually be usable for the poor souls
stucks in DSCMs that are not git, these changes are a must. I propose they be
merged for the next major version of git (v1.8.3) if no issues are found. They
changes pass all the tests, and work on various repositories I've tried. I'll
ask the emacs developers to give them a try, and let's see how it goes.

Felipe Contreras (18):
  remote-bzr: cleanup CustomTree
  remote-bzr: delay blob fetching until the very end
  remote-bzr: fix order of locking in CustomTree
  remote-bzr: always try to update the worktree
  remote-bzr: add support to push merges
  remote-bzr: fixes for branch diverge
  remote-bzr: fix partially pushed merge
  remote-bzr: use branch variable when appropriate
  remote-bzr: add support for bzr repos
  remote-bzr: fix branch names
  remote-bzr: add support for shared repo
  remote-bzr: improve author sanitazion
  remote-bzr: add custom method to find branches
  remote-bzr: add option to specify branches
  remote-bzr: improve progress reporting
  remote-bzr: iterate revisions properly
  remote-bzr: delay peer branch usage
  remote-bzr: access branches only when needed

 contrib/remote-helpers/git-remote-bzr | 305 ++++++++++++++++++++++++----------
 contrib/remote-helpers/test-bzr.sh    |  72 ++++++++
 2 files changed, 293 insertions(+), 84 deletions(-)

-- 
1.8.3.rc0.399.gc96a135

[PATCH v2 01/18] remote-bzr: cleanup CustomTree

[Felipe Contreras]

This code was not used at all.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index c19ed0e..7452a57 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -385,9 +385,6 @@ class CustomTree():
     def __init__(self, repo, revid, parents, files):
         global files_cache
 
-        self.repo = repo
-        self.revid = revid
-        self.parents = parents
         self.updates = {}
 
         def copy_tree(revid):
@@ -435,7 +432,7 @@ class CustomTree():
             if basename == '':
                 return None
             fid = bzrlib.generate_ids.gen_file_id(path)
-            d = add_entry(fid, dirname, 'directory')
+            add_entry(fid, dirname, 'directory')
             return fid
 
         def add_entry(fid, path, kind, mode = None):
@@ -458,7 +455,6 @@ class CustomTree():
                     (None, executable))
             self.files[path] = change[0]
             changes.append(change)
-            return change
 
         def update_entry(fid, path, kind, mode = None):
             dirname, basename = os.path.split(path)
@@ -480,7 +476,6 @@ class CustomTree():
                     (None, executable))
             self.files[path] = change[0]
             changes.append(change)
-            return change
 
         def remove_entry(fid, path, kind):
             dirname, basename = os.path.split(path)
@@ -495,7 +490,6 @@ class CustomTree():
                     (None, None))
             del self.files[path]
             changes.append(change)
-            return change
 
         for fid, f in self.updates.iteritems():
             path = f['path']
-- 
1.8.3.rc0.399.gc96a135

[PATCH v2 02/18] remote-bzr: delay blob fetching until the very end

[Felipe Contreras]

Might be more efficient, but the real reason to use the marks will be
revealed in upcoming patches.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 7452a57..91b5cda 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -506,10 +506,12 @@ class CustomTree():
         return changes
 
     def get_file_with_stat(self, file_id, path=None):
-        return (StringIO.StringIO(self.updates[file_id]['data']), None)
+        mark = self.updates[file_id]['mark']
+        return (StringIO.StringIO(blob_marks[mark]), None)
 
     def get_symlink_target(self, file_id):
-        return self.updates[file_id]['data']
+        mark = self.updates[file_id]['mark']
+        return blob_marks[mark]
 
 def c_style_unescape(string):
     if string[0] == string[-1] == '"':
@@ -553,7 +555,7 @@ def parse_commit(parser):
         if parser.check('M'):
             t, m, mark_ref, path = line.split(' ', 3)
             mark = int(mark_ref[1:])
-            f = { 'mode' : m, 'data' : blob_marks[mark] }
+            f = { 'mode' : m, 'mark' : mark }
         elif parser.check('D'):
             t, path = line.split(' ')
             f = { 'deleted' : True }
-- 
1.8.3.rc0.399.gc96a135

[PATCH v2 03/18] remote-bzr: fix order of locking in CustomTree

[Felipe Contreras]

It doesn't seem to make any difference, but revision_tree() requires a
lock.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 91b5cda..b428244 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -389,8 +389,8 @@ class CustomTree():
 
         def copy_tree(revid):
             files = files_cache[revid] = {}
-            tree = repo.repository.revision_tree(revid)
             repo.lock_read()
+            tree = repo.repository.revision_tree(revid)
             try:
                 for path, entry in tree.iter_entries_by_dir():
                     files[path] = entry.file_id
-- 
1.8.3.rc0.399.gc96a135

[PATCH v2 04/18] remote-bzr: always try to update the worktree

[Felipe Contreras]

And fail properly when we can't.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index b428244..eb91d28 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -641,9 +641,13 @@ def do_export(parser):
                 except bzrlib.errors.DivergedBranches:
                     print "error %s non-fast forward" % ref
                     continue
-            else:
+
+            try:
                 wt = repo.bzrdir.open_workingtree()
                 wt.update()
+            except bzrlib.errors.NoWorkingTree:
+                pass
+
         print "ok %s" % ref
 
     print
-- 
1.8.3.rc0.399.gc96a135

[PATCH v2 05/18] remote-bzr: add support to push merges

[Felipe Contreras]

In order to do that, we need to store the marks of every file, so that
they can be fetched when needed. Unfortunately we can't tell bazaar that
nothing changed, we need to send the data so that it can figure it out
by itself.

And since it will be requesting a bunch of information by the file_id,
it's better to have a helper dict (rev_files), so that we can fetch it
quickly.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 29 ++++++++++++++++++++--------
 contrib/remote-helpers/test-bzr.sh    | 36 +++++++++++++++++++++++++++++++++++
 2 files changed, 57 insertions(+), 8 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index eb91d28..6a7f836 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -393,7 +393,7 @@ class CustomTree():
             tree = repo.repository.revision_tree(revid)
             try:
                 for path, entry in tree.iter_entries_by_dir():
-                    files[path] = entry.file_id
+                    files[path] = [entry.file_id, None]
             finally:
                 repo.unlock()
             return files
@@ -408,12 +408,18 @@ class CustomTree():
                 self.base_files = copy_tree(self.base_id)
 
         self.files = files_cache[revid] = self.base_files.copy()
+        self.rev_files = {}
+
+        for path, data in self.files.iteritems():
+            fid, mark = data
+            self.rev_files[fid] = [path, mark]
 
         for path, f in files.iteritems():
-            fid = self.files.get(path, None)
+            fid, mark = self.files.get(path, [None, None])
             if not fid:
                 fid = bzrlib.generate_ids.gen_file_id(path)
             f['path'] = path
+            self.rev_files[fid] = [path, mark]
             self.updates[fid] = f
 
     def last_revision(self):
@@ -423,10 +429,10 @@ class CustomTree():
         changes = []
 
         def get_parent(dirname, basename):
-            parent_fid = self.base_files.get(dirname, None)
+            parent_fid, mark = self.base_files.get(dirname, [None, None])
             if parent_fid:
                 return parent_fid
-            parent_fid = self.files.get(dirname, None)
+            parent_fid, mark = self.files.get(dirname, [None, None])
             if parent_fid:
                 return parent_fid
             if basename == '':
@@ -453,7 +459,7 @@ class CustomTree():
                     (None, basename),
                     (None, kind),
                     (None, executable))
-            self.files[path] = change[0]
+            self.files[path] = [change[0], None]
             changes.append(change)
 
         def update_entry(fid, path, kind, mode = None):
@@ -474,7 +480,7 @@ class CustomTree():
                     (None, basename),
                     (None, kind),
                     (None, executable))
-            self.files[path] = change[0]
+            self.files[path] = [change[0], None]
             changes.append(change)
 
         def remove_entry(fid, path, kind):
@@ -503,16 +509,23 @@ class CustomTree():
             else:
                 add_entry(fid, path, 'file', f['mode'])
 
+            self.files[path][1] = f['mark']
+            self.rev_files[fid][1] = f['mark']
+
         return changes
 
     def get_file_with_stat(self, file_id, path=None):
-        mark = self.updates[file_id]['mark']
+        path, mark = self.rev_files[file_id]
         return (StringIO.StringIO(blob_marks[mark]), None)
 
     def get_symlink_target(self, file_id):
-        mark = self.updates[file_id]['mark']
+        path, mark = self.rev_files[file_id]
         return blob_marks[mark]
 
+    def id2path(self, file_id):
+        path, mark = self.rev_files[file_id]
+        return path
+
 def c_style_unescape(string):
     if string[0] == string[-1] == '"':
         return string.decode('string-escape')[1:-1]
diff --git a/contrib/remote-helpers/test-bzr.sh b/contrib/remote-helpers/test-bzr.sh
index 34666e1..eca3476 100755
--- a/contrib/remote-helpers/test-bzr.sh
+++ b/contrib/remote-helpers/test-bzr.sh
@@ -228,4 +228,40 @@ test_expect_success 'push utf-8 filenames' '
   test_cmp expected actual
 '
 
+test_expect_success 'pushing a merge' '
+  mkdir -p tmp && cd tmp &&
+  test_when_finished "cd .. && rm -rf tmp" &&
+
+  (
+  bzr init bzrrepo &&
+  cd bzrrepo &&
+  echo one > content &&
+  bzr add content &&
+  bzr commit -m one
+  ) &&
+
+  git clone "bzr::$PWD/bzrrepo" gitrepo &&
+
+  (
+  cd bzrrepo &&
+  echo two > content &&
+  bzr commit -m two
+  ) &&
+
+  (
+  cd gitrepo &&
+  echo three > content &&
+  git commit -a -m three &&
+  git fetch &&
+  git merge origin/master || true &&
+  echo three > content &&
+  git commit -a --no-edit &&
+  git push
+  ) &&
+
+  echo three > expected &&
+  cat bzrrepo/content > actual &&
+  test_cmp expected actual
+'
+
 test_done
-- 
1.8.3.rc0.399.gc96a135

[PATCH v2 06/18] remote-bzr: fixes for branch diverge

[Felipe Contreras]

If the branches diverge we want to reset the pointer to where the remote
actually is. Since we can access remote branches just as easily as local
ones, let's do so.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 6a7f836..bf254a0 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -354,6 +354,7 @@ def do_import(parser):
     if os.path.exists(path):
         print "feature import-marks=%s" % path
     print "feature export-marks=%s" % path
+    print "feature force"
     sys.stdout.flush()
 
     while parser.check('import'):
@@ -716,7 +717,12 @@ def get_repo(url, alias):
             # pull
             d = bzrlib.bzrdir.BzrDir.open(clone_path)
             branch = d.open_branch()
-            result = branch.pull(remote_branch, [], None, False)
+            try:
+                result = branch.pull(remote_branch, [], None, False)
+            except bzrlib.errors.DivergedBranches:
+                # use remote branch for now
+                peer = None
+                return remote_branch
         else:
             # clone
             d = origin.sprout(clone_path, None,
-- 
1.8.3.rc0.399.gc96a135

[PATCH v2 07/18] remote-bzr: fix partially pushed merge

[Felipe Contreras]

If part of the merge was already pushed, we don't have the blob_marks
available, however, the commits are already stored in bazaar, so we can
use the revision_tree to fetch the contents.

We want to do this only when there's no other option.

There's no easy way to test this.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index bf254a0..fdead31 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -387,6 +387,7 @@ class CustomTree():
         global files_cache
 
         self.updates = {}
+        self.branch = repo
 
         def copy_tree(revid):
             files = files_cache[revid] = {}
@@ -515,13 +516,21 @@ class CustomTree():
 
         return changes
 
-    def get_file_with_stat(self, file_id, path=None):
+    def get_content(self, file_id):
         path, mark = self.rev_files[file_id]
-        return (StringIO.StringIO(blob_marks[mark]), None)
+        if mark:
+            return blob_marks[mark]
+
+        # last resort
+        tree = self.branch.repository.revision_tree(self.base_id)
+        return tree.get_file_text(file_id)
+
+    def get_file_with_stat(self, file_id, path=None):
+        content = self.get_content(file_id)
+        return (StringIO.StringIO(content), None)
 
     def get_symlink_target(self, file_id):
-        path, mark = self.rev_files[file_id]
-        return blob_marks[mark]
+        return self.get_content(file_id)
 
     def id2path(self, file_id):
         path, mark = self.rev_files[file_id]
-- 
1.8.3.rc0.399.gc96a135

[PATCH v2 08/18] remote-bzr: use branch variable when appropriate

[Felipe Contreras]

There should be no functional changes. Basically we want to reserve the
'repo' variable.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 35 ++++++++++++++++++-----------------
 1 file changed, 18 insertions(+), 17 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index fdead31..0c67236 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -257,7 +257,8 @@ def export_branch(branch, name):
     tip = marks.get_tip(name)
 
     repo = branch.repository
-    repo.lock_read()
+
+    branch.lock_read()
     revs = branch.iter_merge_sorted_revisions(None, tip, 'exclude', 'forward')
     count = 0
 
@@ -325,7 +326,7 @@ def export_branch(branch, name):
             print "progress revision %s (%d/%d)" % (revid, count, len(revs))
             print "#############################################################"
 
-    repo.unlock()
+    branch.unlock()
 
     revid = branch.last_revision()
 
@@ -383,21 +384,21 @@ def parse_blob(parser):
 
 class CustomTree():
 
-    def __init__(self, repo, revid, parents, files):
+    def __init__(self, branch, revid, parents, files):
         global files_cache
 
         self.updates = {}
-        self.branch = repo
+        self.branch = branch
 
         def copy_tree(revid):
             files = files_cache[revid] = {}
-            repo.lock_read()
-            tree = repo.repository.revision_tree(revid)
+            branch.lock_read()
+            tree = branch.repository.revision_tree(revid)
             try:
                 for path, entry in tree.iter_entries_by_dir():
                     files[path] = [entry.file_id, None]
             finally:
-                repo.unlock()
+                branch.unlock()
             return files
 
         if len(parents) == 0:
@@ -587,20 +588,20 @@ def parse_commit(parser):
         path = c_style_unescape(path).decode('utf-8')
         files[path] = f
 
-    repo = parser.repo
+    branch = parser.repo
 
     committer, date, tz = committer
     parents = [str(mark_to_rev(p)) for p in parents]
     revid = bzrlib.generate_ids.gen_revision_id(committer, date)
     props = {}
-    props['branch-nick'] = repo.nick
+    props['branch-nick'] = branch.nick
 
-    mtree = CustomTree(repo, revid, parents, files)
+    mtree = CustomTree(branch, revid, parents, files)
     changes = mtree.iter_changes()
 
-    repo.lock_write()
+    branch.lock_write()
     try:
-        builder = repo.get_commit_builder(parents, None, date, tz, committer, props, revid)
+        builder = branch.get_commit_builder(parents, None, date, tz, committer, props, revid)
         try:
             list(builder.record_iter_changes(mtree, mtree.last_revision(), changes))
             builder.finish_inventory()
@@ -609,7 +610,7 @@ def parse_commit(parser):
             builder.abort()
             raise
     finally:
-        repo.unlock()
+        branch.unlock()
 
     parsed_refs[ref] = revid
     marks.new_mark(revid, commit_mark)
@@ -653,20 +654,20 @@ def do_export(parser):
         else:
             die('unhandled export command: %s' % line)
 
-    repo = parser.repo
+    branch = parser.repo
 
     for ref, revid in parsed_refs.iteritems():
         if ref == 'refs/heads/master':
-            repo.generate_revision_history(revid, marks.get_tip('master'))
+            branch.generate_revision_history(revid, marks.get_tip('master'))
             if peer:
                 try:
-                    repo.push(peer, stop_revision=revid)
+                    branch.push(peer, stop_revision=revid)
                 except bzrlib.errors.DivergedBranches:
                     print "error %s non-fast forward" % ref
                     continue
 
             try:
-                wt = repo.bzrdir.open_workingtree()
+                wt = branch.bzrdir.open_workingtree()
                 wt.update()
             except bzrlib.errors.NoWorkingTree:
                 pass
-- 
1.8.3.rc0.399.gc96a135

[PATCH v2 09/18] remote-bzr: add support for bzr repos

[Felipe Contreras]

In bazaar, a repository can contain multiple branches, and previously we
were supporting only one branch at a time. Now we fetch them all.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 158 ++++++++++++++++++++++------------
 contrib/remote-helpers/test-bzr.sh    |  36 ++++++++
 2 files changed, 138 insertions(+), 56 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 0c67236..3b33061 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -27,6 +27,7 @@ import bzrlib.generate_ids
 import bzrlib.transport
 import bzrlib.errors
 import bzrlib.ui
+import bzrlib.urlutils
 
 import sys
 import os
@@ -250,12 +251,13 @@ def export_files(tree, files):
 
     return final
 
-def export_branch(branch, name):
+def export_branch(repo, name):
     global prefix
 
     ref = '%s/heads/%s' % (prefix, name)
     tip = marks.get_tip(name)
 
+    branch = branches[name]
     repo = branch.repository
 
     branch.lock_read()
@@ -323,7 +325,7 @@ def export_branch(branch, name):
 
         count += 1
         if (count % 100 == 0):
-            print "progress revision %s (%d/%d)" % (revid, count, len(revs))
+            print "progress revision %s '%s' (%d/%d)" % (revid, name, count, len(revs))
             print "#############################################################"
 
     branch.unlock()
@@ -348,7 +350,7 @@ def export_tag(repo, name):
 def do_import(parser):
     global dirname
 
-    branch = parser.repo
+    repo = parser.repo
     path = os.path.join(dirname, 'marks-git')
 
     print "feature done"
@@ -362,10 +364,10 @@ def do_import(parser):
         ref = parser[1]
         if ref.startswith('refs/heads/'):
             name = ref[len('refs/heads/'):]
-            export_branch(branch, name)
+            export_branch(repo, name)
         if ref.startswith('refs/tags/'):
             name = ref[len('refs/tags/'):]
-            export_tag(branch, name)
+            export_tag(repo, name)
         parser.next()
 
     print 'done'
@@ -551,8 +553,11 @@ def parse_commit(parser):
     ref = parser[1]
     parser.next()
 
-    if ref != 'refs/heads/master':
-        die("bzr doesn't support multiple branches; use 'master'")
+    if ref.startswith('refs/heads/'):
+        name = ref[len('refs/heads/'):]
+        branch = branches[name]
+    else:
+        die('unknown ref')
 
     commit_mark = parser.get_mark()
     parser.next()
@@ -588,8 +593,6 @@ def parse_commit(parser):
         path = c_style_unescape(path).decode('utf-8')
         files[path] = f
 
-    branch = parser.repo
-
     committer, date, tz = committer
     parents = [str(mark_to_rev(p)) for p in parents]
     revid = bzrlib.generate_ids.gen_revision_id(committer, date)
@@ -621,9 +624,6 @@ def parse_reset(parser):
     ref = parser[1]
     parser.next()
 
-    if ref != 'refs/heads/master':
-        die("bzr doesn't support multiple branches; use 'master'")
-
     # ugh
     if parser.check('commit'):
         parse_commit(parser)
@@ -636,7 +636,7 @@ def parse_reset(parser):
     parsed_refs[ref] = mark_to_rev(from_mark)
 
 def do_export(parser):
-    global parsed_refs, dirname, peer
+    global parsed_refs, dirname
 
     parser.next()
 
@@ -654,23 +654,24 @@ def do_export(parser):
         else:
             die('unhandled export command: %s' % line)
 
-    branch = parser.repo
-
     for ref, revid in parsed_refs.iteritems():
-        if ref == 'refs/heads/master':
-            branch.generate_revision_history(revid, marks.get_tip('master'))
-            if peer:
-                try:
-                    branch.push(peer, stop_revision=revid)
-                except bzrlib.errors.DivergedBranches:
-                    print "error %s non-fast forward" % ref
-                    continue
+        name = ref[len('refs/heads/'):]
+        branch = branches[name]
+        branch.generate_revision_history(revid, marks.get_tip(name))
 
+        if name in peers:
+            peer = peers[name]
             try:
-                wt = branch.bzrdir.open_workingtree()
-                wt.update()
-            except bzrlib.errors.NoWorkingTree:
-                pass
+                peer.bzrdir.push_branch(branch, revision_id=revid)
+            except bzrlib.errors.DivergedBranches:
+                print "error %s non-fast forward" % ref
+                continue
+
+        try:
+            wt = branch.bzrdir.open_workingtree()
+            wt.update()
+        except bzrlib.errors.NoWorkingTree:
+            pass
 
         print "ok %s" % ref
 
@@ -697,9 +698,15 @@ def ref_is_valid(name):
 
 def do_list(parser):
     global tags
-    print "? refs/heads/%s" % 'master'
 
-    branch = parser.repo
+    master_branch = None
+
+    for name in branches:
+        if not master_branch:
+            master_branch = name
+        print "? refs/heads/%s" % name
+
+    branch = branches[master_branch]
     branch.lock_read()
     for tag, revid in branch.tags.get_tag_dict().items():
         try:
@@ -711,41 +718,77 @@ def do_list(parser):
         print "? refs/tags/%s" % tag
         tags[tag] = revid
     branch.unlock()
-    print "@refs/heads/%s HEAD" % 'master'
+
+    print "@refs/heads/%s HEAD" % master_branch
     print
 
+def get_remote_branch(origin, remote_branch, name):
+    global dirname, peers
+
+    branch_path = os.path.join(dirname, 'clone', name)
+    if os.path.exists(branch_path):
+        # pull
+        d = bzrlib.bzrdir.BzrDir.open(branch_path)
+        branch = d.open_branch()
+        try:
+            branch.pull(remote_branch, [], None, False)
+        except bzrlib.errors.DivergedBranches:
+            # use remote branch for now
+            return remote_branch
+    else:
+        # clone
+        d = origin.sprout(branch_path, None,
+                hardlink=True, create_tree_if_local=False,
+                force_new_repo=False,
+                source_branch=remote_branch)
+        branch = d.open_branch()
+
+    return branch
+
 def get_repo(url, alias):
-    global dirname, peer
+    global dirname, peer, branches
 
+    normal_url = bzrlib.urlutils.normalize_url(url)
     origin = bzrlib.bzrdir.BzrDir.open(url)
-    branch = origin.open_branch()
-
-    if not isinstance(origin.transport, bzrlib.transport.local.LocalTransport):
-        clone_path = os.path.join(dirname, 'clone')
-        remote_branch = branch
-        if os.path.exists(clone_path):
-            # pull
-            d = bzrlib.bzrdir.BzrDir.open(clone_path)
-            branch = d.open_branch()
-            try:
-                result = branch.pull(remote_branch, [], None, False)
-            except bzrlib.errors.DivergedBranches:
-                # use remote branch for now
-                peer = None
-                return remote_branch
+    is_local = isinstance(origin.transport, bzrlib.transport.local.LocalTransport)
+
+    clone_path = os.path.join(dirname, 'clone')
+
+    try:
+        repo = origin.open_repository()
+    except bzrlib.errors.NoRepositoryPresent:
+        # branch
+
+        name = 'master'
+        branch = origin.open_branch()
+
+        if not is_local:
+            if not os.path.exists(clone_path):
+                os.mkdir(clone_path)
+            peers[name] = branch
+            branches[name] = get_remote_branch(origin, branch, name)
         else:
-            # clone
-            d = origin.sprout(clone_path, None,
-                    hardlink=True, create_tree_if_local=False,
-                    source_branch=remote_branch)
-            branch = d.open_branch()
-            branch.bind(remote_branch)
-
-        peer = remote_branch
+            branches[name] = branch
+
+        return branch.repository
     else:
-        peer = None
+        # repository
 
-    return branch
+        if not is_local and not os.path.exists(clone_path):
+            clonedir = bzrlib.bzrdir.BzrDir.create(clone_path)
+
+        for branch in repo.find_branches():
+
+            name = repo.user_transport.relpath(branch.base)
+            name = name if name != '' else 'master'
+
+            if not is_local:
+                peers[name] = branch
+                branches[name] = get_remote_branch(origin, branch, name)
+            else:
+                branches[name] = branch
+
+        return repo
 
 def fix_path(alias, orig_url):
     url = urlparse.urlparse(orig_url, 'file')
@@ -762,6 +805,7 @@ def main(args):
     global parsed_refs
     global files_cache
     global is_tmp
+    global branches, peers
 
     alias = args[1]
     url = args[2]
@@ -772,6 +816,8 @@ def main(args):
     parsed_refs = {}
     files_cache = {}
     marks = None
+    branches = {}
+    peers = {}
 
     if alias[5:] == url:
         is_tmp = True
diff --git a/contrib/remote-helpers/test-bzr.sh b/contrib/remote-helpers/test-bzr.sh
index eca3476..cec55f1 100755
--- a/contrib/remote-helpers/test-bzr.sh
+++ b/contrib/remote-helpers/test-bzr.sh
@@ -264,4 +264,40 @@ test_expect_success 'pushing a merge' '
   test_cmp expected actual
 '
 
+cat > expected <<EOF
+origin/HEAD
+origin/branch
+origin/trunk
+EOF
+
+test_expect_success 'proper bzr repo' '
+  mkdir -p tmp && cd tmp &&
+  test_when_finished "cd .. && rm -rf tmp" &&
+
+  bzr init-repo bzrrepo &&
+
+  bzr init bzrrepo/trunk &&
+  (
+  cd bzrrepo/trunk &&
+  echo one >> content &&
+  bzr add content &&
+  bzr commit -m one
+  ) &&
+
+  bzr branch bzrrepo/trunk bzrrepo/branch &&
+  (
+  cd bzrrepo/branch &&
+  echo two >> content &&
+  bzr commit -m one
+  ) &&
+
+  git clone "bzr::$PWD/bzrrepo" gitrepo &&
+  (
+  cd gitrepo &&
+  git for-each-ref --format "%(refname:short)" refs/remotes/origin > ../actual
+  ) &&
+
+  test_cmp ../expected actual
+'
+
 test_done
-- 
1.8.3.rc0.399.gc96a135

[PATCH v2 10/18] remote-bzr: fix branch names

[Felipe Contreras]

When branches have '/' in their name (aka. sub-branches), bazaar seems
to choke while creating the new directory.

Also, git cannot have both 'foo' and 'foo/bar'.

So let's replace slashes with a plus sign.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 1 +
 1 file changed, 1 insertion(+)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 3b33061..38cec07 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -781,6 +781,7 @@ def get_repo(url, alias):
 
             name = repo.user_transport.relpath(branch.base)
             name = name if name != '' else 'master'
+            name = name.replace('/', '+')
 
             if not is_local:
                 peers[name] = branch
-- 
1.8.3.rc0.399.gc96a135

[PATCH v2 11/18] remote-bzr: add support for shared repo

[Felipe Contreras]

This way all the remotes share the same data, so adding multiple
remotes, or renaming them doesn't create extra overhead.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 38cec07..9fe830e 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -752,7 +752,20 @@ def get_repo(url, alias):
     origin = bzrlib.bzrdir.BzrDir.open(url)
     is_local = isinstance(origin.transport, bzrlib.transport.local.LocalTransport)
 
-    clone_path = os.path.join(dirname, 'clone')
+    shared_path = os.path.join(gitdir, 'bzr')
+    try:
+        shared_dir = bzrlib.bzrdir.BzrDir.open(shared_path)
+    except bzrlib.errors.NotBranchError:
+        shared_dir = bzrlib.bzrdir.BzrDir.create(shared_path)
+    try:
+        shared_repo = shared_dir.open_repository()
+    except bzrlib.errors.NoRepositoryPresent:
+        shared_repo = shared_dir.create_repository(shared=True)
+
+    if not is_local:
+        clone_path = os.path.join(dirname, 'clone')
+        if not os.path.exists(clone_path):
+            os.mkdir(clone_path)
 
     try:
         repo = origin.open_repository()
@@ -763,8 +776,6 @@ def get_repo(url, alias):
         branch = origin.open_branch()
 
         if not is_local:
-            if not os.path.exists(clone_path):
-                os.mkdir(clone_path)
             peers[name] = branch
             branches[name] = get_remote_branch(origin, branch, name)
         else:
@@ -774,9 +785,6 @@ def get_repo(url, alias):
     else:
         # repository
 
-        if not is_local and not os.path.exists(clone_path):
-            clonedir = bzrlib.bzrdir.BzrDir.create(clone_path)
-
         for branch in repo.find_branches():
 
             name = repo.user_transport.relpath(branch.base)
@@ -800,7 +808,7 @@ def fix_path(alias, orig_url):
     subprocess.call(cmd)
 
 def main(args):
-    global marks, prefix, dirname
+    global marks, prefix, gitdir, dirname
     global tags, filenodes
     global blob_marks
     global parsed_refs
-- 
1.8.3.rc0.399.gc96a135

[PATCH v2 12/18] remote-bzr: improve author sanitazion

[Felipe Contreras]

So that we don't end up with '<None>', and also synchronize it with the
one from remote-hg.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 9fe830e..af46016 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -38,6 +38,7 @@ import atexit, shutil, hashlib, urlparse, subprocess
 
 NAME_RE = re.compile('^([^<>]+)')
 AUTHOR_RE = re.compile('^([^<>]+?)? ?<([^<>]*)>$')
+EMAIL_RE = re.compile('^([^<>]+[^ \\\t<>])?\\b(?:[ \\t<>]*?)\\b([^ \\t<>]+@[^ \\t<>]+)')
 RAW_AUTHOR_RE = re.compile('^(\w+) (.+)? <(.*)> (\d+) ([+-]\d+)')
 
 def die(msg, *args):
@@ -175,9 +176,19 @@ def fixup_user(user):
         name = m.group(1)
         mail = m.group(2).strip()
     else:
-        m = NAME_RE.match(user)
+        m = EMAIL_RE.match(user)
         if m:
-            name = m.group(1).strip()
+            name = m.group(1)
+            mail = m.group(2)
+        else:
+            m = NAME_RE.match(user)
+            if m:
+                name = m.group(1).strip()
+
+    if not name:
+        name = 'unknown'
+    if not mail:
+        mail = 'Unknown'
 
     return '%s <%s>' % (name, mail)
 
-- 
1.8.3.rc0.399.gc96a135

[PATCH v2 13/18] remote-bzr: add custom method to find branches

[Felipe Contreras]

The official method is incredibly inefficient and slow.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 25 ++++++++++++++++++++-----
 1 file changed, 20 insertions(+), 5 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index af46016..11f2415 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -756,6 +756,25 @@ def get_remote_branch(origin, remote_branch, name):
 
     return branch
 
+def find_branches(repo):
+    transport = repo.user_transport
+
+    for fn in transport.iter_files_recursive():
+        if not fn.endswith('.bzr/branch-format'):
+            continue
+
+        name = subdir = fn[:-len('/.bzr/branch-format')]
+        name = name if name != '' else 'master'
+        name = name.replace('/', '+')
+
+        try:
+            cur = transport.clone(subdir)
+            branch = bzrlib.branch.Branch.open_from_transport(cur)
+        except bzrlib.errors.NotBranchError:
+            continue
+        else:
+            yield name, branch
+
 def get_repo(url, alias):
     global dirname, peer, branches
 
@@ -796,11 +815,7 @@ def get_repo(url, alias):
     else:
         # repository
 
-        for branch in repo.find_branches():
-
-            name = repo.user_transport.relpath(branch.base)
-            name = name if name != '' else 'master'
-            name = name.replace('/', '+')
+        for name, branch in find_branches(repo):
 
             if not is_local:
                 peers[name] = branch
-- 
1.8.3.rc0.399.gc96a135

[PATCH v2 14/18] remote-bzr: add option to specify branches

[Felipe Contreras]

We might not want all the branches. And branch handling in bazaar is
rather tricky, so it's safer to simply specify them.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 11f2415..d284afc 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -13,6 +13,9 @@
 # or
 # % git clone bzr::lp:myrepo
 #
+# If you want to specify which branches you want track (per repo):
+# git config remote-bzr.branches 'trunk, devel, test'
+#
 
 import sys
 
@@ -51,6 +54,12 @@ def warn(msg, *args):
 def gittz(tz):
     return '%+03d%02d' % (tz / 3600, tz % 3600 / 60)
 
+def get_config(config):
+    cmd = ['git', 'config', '--get', config]
+    process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
+    output, _ = process.communicate()
+    return output
+
 class Marks:
 
     def __init__(self, path):
@@ -756,7 +765,7 @@ def get_remote_branch(origin, remote_branch, name):
 
     return branch
 
-def find_branches(repo):
+def find_branches(repo, wanted):
     transport = repo.user_transport
 
     for fn in transport.iter_files_recursive():
@@ -767,6 +776,9 @@ def find_branches(repo):
         name = name if name != '' else 'master'
         name = name.replace('/', '+')
 
+        if wanted and not name in wanted:
+            continue
+
         try:
             cur = transport.clone(subdir)
             branch = bzrlib.branch.Branch.open_from_transport(cur)
@@ -815,7 +827,11 @@ def get_repo(url, alias):
     else:
         # repository
 
-        for name, branch in find_branches(repo):
+        wanted = get_config('remote-bzr.branches').rstrip().split(', ')
+        # stupid python
+        wanted = [e for e in wanted if e]
+
+        for name, branch in find_branches(repo, wanted):
 
             if not is_local:
                 peers[name] = branch
-- 
1.8.3.rc0.399.gc96a135

[PATCH v2 15/18] remote-bzr: improve progress reporting

[Felipe Contreras]

No need to manually count the revisions, and also, this would help to
iterate more properly.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index d284afc..2e21008 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -282,13 +282,16 @@ def export_branch(repo, name):
 
     branch.lock_read()
     revs = branch.iter_merge_sorted_revisions(None, tip, 'exclude', 'forward')
-    count = 0
+    tip_revno = branch.revision_id_to_revno(tip)
+    last_revno, _ = branch.last_revision_info()
+    total = last_revno - tip_revno
 
-    revs = [revid for revid, _, _, _ in revs if not marks.is_marked(revid)]
+    revs = [revid, seq for revid, _, seq, _ in revs if not marks.is_marked(revid)]
 
-    for revid in revs:
+    for revid, seq in revs:
 
         rev = repo.get_revision(revid)
+        revno = seq[0]
 
         parents = rev.parent_ids
         time = rev.timestamp
@@ -343,10 +346,13 @@ def export_branch(repo, name):
             print "M %s :%u %s" % f
         print
 
-        count += 1
-        if (count % 100 == 0):
-            print "progress revision %s '%s' (%d/%d)" % (revid, name, count, len(revs))
-            print "#############################################################"
+        if len(seq) > 1:
+            # let's skip branch revisions from the progress report
+            continue
+
+        progress = (revno - tip_revno)
+        if (progress % 100 == 0):
+            print "progress revision %d '%s' (%d/%d)" % (revno, name, progress, total)
 
     branch.unlock()
 
-- 
1.8.3.rc0.399.gc96a135

[PATCH v2 16/18] remote-bzr: iterate revisions properly

[Felipe Contreras]

This way we don't need to store the list of all the revisions, which
doesn't seem to be very memory efficient with bazaar's design, for
whatever reason.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 2e21008..66e3861 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -286,9 +286,10 @@ def export_branch(repo, name):
     last_revno, _ = branch.last_revision_info()
     total = last_revno - tip_revno
 
-    revs = [revid, seq for revid, _, seq, _ in revs if not marks.is_marked(revid)]
+    for revid, _, seq, _ in revs:
 
-    for revid, seq in revs:
+        if marks.is_marked(revid):
+            continue
 
         rev = repo.get_revision(revid)
         revno = seq[0]
-- 
1.8.3.rc0.399.gc96a135

[PATCH v2 17/18] remote-bzr: delay peer branch usage

[Felipe Contreras]

So it doesn't time out.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 66e3861..b2d67f7 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -687,7 +687,7 @@ def do_export(parser):
         branch.generate_revision_history(revid, marks.get_tip(name))
 
         if name in peers:
-            peer = peers[name]
+            peer = bzrlib.branch.Branch.open(peers[name])
             try:
                 peer.bzrdir.push_branch(branch, revision_id=revid)
             except bzrlib.errors.DivergedBranches:
@@ -825,7 +825,7 @@ def get_repo(url, alias):
         branch = origin.open_branch()
 
         if not is_local:
-            peers[name] = branch
+            peers[name] = branch.base
             branches[name] = get_remote_branch(origin, branch, name)
         else:
             branches[name] = branch
@@ -841,7 +841,7 @@ def get_repo(url, alias):
         for name, branch in find_branches(repo, wanted):
 
             if not is_local:
-                peers[name] = branch
+                peers[name] = branch.base
                 branches[name] = get_remote_branch(origin, branch, name)
             else:
                 branches[name] = branch
-- 
1.8.3.rc0.399.gc96a135

[PATCH v2 18/18] remote-bzr: access branches only when needed

[Felipe Contreras]

Bazaar doesn't seem to be tested for multiple usage of branches, so
resources seem to be leaked all over. Let's try to minimize this by
accessing the Branch objects only when needed.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 28 ++++++++++++++++------------
 1 file changed, 16 insertions(+), 12 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index b2d67f7..161f831 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -277,7 +277,7 @@ def export_branch(repo, name):
     ref = '%s/heads/%s' % (prefix, name)
     tip = marks.get_tip(name)
 
-    branch = branches[name]
+    branch = bzrlib.branch.Branch.open(branches[name])
     repo = branch.repository
 
     branch.lock_read()
@@ -582,7 +582,7 @@ def parse_commit(parser):
 
     if ref.startswith('refs/heads/'):
         name = ref[len('refs/heads/'):]
-        branch = branches[name]
+        branch = bzrlib.branch.Branch.open(branches[name])
     else:
         die('unknown ref')
 
@@ -683,7 +683,7 @@ def do_export(parser):
 
     for ref, revid in parsed_refs.iteritems():
         name = ref[len('refs/heads/'):]
-        branch = branches[name]
+        branch = bzrlib.branch.Branch.open(branches[name])
         branch.generate_revision_history(revid, marks.get_tip(name))
 
         if name in peers:
@@ -733,7 +733,7 @@ def do_list(parser):
             master_branch = name
         print "? refs/heads/%s" % name
 
-    branch = branches[master_branch]
+    branch = bzrlib.branch.Branch.open(branches[master_branch])
     branch.lock_read()
     for tag, revid in branch.tags.get_tag_dict().items():
         try:
@@ -822,13 +822,15 @@ def get_repo(url, alias):
         # branch
 
         name = 'master'
-        branch = origin.open_branch()
+        remote_branch = origin.open_branch()
 
         if not is_local:
-            peers[name] = branch.base
-            branches[name] = get_remote_branch(origin, branch, name)
+            peers[name] = remote_branch.base
+            branch = get_remote_branch(origin, remote_branch, name)
         else:
-            branches[name] = branch
+            branch = remote_branch
+
+        branches[name] = branch.base
 
         return branch.repository
     else:
@@ -838,13 +840,15 @@ def get_repo(url, alias):
         # stupid python
         wanted = [e for e in wanted if e]
 
-        for name, branch in find_branches(repo, wanted):
+        for name, remote_branch in find_branches(repo, wanted):
 
             if not is_local:
-                peers[name] = branch.base
-                branches[name] = get_remote_branch(origin, branch, name)
+                peers[name] = remote_branch.base
+                branch = get_remote_branch(origin, remote_branch, name)
             else:
-                branches[name] = branch
+                branch = remote_branch
+
+            branches[name] = branch.base
 
         return repo
 
-- 
1.8.3.rc0.399.gc96a135

Re: [PATCH v2 00/18] remote-bzr: massive changes

[Junio C Hamano]

Felipe Contreras <felipe.contreras@gmail.com> writes:

> After being contacted by the emacs developers and others who are stuck with
> Bazaar, which at this point seems to be utterly abandoned, I realized the
> current implementation is too crude.
> ...
> That is of course, if pushing actually worked (which in many cases doesn't).
>
> In short, our support for real-world projects suck.
>
> These patches fix all the issues I encountrered.
> ...
> Finally, after all these changes I was finally able to clone the whole emacs
> repository, all 130685 commits, and 56 branches without running out of memory
> in my modest laptop.

Yay ;-)

I assume that the trees at a handful of key points (e.g. releases)
were verified to be identical with the original history and the
conversion result.

> Since the purpose of remote-bzr is to actually be usable for the
> poor souls stucks in DSCMs that are not git, these changes are a
> must. I propose they be merged for the next major version of git
> (v1.8.3) if no issues are found.  They changes pass all the tests,
> and work on various repositories I've tried.

Nice.

> I'll ask the emacs developers to give them a try, and let's see
> how it goes.

Yeah, that's the least we can do for both existing and future users.

Generally speaking, post -rc0 is too late for "if no issues are
found", simply because no existing user has enough time to find
corner case regressions in her work using the new software (I do
not expect a trivial bug that can be uncovered in a few weeks of use
would remain in a version that has successfully converted the Emacs
history; but real world users always have different needs than what
we anticipate).

I however am finding myself moderately receptive to this series.
That is primarily because this series touches only two files that
are totally isolated from the rest of the system.  Even if they did
not work at all, there is no risk for the remainder of Git.  Nobody
other than existing users of remote-bzr will even notice if we
merged this by the final.

For existing users of remote-bzr that we shipped in 1.8.2, the story
is a bit different, though.  If this series makes things worse in a
way your tests did not reveal, and if such a regression is not
reported and/or cannot be fixed by 1.8.3 final, that will mean a
real regression in the released version for them.

If that ever happens, that would be the time for us to regret the
hasty decision to merge remote-bzr in 1.8.2, justifying that with a
"There wasn't anything working for interoperating with bzr, and here
is one to do so; anything is better than nothing", and learn from
that mistake (it is not an option to say "the 1.8.2 users chose to
use contrib/ material that are clearly marked as sub-par quality
with their own risk".  If we did not ship it in 1.8.2, they did not
have to get burned with any regression and could have kept working
with bzr a bit longer.  "Anything" is not necessarily better than
"nothing").

Hopefully, such a regression will not have to happen (for one thing,
I would expect that the existing 1.8.2 remote-bzr user base would be
very small).  Also I somehow have a feeling that it is very unlikely
to happen, especially given your report:

 (1) the series converts Emacs history without barfing; and

 (2) you have some confidence in the conversion result after
     inspecting at least a handful of key release points and trees
     and metainformation match between the original and the
     converted history.

So let's go ahead and apply these directly on top of 'master', once
we hear from Emacs folks and they are happy with it.  I'll queue it
on 'pu' so that I do not have to go back to the list archive when it
happens.

Thanks.

Re: [PATCH v2 00/18] remote-bzr: massive changes

[Felipe Contreras]

On Wed, May 1, 2013 at 12:44 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Felipe Contreras <felipe.contreras@gmail.com> writes:
>
>> After being contacted by the emacs developers and others who are stuck with
>> Bazaar, which at this point seems to be utterly abandoned, I realized the
>> current implementation is too crude.
>> ...
>> That is of course, if pushing actually worked (which in many cases doesn't).
>>
>> In short, our support for real-world projects suck.
>>
>> These patches fix all the issues I encountrered.
>> ...
>> Finally, after all these changes I was finally able to clone the whole emacs
>> repository, all 130685 commits, and 56 branches without running out of memory
>> in my modest laptop.
>
> Yay ;-)
>
> I assume that the trees at a handful of key points (e.g. releases)
> were verified to be identical with the original history and the
> conversion result.

Not really. I don't think the users are that interested in the history
being identical at this point, merely that they can use it as a proxy
to interact with bazaar repositories.

People have found discrepancies, so I assume they have compared at
least the tip of the branches, and found them. This probably means
that the history is correct, since bazaar deals with changesets (git
is one of the few DSCMs that don't).

Also, there's further news on this, pushes seem to work correctly to
the emacs' repo[1].

>> I'll ask the emacs developers to give them a try, and let's see
>> how it goes.
>
> Yeah, that's the least we can do for both existing and future users.
>
> Generally speaking, post -rc0 is too late for "if no issues are
> found", simply because no existing user has enough time to find
> corner case regressions in her work using the new software (I do
> not expect a trivial bug that can be uncovered in a few weeks of use
> would remain in a version that has successfully converted the Emacs
> history; but real world users always have different needs than what
> we anticipate).
>
> I however am finding myself moderately receptive to this series.
> That is primarily because this series touches only two files that
> are totally isolated from the rest of the system.  Even if they did
> not work at all, there is no risk for the remainder of Git.  Nobody
> other than existing users of remote-bzr will even notice if we
> merged this by the final.
>
> For existing users of remote-bzr that we shipped in 1.8.2, the story
> is a bit different, though.  If this series makes things worse in a
> way your tests did not reveal, and if such a regression is not
> reported and/or cannot be fixed by 1.8.3 final, that will mean a
> real regression in the released version for them.
>
> If that ever happens, that would be the time for us to regret the
> hasty decision to merge remote-bzr in 1.8.2, justifying that with a
> "There wasn't anything working for interoperating with bzr, and here
> is one to do so; anything is better than nothing", and learn from
> that mistake (it is not an option to say "the 1.8.2 users chose to
> use contrib/ material that are clearly marked as sub-par quality
> with their own risk".  If we did not ship it in 1.8.2, they did not
> have to get burned with any regression and could have kept working
> with bzr a bit longer.  "Anything" is not necessarily better than
> "nothing").

Fortunately there seem to be at least some users that find what is in
1.8.2 working to some extent, not in all the repositories, and not all
the features, but at least something, which is much better than the
alternatives, even the best one has been blocked for years, even when
a solution is known[2].

> Hopefully, such a regression will not have to happen (for one thing,
> I would expect that the existing 1.8.2 remote-bzr user base would be
> very small).  Also I somehow have a feeling that it is very unlikely
> to happen, especially given your report:
>
>  (1) the series converts Emacs history without barfing; and
>
>  (2) you have some confidence in the conversion result after
>      inspecting at least a handful of key release points and trees
>      and metainformation match between the original and the
>      converted history.
>
> So let's go ahead and apply these directly on top of 'master', once
> we hear from Emacs folks and they are happy with it.  I'll queue it
> on 'pu' so that I do not have to go back to the list archive when it
> happens.

I already heard that everything seems to be working correctly, except
one feature, the biggest change, which I screwed up with a one-liner
commit. That's why I added a test. Anyway, I've fixed it in my github
branch and in this patch series, and I've told them to try the fix.

Let's see.

Cheers.

[1] http://bzr.savannah.gnu.org/lh/emacs/xwidget/revision/101292
[2] https://bugs.launchpad.net/bzr/+bug/541626

-- 
Felipe Contreras

Re: [PATCH v2 00/18] remote-bzr: massive changes

[Junio C Hamano]

Felipe Contreras <felipe.contreras@gmail.com> writes:

>> So let's go ahead and apply these directly on top of 'master', once
>> we hear from Emacs folks and they are happy with it.  I'll queue it
>> on 'pu' so that I do not have to go back to the list archive when it
>> happens.
>
> I already heard that everything seems to be working correctly, except
> one feature, the biggest change, which I screwed up with a one-liner
> commit. That's why I added a test. Anyway, I've fixed it in my github
> branch and in this patch series, and I've told them to try the fix.

Let us know when they make progress on that front.

If Emacs decides to switch to Git and decides to use this version of
remote-bzr for their conversion, or at least a nontrivial group of
developers favor to do so, without seeing concrete technical points
that say remote-bzr is not yet ready (e.g. "the conversion is still
wrong and X, Y and Z needs to be fixed"), that would be a very
welcome solid vote of confidence in favor of us going ahead with
this.

Thanks.

Re: [PATCH v2 00/18] remote-bzr: massive changes

[Felipe Contreras]

On Wed, May 1, 2013 at 11:39 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Felipe Contreras <felipe.contreras@gmail.com> writes:
>
>>> So let's go ahead and apply these directly on top of 'master', once
>>> we hear from Emacs folks and they are happy with it.  I'll queue it
>>> on 'pu' so that I do not have to go back to the list archive when it
>>> happens.
>>
>> I already heard that everything seems to be working correctly, except
>> one feature, the biggest change, which I screwed up with a one-liner
>> commit. That's why I added a test. Anyway, I've fixed it in my github
>> branch and in this patch series, and I've told them to try the fix.
>
> Let us know when they make progress on that front.
>
> If Emacs decides to switch to Git and decides to use this version of
> remote-bzr for their conversion, or at least a nontrivial group of
> developers favor to do so, without seeing concrete technical points
> that say remote-bzr is not yet ready (e.g. "the conversion is still
> wrong and X, Y and Z needs to be fixed"), that would be a very
> welcome solid vote of confidence in favor of us going ahead with
> this.

Seems unlikely for political reasons (isn't it always for GNU?), since
RMS is heavily involved in the decision.

-- 
Felipe Contreras

Re: [PATCH v2 00/18] remote-bzr: massive changes

[Junio C Hamano]

Felipe Contreras <felipe.contreras@gmail.com> writes:

> On Wed, May 1, 2013 at 11:39 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> Felipe Contreras <felipe.contreras@gmail.com> writes:
>>
>>>> So let's go ahead and apply these directly on top of 'master', once
>>>> we hear from Emacs folks and they are happy with it.  I'll queue it
>>>> on 'pu' so that I do not have to go back to the list archive when it
>>>> happens.
>>>
>>> I already heard that everything seems to be working correctly, except
>>> one feature, the biggest change, which I screwed up with a one-liner
>>> commit. That's why I added a test. Anyway, I've fixed it in my github
>>> branch and in this patch series, and I've told them to try the fix.
>>
>> Let us know when they make progress on that front.
>>
>> If Emacs decides to switch to Git and decides to use this version of
>> remote-bzr for their conversion, or at least a nontrivial group of
>> developers favor to do so, without seeing concrete technical points
>> that say remote-bzr is not yet ready (e.g. "the conversion is still
>> wrong and X, Y and Z needs to be fixed"), that would be a very
>> welcome solid vote of confidence in favor of us going ahead with
>> this.
>
> Seems unlikely for political reasons (isn't it always for GNU?), since
> RMS is heavily involved in the decision.

I am very aware of that discussion (and the original one when they
decided to use bzr).  That is exactly why I said "at least ... favor
to do so".

Re: [PATCH v2 00/18] remote-bzr: massive changes

[Ted Zlatanov]

On Wed, 01 May 2013 11:38:47 -0700 Junio C Hamano <gitster@pobox.com> wrote: 

JCH> Felipe Contreras <felipe.contreras@gmail.com> writes:
>> On Wed, May 1, 2013 at 11:39 AM, Junio C Hamano <gitster@pobox.com> wrote:
>>> Felipe Contreras <felipe.contreras@gmail.com> writes:
>>> 
>>>>> So let's go ahead and apply these directly on top of 'master', once
>>>>> we hear from Emacs folks and they are happy with it.  I'll queue it
>>>>> on 'pu' so that I do not have to go back to the list archive when it
>>>>> happens.
>>>> 
>>>> I already heard that everything seems to be working correctly, except
>>>> one feature, the biggest change, which I screwed up with a one-liner
>>>> commit. That's why I added a test. Anyway, I've fixed it in my github
>>>> branch and in this patch series, and I've told them to try the fix.
>>> 
>>> Let us know when they make progress on that front.
>>> 
>>> If Emacs decides to switch to Git and decides to use this version of
>>> remote-bzr for their conversion, or at least a nontrivial group of
>>> developers favor to do so, without seeing concrete technical points
>>> that say remote-bzr is not yet ready (e.g. "the conversion is still
>>> wrong and X, Y and Z needs to be fixed"), that would be a very
>>> welcome solid vote of confidence in favor of us going ahead with
>>> this.
>> 
>> Seems unlikely for political reasons (isn't it always for GNU?), since
>> RMS is heavily involved in the decision.

JCH> I am very aware of that discussion (and the original one when they
JCH> decided to use bzr).  That is exactly why I said "at least ... favor
JCH> to do so".

FYI, in case you're not aware, there's a pretty strong feeling on
emacs-devel that the switch to Git will happen and RMS is not opposed.

I don't know if they'll use remote-bzr, though.  It's more likely
they'll use one of the already-existing mirrors and sync it up, based on
the feedback so far.  It's a good time to bring remote-bzr up on
emacs-devel if you want it to be considered.

HTH
Ted