"Branch objects" (was: Re: cherry picking and merge)

"Branch objects" (was: Re: cherry picking and merge)

[Jakub Narębski]

On Wed, Aug 6, 2014 at 6:26 PM, Nico Williams <nico@cryptonector.com> wrote:
> On Wed, Aug 6, 2014 at 10:58 AM, Jakub Narębski <jnareb@gmail.com> wrote:
>> W dniu 2014-08-01 22:55, Nico Williams pisze:
>>>
>>> It would help if cherry-pick history where recorded somewhere (beyond
>>> the reflog)...
>>>
>>> Cherry-picks should record two parents, like merges.
>>>
>>> (Of course, it does no good to know about an unreachable parent, when
>>> a commit with two parents is pushed to a repo that doesn't have one of
>>> those parents, which can happen when topic branches aren't pushed
>>> upstream.)
>>
>> There was (long time ago) a long thread about idea of adding some
>> kind of 'weak' references (links), 'weakparent' that can be automatically
>> used by Git but do not pollute the commit message,
>> and do not affect reachability calculations.  Ultimately it went
>> nowhere (as you can see) - there were many problems.
>
> My proposal was to put this sort of ancillary history info in a
> "branch object" (and that branches should be objects).  This would
> have a number of benefits, not the least of which is that at push time
> you can drop such ancillary history without having to alter the
> commits being pushed.

Is it something like object-ified reflog, similar to how replacement
objects (git-replace) can be thought to be object-ified grafts (I know
they are more)? Do I understand it correctly?

Did you plan to (ab)use known object types: tree and commit (a bit
similar to git-replace and git-note object, though there is no need for
fanout trees - the top level tree can reproduce refs hierarchy)? I see
that you planned to (ab)use existing transfer mechanism of refs and
objects...


BTW. sometimes I do wonder if we are not making a mistake trying
to shoehorn new features like stash, replacements and notes into
DAG, objects (commit, tree, blob), refs and reflogs. I'd rather Git
did not make the same mistake (well, I think it was a mistake) that
Mercurial did with .hgtags file, (ab)using file transfer for tags, instead
of adding separate transfer mechanism like Git has... which led to
contortions in interpreting / deling with said file (most recent version
is used, not the one in checked out revision) and things like having
to commit creating a tag for it to be transferrable.

>> For example: how it would work for reverts and rebases?
>
> Reverts upstream?  The revert should record the commit hash of the
> commit it reverts (but file-level reverts lose), so that this could be
> noticed.

If it is object-ified reflog then reverts are not a problem...

> Rebases upstream?  Well, that shouldn't happen, but if it does then
> you must rebase --onto and any cherry-picks of upstream rebased
> commits lose their ties to those (but this can be detected).

With rebases the problem is that it would be nice to have (at least
for a short time) the history of series of patches (the metahistory,
or history of a branch), but usually one doesn't need old pre-rebase
version after cleaning up the history for publishing.

> In general recording more metadata (assuming there's not privacy
> issues to consider) can't hurt.  Using it might, but having the option
> to can also help.

True...

-- 
Jakub Narębski

Re: "Branch objects" (was: Re: cherry picking and merge)

[Nico Williams]

On Wed, Aug 06, 2014 at 08:31:18PM +0200, Jakub Narębski wrote:
> On Wed, Aug 6, 2014 at 6:26 PM, Nico Williams <nico@cryptonector.com> wrote:
> > My proposal was to put this sort of ancillary history info in a
> > "branch object" (and that branches should be objects).  This would
> > have a number of benefits, not the least of which is that at push time
> > you can drop such ancillary history without having to alter the
> > commits being pushed.
> 
> Is it something like object-ified reflog, similar to how replacement
> objects (git-replace) can be thought to be object-ified grafts (I know
> they are more)? Do I understand it correctly?

Yes, per-branch.  At push time a commit would be pushed to the upstream
branch listing the commits pushed now (and who by).  Locally every
rebase/cherry-pick/merge/commit onto the branch would appear in the
branch object's history, kinda just like the reflog.  The main
difference is that the upstream branch's history could be viewed.

> Did you plan to (ab)use known object types: tree and commit (a bit
> similar to git-replace and git-note object, though there is no need for
> fanout trees - the top level tree can reproduce refs hierarchy)? I see
> that you planned to (ab)use existing transfer mechanism of refs and
> objects...

Just like signed tags, basically.

> > Reverts upstream?  The revert should record the commit hash of the
> > commit it reverts (but file-level reverts lose), so that this could be
> > noticed.
> 
> If it is object-ified reflog then reverts are not a problem...

Right.

> > Rebases upstream?  Well, that shouldn't happen, but if it does then
> > you must rebase --onto and any cherry-picks of upstream rebased
> > commits lose their ties to those (but this can be detected).
> 
> With rebases the problem is that it would be nice to have (at least
> for a short time) the history of series of patches (the metahistory,
> or history of a branch), but usually one doesn't need old pre-rebase
> version after cleaning up the history for publishing.

Right.

> > In general recording more metadata (assuming there's not privacy
> > issues to consider) can't hurt.  Using it might, but having the option
> > to can also help.
> 
> True...

The principle should be to record as much metadata as possible, pruning
ancillary metadata (reflog-like metadata that isn't on the commits) only
at push time.  

Nico
-- 

Re: "Branch objects" (was: Re: cherry picking and merge)

[Tony Finch]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2301 bytes --]

Nico Williams <nico@cryptonector.com> wrote:
> On Wed, Aug 06, 2014 at 08:31:18PM +0200, Jakub Narębski wrote:
> > On Wed, Aug 6, 2014 at 6:26 PM, Nico Williams <nico@cryptonector.com> wrote:
> > >
> > > My proposal was to put this sort of ancillary history info in a
> > > "branch object" (and that branches should be objects).
> >
> > Is it something like object-ified reflog, similar to how replacement
> > objects (git-replace) can be thought to be object-ified grafts (I know
> > they are more)? Do I understand it correctly?
>
> Yes, per-branch.  At push time a commit would be pushed to the upstream
> branch listing the commits pushed now (and who by).  Locally every
> rebase/cherry-pick/merge/commit onto the branch would appear in the
> branch object's history, kinda just like the reflog.  The main
> difference is that the upstream branch's history could be viewed.
>
> > With rebases the problem is that it would be nice to have (at least
> > for a short time) the history of series of patches (the metahistory,
> > or history of a branch), but usually one doesn't need old pre-rebase
> > version after cleaning up the history for publishing.
>
> Right.

I have been fiddling around in this area.

What I want to be able to do is develop fixes for open source code that I
run, and get those fixes upstream. This means I need a rebasing workflow,
to keep the patches up-to-date and to deal with code review feedback.

But this is inconvenient for deploying the patched version to production
(which is the point of developing the fixes) - I want a fast-forwarding
branch for that. And it would be nice to be able to share the history of
the patch series, so others can see what changed between revisions more
easily.

So I have a small tool which maintains a publication branch which tracks
the head of a rebasing branch. It's reasonably satisfactory so far...

https://git.csx.cam.ac.uk/x/ucs/git/git-repub.git

... though the structure of the publication branch is weird and not very
easy to navigate. You can see it in action in my git.git repo:

https://git.csx.cam.ac.uk/x/ucs/git/git.git/shortlog/refs/heads/ucam/fanf2/patch

Tony.
-- 
f.anthony.n.finch  <dot@dotat.at>  http://dotat.at/
Irish Sea: Variable 4. Slight. Showers. Good.

Re: "Branch objects" (was: Re: cherry picking and merge)

[Nico Williams]

On Thu, Aug 07, 2014 at 12:38:48PM +0100, Tony Finch wrote:
> I have been fiddling around in this area.
> 
> What I want to be able to do is develop fixes for open source code
> that I run, and get those fixes upstream. This means I need a rebasing
> workflow, to keep the patches up-to-date and to deal with code review
> feedback.

Right.

> But this is inconvenient for deploying the patched version to
> production (which is the point of developing the fixes) - I want a

I'm not sure I follow this.  You deploy what you build, and you build
the HEAD of the production branch, whatever that is.  If it gets
rebased, so it it does.

> fast-forwarding branch for that. And it would be nice to be able to
> share the history of the patch series, so others can see what changed
> between revisions more easily.

But yes, it's nice to have a history of all the rebases.  For example:
so you can show the work you've done (rebasing to please an upstream is
work).

The reflog does this, of course, but you can't push it.  Of course, my
conception of branch object wouldn't push rebase history to an upstream
that doesn't want it, but you could push it to repos that do.

> So I have a small tool which maintains a publication branch which
> tracks the head of a rebasing branch. It's reasonably satisfactory so
> far...
> 
> https://git.csx.cam.ac.uk/x/ucs/git/git-repub.git

Yeah, that's useful.

Nico
-- 

Re: "Branch objects" (was: Re: cherry picking and merge)

[Tony Finch]

Nico Williams <nico@cryptonector.com> wrote:
> On Thu, Aug 07, 2014 at 12:38:48PM +0100, Tony Finch wrote:
>
> > But [a rebasing workflow] is inconvenient for deploying the patched
> > version to production (which is the point of developing the fixes) - I
> > want a fast-forwarding branch for that.
>
> I'm not sure I follow this.  You deploy what you build, and you build
> the HEAD of the production branch, whatever that is.  If it gets
> rebased, so it it does.

The problem is that the production branch gets copied around: pushed to
the repo server, pulled by other team members, etc. Forced pushes
are accident-prone, as is resetting a rebased branch after a pull.

> > So I have a small tool which maintains a publication branch which
> > tracks the head of a rebasing branch. It's reasonably satisfactory so
> > far...
> >
> > https://git.csx.cam.ac.uk/x/ucs/git/git-repub.git
>
> Yeah, that's useful.

Glad you think so :-)

Tony.
-- 
f.anthony.n.finch  <dot@dotat.at>  http://dotat.at/
Thames: Northeast veering southeast 4 or 5, occasionally 6. Slight, becoming
slight or moderate. Fair then rain or thundery showers. Good, becoming
moderate or poor for a time.

Re: "Branch objects" (was: Re: cherry picking and merge)

[Nico Williams]

On Thu, Aug 07, 2014 at 05:42:34PM +0100, Tony Finch wrote:
> Nico Williams <nico@cryptonector.com> wrote:
> > On Thu, Aug 07, 2014 at 12:38:48PM +0100, Tony Finch wrote:
> > > But [a rebasing workflow] is inconvenient for deploying the patched
> > > version to production (which is the point of developing the fixes) - I
> > > want a fast-forwarding branch for that.
> >
> > I'm not sure I follow this.  You deploy what you build, and you build
> > the HEAD of the production branch, whatever that is.  If it gets
> > rebased, so it it does.
> 
> The problem is that the production branch gets copied around: pushed to
> the repo server, pulled by other team members, etc. Forced pushes
> are accident-prone, as is resetting a rebased branch after a pull.

When I rebase and I need the old HEAD around I do something like this:

$ git checkout $branch_to_rebase
$ ver=${branch_to_rebase##*-}
$ git checkout -b ${branch_to_rebase%-${ver}}-$((ver+1))
$ git rebase ...

or like this:

$ git checkout $branch_to_rebase
$ git branch ${branch_to_rebase}-$(date +%Y-%m-%d)
$ git rebase ...

Either way I retain the old HEAD with some name.  This requires
discipline, so scripting it is useful.  But if you want discipline then
you want git to know that "for this branch, don't prune/gc old HEADs
orphaned after rebases" and "push the rebase history for this branch".

> > > https://git.csx.cam.ac.uk/x/ucs/git/git-repub.git
> >
> > Yeah, that's useful.
> 
> Glad you think so :-)

Thank you.

Nico
-- 

Re: "Branch objects" (was: Re: cherry picking and merge)

[Tony Finch]

Nico Williams <nico@cryptonector.com> wrote:
> On Thu, Aug 07, 2014 at 05:42:34PM +0100, Tony Finch wrote:
> >
> > The problem is that the production branch gets copied around: pushed to
> > the repo server, pulled by other team members, etc. Forced pushes
> > are accident-prone, as is resetting a rebased branch after a pull.
>
> When I rebase and I need the old HEAD around I do something like this:
> [...]
> Either way I retain the old HEAD with some name.

Hmm, yes, I can see that would work. However my previous workflow was
rather branch-heavy and I found the accumulation of names annoying. I have
not yet had enough usage out of git-repub to see if it goes too far in the
direction of lack-of-names. A big omission is no opportunity to edit its
commit messages.

Tony.
-- 
f.anthony.n.finch  <dot@dotat.at>  http://dotat.at/
Rockall: Southwesterly becoming cyclonic in north, 5 to 7. Moderate or rough.
Rain or showers. Moderate or good.

Re: "Branch objects" (was: Re: cherry picking and merge)

[Nico Williams]

On Thu, Aug 7, 2014 at 12:47 PM, Tony Finch <dot@dotat.at> wrote:
> Nico Williams <nico@cryptonector.com> wrote:
>> Either way I retain the old HEAD with some name.
>
> Hmm, yes, I can see that would work. However my previous workflow was
> rather branch-heavy and I found the accumulation of names annoying. I have
> not yet had enough usage out of git-repub to see if it goes too far in the
> direction of lack-of-names. A big omission is no opportunity to edit its
> commit messages.

Oh, I just read your script more carefully and looked at your example
history again.  You're using parent metadata in the commits to keep
the history alive without the extra names, correct?  *That* is
_clever_.  Hats off.  I may have to steal this script :)

Nico
--

Re: "Branch objects" (was: Re: cherry picking and merge)

[Nico Williams]

On Thu, Aug 7, 2014 at 6:38 AM, Tony Finch <dot@dotat.at> wrote:
> So I have a small tool which maintains a publication branch which tracks
> the head of a rebasing branch. It's reasonably satisfactory so far...
>
> https://git.csx.cam.ac.uk/x/ucs/git/git-repub.git
>
> ... though the structure of the publication branch is weird and not very
> easy to navigate. You can see it in action in my git.git repo:

You know, maybe you could even use this to automatically figure out
the merge base for downstreams that follow your rebased branch:
auto-generate the git rebase --onto <head> <head as it was prior to
all the upstream rebases>.

That would be awesome, particularly if integrated into git.  It would
then be fine to rebase published branches in most cases, for example.

Nico