[U-Boot] U-Boot git usage model

[Stephen Warren <swarren@wwwdotorg.org>]

On 10/09/2012 05:00 PM, Scott Wood wrote:
> On 10/09/2012 05:14:23 PM, Stephen Warren wrote:
>> On 10/09/2012 03:32 PM, Tom Rini wrote:
>> > On Tue, Oct 09, 2012 at 03:03:28PM -0600, Stephen Warren wrote:
>> >> On 10/09/2012 08:23 AM, Tom Rini wrote:
>> >>> On Sun, Oct 07, 2012 at 08:49:00PM +0200, Marek Vasut wrote:
>> >>>
>> >>>> NOTE: I get a few more size issues with ELDK 4.2 on IXP
>> >>>> (that big-endian ARM) after this patchset is applied. I
>> >>>> wonder if we shouldn't just throw these away, since they're
>> >>>> dead code mostly.
>> >>>>
>> >>>> The following changes since commit
>> >>>> c7ee66a8222660b565e9240775efa4c82cb348c2:
>> >>>>
>> >>>> Merge branch 'next' of git://www.denx.de/git/u-boot-ppc4xx
>> >>>> into next (2012-10-02 10:16:40 -0700)
>> >>>>
>> >>>> are available in the git repository at:
>> >>>>
>> >>>>
>> >>>> git://git.denx.de/u-boot-usb.git next
>> >>>>
>> >>>> for you to fetch changes up to
>> >>>> f0ede0e8305bc3c959862446bce40cb028b36293:
>> >>>>
>> >>>> usb.h: Add udc_disconnect prototype to usb.h (2012-10-07
>> >>>> 02:08:48 +0200)
>> >>>
>> >>> I had to rebase this locally to merge (such is next), and now
>> >>> it's applied to u-boot/next, thanks!
>> >>
>> >> Hmm. Can't "git merge" solve merge conflicts just as well as "git
>> >> rebase"?
>> >>
>> >> The problem with rebasing when pulling is that git commit IDs
>> >> change, so it's much more difficult to determine when a commit is
>> >> merged into a parent tree; one has to search by commit subject
>> >> rather than just executing e.g. git branch -a --contains XXX. I
>> >> thought Albert just agreed to use merges rather than rebases for
>> >> u-boot-arm for this and perhaps other reasons.
>> >
>> > The short answer is that right now, u-boot/next follows the
>> > linux-next model and we rebase as needed.
>>
>> I don't quite follow that; linux-next is also purely merge-based. Are
>> you referring to the fact that it's re-created every day, and the
>> source branches that go into the merge can be rebased if needed?
> 
> What's the difference between "re-created every day" and "rebased every
> day"?

The linux-next model (and what I mean by "re-created every day") is
roughly the following approximately daily:

git fetch --all
git branch -D tmp
git checkout -b tmp u-boot/master # or latest rc in fact
git merge u-boot-arm/next
git merge u-boot-i2c/next
git merge u-boot-video/next
etc.
git tag next-${datestamp}

I'm not sure what "rebased every day means"; perhaps it's running the
following on u-boot/next every day?

git rebase u-boot/master

That doesn't pull in any new commits from child trees though.

>> Instead, I think u-boot/next is just a place where patches get
>> applied, or branches get merged, before u-boot/master is open to
>> accept new patches for the next release. Unless I'm misunderstanding
>> it purpose of course...
> 
> That was my impression as well.
> 
>> Now, having a linux-next style daily merge of u-boot-*/next would be
>> pretty awesome.
> 
> Not really needed if the main next tree can permanently merge those
> branches.

Yes, if u-boot/next is the collection point where downstream branches
are permanently merged into, and once release N is made, u-boot/master
is reset to (or merged from) u-boot/next, then we effectively are
maintaining branches for two releases in parallel at once, with critical
bugfixes going into u-boot/master for the release, and everything else
going quickly into u-boot/next for the next release. That would
certainly avoid the need for "rebuilding" u-boot/next every day, since
u-boot/next would be very current already.

>> >> It would be awesome if U-Boot could adopt something more similar
>> >> to the Linux kernel's git usage model, namely:
>> >>
>> >> * All downstream branches are based off some known stable point
>> >> in the master branch (e.g. 2012.10-rc1). Before these branches
>> >> are merged into any other branch, they can be rebased if
>> >> absolutely needed, but preferably not.
>> >>
>> >> * Once a downstream branch is merged upwards, the downstream
>> >> branch doesn't merge upstream back down into the downstream
>> >> branch, but either:
>> >>
>> >> a) Keeps adding to the existing branch so that incremental pull
>> >> requests can be sent.
> 
> How does merging back down prevent incremental pull requests?

It doesn't prevent incremental pull requests, but it does pollute the
history if you merge back down. Instead of a fairly simple:

(M* == main branch, B* == side branch)

    B1-B2-B3-B4-B5-B6
   /       \        \
M1-M2-M3-M4-M4-M5-M6-M7

you might end up with:

    B1-B2-B3-B4-X1-B6
   /       \    /    \
M1-M2-M3-M4-M4-M5-M6-M7

... where X1 is the merge back from main to the side-branch. That
doesn't look a lot more complex, but once there are many side-branches,
and the master branch ends up merging a whole bunch of side-branches
between M4 and M5 above, the commits in B* that add actual new work are
split between a point before the X1 merge-back and after it. This can
make tracking down what exactly will get merged into M* when B* is
re-merged a bit trickier. git log M..B can probably show it fine, but if
you start looking at gitk it'll look quite complex.

>> >> Or often when u-boot/master has made a complete new release
>> >> does:
>> >>
>> >> b) Creates a new branch based on the latest rc or release from
>> >> u-boot/master.
> 
> That's a rebase.  How is that better than a (likely fast-forward) merge?

No, that's not a rebase. Rebase is when you take some existing commits
based on one commit and apply them to a different baseline commit
instead. If you're creating a new branch to take commits for a new
release, you're simply not applying the commits for release N+1 until
there's a branch ready to take them.

>> >> (in practice, downstream branches typically end up with something
>> >> like for-3.5 based on v3.4-rcN, for-3.6 based on v3.5-rcN,
>> >> for-3.7 based on v3.6-rcN, some running in parallel containing
>> >> either important bugfixes for the release or new development as
>> >> determined by the current state of the various releases in the
>> >> mainline tree).
> 
> I thought you said your way was less work? :-)

And I believe it is; no rebasing required. The difference probably isn't
that big though I admit. Still, creating a fresh branch from scratch for
each release one time and only then applying patches for that release
seems a lot simpler that constantly rebasing stuff all over the place.

>> >> * When a branch is merged from a repo to a parent repo, it's
>> >> always a git merge --no-ff; never a rebase or fast-forward.
>> >>
>> >> * In order to resolve merge conflicts/dependencies between
>> >> different downstream branches, one of the following happens:
>> >>
>> >> 1)
>> >>
>> >> a) The first downstream branch gets merged into u-boot/master. b)
>> >> The second downstream branch creates a new branch starting at an
>> >> an rc or release in u-boot-master that contains it the required
>> >> patches. c) The dependent patches are applied to the second
>> >> downstream branch. d) The second downstream branch gets merged
>> >> into u-boot/master.
>> >>
>> >> 2)
>> >>
>> >> All the patches that would usually be merged through downstream
>> >> branch 2 actually get ack'd by the maintainer of downstream
>> >> branch 2 and applied to downstream branch 1 after the patches
>> >> they depend on. This is simplest, but may cause complications if
>> >> both branches need to take patches that build on the merged
>> >> patches they're merged into an rc or release in u-boot/master.
>> >>
>> >> 3)
>> >>
>> >> A topic branch is created by one of the downstream maintainers,
>> >> branched from a u-boot/master rc or release, and containing just
>> >> the patches that other patches depend on, and this topic branch
>> >> gets merged into both the two downstream branches for further
>> >> work.
>> >>
>> >> Yes, this does all take a little bit more thought, planning, and
>> >> co-ordination, but I think having a simpler and more stable git
>> >> history is worth it.
> 
> What is the specific improvement in git history as a result of this?

I ended up describing this above; a much more linear less spaghetti history.

>> > Interesting.  As this is more work on the custodians end, what
>> > does everyone else say?
>>
>> This actually turns out to be less work for custodians if there aren't
>> any dependencies between patch series, since whenever you send a pull
>> request right now, you do:
>>
>> a) Fetch latest upstream.
>> b) Rebase onto it.
>> c) Send pull request.
> 
> That's what I used to do, but recently Wolfgang said no rebases, so I
> merge instead.
> 
>> but with the Linux model, you simply:
>>
>> a) Send pull request.
>>
>> Admittedly the recipient then might need to resolve some merge
>> conflicts. However, hopefully people have been planning for these and
>> have avoided them.
> 
> How do you plan for them and avoid them, and how is that less work that
> what we do now?

People have to be aware what is going on.

If you're submitting a bunch of patches which depend on each-other, the
submitter had better call that out when sending the patch series.

People performing large tree-wide rework need to communicate it ahead of
time so that everyone is aware it will happen, and plan to need to
either resolve conflicts when merging, /or/ hold off applying patches
until the rework has been performed and applied, and then apply patches
on top of that.

> It's one thing if a merge conflict comes from multiple pull requests
> being processed at once, but someone submitting a pull request should at
> least make sure that it doesn't conflict with top-of-tree by itself
> (some actual testing of the merge would be good too...).  And to do
> that, you've got to either merge or rebase, not just blindly request a
> pull.
> 
> I especially do not want to have to work with some artificially chosen
> old tree as my base.  I also do not want to create a bunch of named
> branches for each

Linux requires that branches be based only on rc or release tags in the
equivalent of u-boot/master. U-Boot could choose to be different, and
allow them to be based on any commit in u-boot/master or u-boot/next at
the branch creator's discretion; whatever baseline is needed to pick up
any required dependencies.