Tracking vendor release with Git

Tracking vendor release with Git

[Yann Droneaud]

Tracking vendor release with Git

[Yann Droneaud]

Hi,

I'm trying to setup a workflow to track vendor releases (upstream).
Each new release are provided as an archive of source code, data,
documentation, etc.

For each vendor releases, fixes need to be applied before making them
available to users (downstream).

Seems to be a rather common use case, applied by most Linux distribution
for decades.

In my case, on top of each releases, a common set of patches will be applied,
the biggest, the most intrusive one, being converting CRLF to LF using dos2unix,
the others being small portability fixes. In this case, fixes are not going to
be applied by upstream.

I'm trying to "design" (copy ;) a workflow with following properties,
in order of importance:

1- I wish to keep a branch with each new vendor release as a commit.
   This branch's history is only about vendor releases,
   so it's easy to read the "changelog" of the vendor releases
   with command such as git log <vendor-release-branch>

2- I'd like to ease the process of applying our patches on top
   of each new vendor release, eg. reduces the likeliness of conflicts.

3- I wish to keep a branch with each new fixed vendor release as a commit.
   Just like the upstream <vendor-release-branch>, only one commit
   per release, so it's easy to read the "changelog" of the vendor releases
   with command such as git log <patched-release-branch>
   
(4- I wish nobody is going to think about ponies ...)

The workflow, as I tried to implement it is, can be described like this:
For each release, the new sources are imported in the upstream branch,
then the previous branch holding fixes is rebased against the updated
upstream branch, and finally, the updated fixes branch is merged in
the downstream branch.

Please find a diagram trying to explain the following workflow:

                           Input       Output

                             .     *(1)  .
                             .    / \    .
                             .   /   \   .
                             .  /     \  .
                             . /       \ .
                             ./         \.
                             /           \
Upstream release 1           |           |
                  \          v           |
                   --------->*           |
                             |\          |
                             | \         |
                             |  |        |
                             |  v        |
Upstream release 2           |  * fix 1  |
                  \          |   \       |
                   \         |    \      v
                    \        |     ----->*  Downstream release 1
                     \       v           |
                      ------>*           |
                             |\          |
                             | \         |
                             |  |        |
                             |  v        |
                             |  * fix 1' |
                             |  |        |
Upstream release 3           |  v        |
                  \          |  * fix 2' |
                   \         |   \       |
                    \        |    \      v
                     \       |     ----->* Downstream release 2
                      \      |           |
                       \     v           |
                        ---->*           |
                             |\          |
                             . \         |
                             |  |        |
                             .  v        |
                             .  * fix 1" |
                             |  |        |
                             .  v        |
                             .  * fix 2" |
                             .  |        |
                             |  v        |
                                * fix 3" |
                                 \       |
                                  \      v
                                   ----->* Downstream release 3
                                         |
                                         .
                                         |
                                         .
                                         .

(1) empty root commit

I hope someone would come with an easier workflow, more sustainable.
At least, please help me find flaws in this workflow, tell me where
I can found the documentation of others workflows achieving the same
results.

BTW while testing the workflow, I tried "git merge -s theirs"
and found it doesn't exist. I thought it would be available for
such a common use case. For each merge to the of the fixes branch
to the downstream branch, something like a whole new content is dropped
to the branch. The previous content is still needed but its only for history.
So I prefer to overwrite the content of the downstream branch, instead
of fixing each conflicts "manually" when using "git merge" with default
strategy and option.

In some article[1], I've found an explanation of the lack of
"git merge -s theirs", quoting maintainer opinion[2] regarding it.

[1] <http://www.seanius.net/blog/2011/02/git-merge-s-theirs/>
[2] <http://thread.gmane.org/gmane.comp.version-control.git/76650/focus=89024>

I was able to use "git merge --strategy=recursive --strategy-option=theirs" aka.
"git merge -X theirs". So I'm only frustrated to see my workflow not being
considered. But happy to be able to do it with "uncommon" options.


Appended here a script to reproduce the workflow described previously:

#! /bin/sh

set -e

GIT_AUTHOR_DATE=`date -R`
GIT_COMMITTER_DATE="$GIT_AUTHOR_DATE"

export GIT_AUTHOR_DATE
export GIT_COMMITTER_DATE

# cleanup
rm -rf .git main.c

# prepare
git init
git commit --allow-empty -m "empty root commit"
git branch downstream master
git branch upstream master

#
# import first upstream release: v1
git checkout upstream
cat > main.c <<EOF

/* version string */
#define VERSION "1"

int
main(void)
{
    printf("version " VERSION);
}
EOF
git add main.c
git commit -m "Upstream release 1"

# apply fix on top of upstream release
# in an integration branch
git checkout -b upstream-1-fix upstream
cat > main.c <<EOF

/* version string */
#define VERSION "1"

int
main(void)
{
    printf("version " VERSION "\n");
}
EOF
git add main.c
git commit -m "Fix 1"

# make the "fixed" release available
# merge the integration branch
git checkout downstream
git merge --no-edit --no-ff upstream-1-fix
git tag downstream-default-1

#
# new upstream release: v2
git checkout upstream
cat > main.c <<EOF

/* version string */
#define VERSION "2"

int
main(void)
{
    printf("version " VERSION);
}
EOF
git add main.c
git commit -m "Upstream release 2"

# re-apply fix on top of new upstream release
# create a new integration branch to apply
# previous fix on top of the new release
git checkout -b upstream-2-fix upstream-1-fix
git rebase upstream
# apply a new fix
cat > main.c <<EOF

#include <stdio.h>

/* version string */
#define VERSION "2"

int
main(void)
{
    printf("version " VERSION "\n");
}
EOF
git add main.c
git commit -m "Fix 2"

# make the "fixed" release available
# merge the integration branch
git checkout downstream
git merge --no-edit --no-ff upstream-2-fix
git tag downstream-default-2

#
# new upstream release: v3
git checkout upstream
cat > main.c <<EOF

/* version string */
#define VERSION "3"
#define PACKAGE "foo"

int
main(void)
{
    printf("version " VERSION);
}
EOF
git add main.c
git commit -m "Upstream release 3"

# re-apply fixes on top of new upstream release
# create a new integration branch to apply
# previous fixes on top of the new release
git checkout -b upstream-3-fix upstream-2-fix
git rebase upstream
# apply a new fix
cat > main.c <<EOF

#include <stdio.h>

/* version string */
#define VERSION "3"
#define PACKAGE "foo"

int
main(void)
{
    printf("version " VERSION "\n");

    return 0;
}
EOF
git add main.c
git commit -m "Fix 3"

# make the "fixed" release available
# try to merge the integration branch
git checkout downstream
git merge --no-edit --no-ff upstream-3-fix || { 
echo "Merge failed, \"manually\" resolve conflict ..."
cat > main.c <<EOF

#include <stdio.h>

/* version string */
#define VERSION "3"
#define PACKAGE "foo"

int
main(void)
{
    printf("version " VERSION "\n");

    return 0;
}
EOF
git add main.c
git commit --no-edit
}
git tag downstream-default-3

# now, try a different merge strategy
git checkout master
git branch -D downstream
git branch downstream master

git checkout downstream

git merge --no-edit --no-ff --strategy-option theirs upstream-1-fix
git tag downstream-theirs-1
git merge --no-edit --no-ff --strategy-option theirs upstream-2-fix
git tag downstream-theirs-2
git merge --no-edit --no-ff --strategy-option theirs upstream-3-fix
git tag downstream-theirs-3

git checkout master

# "default" and "theirs" should match
# but have a different commit message:
# no conflict reported for "theirs"
git diff downstream-default-3 downstream-theirs-3


# Regards !

-- 
Yann Droneaud <ydroneaud@opteya.com>
OPTEYA

Re: Tracking vendor release with Git

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 1170 bytes --]


  I'm trying to setup a workflow to track vendor releases (upstream).
  Each new release are provided as an archive of source code, data,
  documentation, etc.

I've been doing more or less this.  A few comments:

  I suggest that you not view CRLF->LF as a "patch".  I would do EOL
  hygiene as a preprocessing script, with a checked-in script, after
  unpacking the tarball or whatever, and before 'import'.  Otherwise
  it's just going to be too messy.

  I use "vendor.foo" as the branch name.

  If your repo is only for this program, you can ignore this, but
  otherwise you way want to use subtree merge so that vendor.foo: maps
  to master:foo (putting foo in a subtree in master).  This lets you
  have multiple upstreams in one repo, which is useful for system
  building more than maintaining.

  I would avoid rebase.  You are essentially merging someone else's
  branch (that they aren't putting in git, but you are with the
  vendor.foo) into your master.   With regular merge, you can still
  diff, but the natural history will be right.  With rebase each "local
  version" as you call it will have different commits that will not have
  clear ancestry.

[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: Tracking vendor release with Git

[Johannes Sixt]

Am 11.06.2013 19:06, schrieb Yann Droneaud:
> Hi,
> 
> I'm trying to setup a workflow to track vendor releases (upstream).
> Each new release are provided as an archive of source code, data,
> documentation, etc.
> 
> For each vendor releases, fixes need to be applied before making them
> available to users (downstream).
> 
> Seems to be a rather common use case, applied by most Linux distribution
> for decades.
> 
> In my case, on top of each releases, a common set of patches will be applied,
> the biggest, the most intrusive one, being converting CRLF to LF using dos2unix,
> the others being small portability fixes. In this case, fixes are not going to
> be applied by upstream.
> 
> I'm trying to "design" (copy ;) a workflow with following properties,
> in order of importance:
> 
> 1- I wish to keep a branch with each new vendor release as a commit.
>    This branch's history is only about vendor releases,
>    so it's easy to read the "changelog" of the vendor releases
>    with command such as git log <vendor-release-branch>
> 
> 2- I'd like to ease the process of applying our patches on top
>    of each new vendor release, eg. reduces the likeliness of conflicts.
> 
> 3- I wish to keep a branch with each new fixed vendor release as a commit.
>    Just like the upstream <vendor-release-branch>, only one commit
>    per release, so it's easy to read the "changelog" of the vendor releases
>    with command such as git log <patched-release-branch>

I suggest you aim for the following history (time flows from left to right):

  U---V-----W          <-- upstream branch
   \   \     \
    C---D-----E        <-- CRLF conversion branch
     \   \     \
      K---L--M--N--O   <-- downstream branch

U, V, W are the upstream releases.

C is the initial CRLF->LF conversion. D merges the second upstream
release into the CRLF branch, E the third upstream release. These merges
very likely create tons of conflicts. But that does not matter, because
you know that the only change in "our" side is CRLF conversion. The
commits on this branch can easily be automated. That's the primary
motivation for this scheme.

K is your first small bugfix and also your first downstream release.

After merging L, the second, CRLF-converted, upstream release, you make
your second small change, M, which is also your second downstream release.

Rinse and repeat with N and O for the third release.

-- Hannes

Re: Tracking vendor release with Git

[Philip Oakley]

From: "Yann Droneaud" <ydroneaud@opteya.com>
Sent: Tuesday, June 11, 2013 6:06 PM
> Hi,
>
> I'm trying to setup a workflow to track vendor releases (upstream).
> Each new release are provided as an archive of source code, data,
> documentation, etc.
>
> For each vendor releases, fixes need to be applied before making them
> available to users (downstream).
>
> Seems to be a rather common use case, ...

Have you looked at the msysgit process that has to cope with upstream 
git ;-)

e.g. 
https://code.google.com/p/msysgit/source/browse/share/msysGit/merging-rebase.sh?name=python
https://github.com/msysgit/msysgit/blob/master/share/msysGit/merging-rebase.sh
and
https://github.com/msysgit/msysgit/blob/master/share/msysGit/rebasing-merge.sh

whereby the guys re-apply all the patches that haven't been accepted 
upstream, along with local fixups to get each new release working.

Philip 

Re: Tracking vendor release with Git

[Carsten Fuchs]

Hi Yann,

Am 2013-06-11 19:06, schrieb Yann Droneaud:
> I'm trying to setup a workflow to track vendor releases (upstream).
> Each new release are provided as an archive of source code, data,
> documentation, etc.
>
> For each vendor releases, fixes need to be applied before making them
> available to users (downstream).
>
> Seems to be a rather common use case, applied by most Linux distribution
> for decades.
>
> In my case, on top of each releases, a common set of patches will be applied,
> the biggest, the most intrusive one, being converting CRLF to LF using dos2unix,
> the others being small portability fixes. In this case, fixes are not going to
> be applied by upstream.


If you did the end-of-line conversion via .gitattributes rather than explicitly as a 
patch, maybe the strategy described at 
http://happygiraffe.net/blog/2008/02/07/vendor-branches-in-git/ is what you're looking for?

If besides the <pristine-vendor> branch you need another <patched-vendor> branch, this 
should be extensible, inserting another "layer" into the middle.
Copying and modifying Johannes' graph:

   U---V-----W          <-- upstream branch (pristine vendor)
    \   \     \
     C---D-----E        <-- patched vendor
      \   \     \
       K---L--M--N--O   <-- downstream branch ("master" in above linked text)


Best regards,
Carsten



-- 
Dipl.-Inf. Carsten Fuchs

Carsten Fuchs Software
Industriegebiet 3, c/o Rofu, 55768 Hoppstädten-Weiersbach, Germany
Internet: http://www.cafu.de | E-Mail: info@cafu.de

Cafu - the open-source game and graphics engine for multiplayer 3D action