Private/public branches/repos workflow

Private/public branches/repos workflow

[Jerome Martin]

Hello list,

I have an application that lives in a git tree. That application is a
commercial product, but we have decided to dual-licence it and release
an open-source version of it, along with all further commits to it.
Preparatory work for this has been done, and I now have two (let's
ignore the other) branches in my git tree, 'public' and 'private'.

The problem is, I cannot simply push the public branch on a public
repository, because the history contains a lot of stuff that are not
to be publicly released.

I thought I could squash all commits on that 'public' branch for the
initial public release and resume working from there, keeping the
ability to happily merge subsequent commits between 'public' and
'private', but this does not work: I am always left with the initial
commit of my whole tree (which unfortunately is not releasable) and
the big sqashed commit.

So what we did for now is to create a new git tree completely from
scratch, containing only the public code tree. However, I am afraid
this will lead to a nightmarish workflow, as it involves me working on
our internal tree 'public'/'private' branches, and from there,
replicate the commits to my internal 'public' branch to that published
repository. As pushing/pulling there results in having all the history
committed back to the published repo, I must resort to either having
to cherry pick all those commits one by one, or manually apply them to
the published repo. Maintenance hell.

Is there any recommended workflow to achieve the result I am looking
for ? How are other people doing it ?

Best Regards,
-- 
Jérôme Martin

Re: Private/public branches/repos workflow

[Seth Robertson]

In message <BANLkTi=TLeePH0JDvnE0XPiCzXFxid9BWQ@mail.gmail.com>, Jerome Martin
writes:

    Is there any recommended workflow to achieve the result I am looking
    for ? How are other people doing it ?

Create an orphan branch.  Commit what you want onto it.  Push branch
to a new repo as master.

git checkout <basesha>
git checkout --orphan cleanmaster
git add -A
git commit -m "base of public release"
(mkdir ../newrepo; cd ../newrepo; git init --bare)
git remote add cleanremote ../newrepo
git push cleanremote cleanmaster:master

You can then cherry pick/rebase from previous branches onto
cleanmaster any subsequent commits that you want to keep history for.

Another approach might be to finesse a shallow clone to do what you
want.

					-Seth Robertson

Re: Private/public branches/repos workflow

[Jerome Martin]

Thanks for your quick answer!
It seems to be a solution a bit alike one I tested before sending my
initial question, using :
git symbolic-ref HEAD refs/heads/cleanmaster
But I am unsure of the result I am getting.
Are you suggesting that I should entirely replace my current 'public'
branch (there is no important history in there to keep apart from what
is already in the other, private, branch) by cleanmaster or that I
should keep using three branches and push from that third one
(cleanmaster) to the published repo ?
One thing I do not understand clearly is why with a detached (I guess
this is synonym to orphan ? At least when creating it ?) branch cannot
be then merged with the branch that was initially copied to create it
and just get the additional commits. Instead, it gets the whole
history. Being used to using merge between my public/private branches,
I am afraid that using that orphan branch might lead to disaster
eventually.

On Thu, May 5, 2011 at 5:35 PM, Seth Robertson <in-gitvger@baka.org> wrote:
>
> In message <BANLkTi=TLeePH0JDvnE0XPiCzXFxid9BWQ@mail.gmail.com>, Jerome Martin
> writes:
>
>    Is there any recommended workflow to achieve the result I am looking
>    for ? How are other people doing it ?
>
> Create an orphan branch.  Commit what you want onto it.  Push branch
> to a new repo as master.
>
> git checkout <basesha>
> git checkout --orphan cleanmaster
> git add -A
> git commit -m "base of public release"
> (mkdir ../newrepo; cd ../newrepo; git init --bare)
> git remote add cleanremote ../newrepo
> git push cleanremote cleanmaster:master
>
> You can then cherry pick/rebase from previous branches onto
> cleanmaster any subsequent commits that you want to keep history for.
>
> Another approach might be to finesse a shallow clone to do what you
> want.
>
>                                        -Seth Robertson



--
Jérôme Martin

Re: Private/public branches/repos workflow

[Junio C Hamano]

Jerome Martin <tramjoe.merin@gmail.com> writes:

> I have an application that lives in a git tree. That application is a
> commercial product, but we have decided to dual-licence it and release
> an open-source version of it, along with all further commits to it.
> Preparatory work for this has been done, and I now have two (let's
> ignore the other) branches in my git tree, 'public' and 'private'.
>
> The problem is, I cannot simply push the public branch on a public
> repository, because the history contains a lot of stuff that are not
> to be publicly released.

To do this properly without heavy cherry-picking, you would need a lot of
discipline.  Not just "you" as an integrator, but everybody in your
development organization who has an access to the 'private' branch.

What you have is something like this:

                   O   public

 ...---x---x---x---X   private

where X is the tip of private branch that contains unreleasable stuff,
while O is the tip of public branch, a redacted version fit for releasing
to the general public.  The former has a lot of history behind it, commits
marked with 'x' that you are ashamed of showing to the public ;-), while
the latter does not have any history behind it (the initial code dump).

Of course, if you build some feature that is not proprietary directly on
top of X and make a commit 'a', and try to merge that to the public
branch, the resulting history will become this, pulling all the history
behind X:
                    O----*   public
                        /
 ...---x---x---x---X---a   private

Unlike "merge tracking" tacked on to some other systems, a merge in git is
a statement you make: "I looked at all the histories behind all the
commits I am merging, and decided that this tree I am recording as the
merge suits the purpose of my branch better than any of the parents".
Merging 'a' is not "The change between a's parent and a was cherry picked
onto this branch" (i.e. what mergeinfo records for SVN).  In short, if you
want to keep your x's private, you have to promise yourself that you will
never ever merge private to public.

        Side note.  This is exactly a same discipline for managing the
        maintenance track for the released version and the development
        track for the next release in the open source world.  You queue
        fixes to the maintenance track, and you may merge the maintenance
        to the development, but you never merge the development track to
        the maintenance, because you do not want to pull new features from
        the development branch into the bugfix only branch.

First of all, you would do this once before making any changes:

    $ git checkout private
    $ git merge -s ours public

This will create a new commit at the tip of private, that records that all
the histories of public is now contained in private, but the recorded
state (i.e. the tree) is identical to that of X.  Let's call this commit S
(it is a merge that synchronizes two independent histories):

                     O   public
                      \
 ...---x---x---x---X---S   private

Then, from here on, your developers have a choice to make whenever
starting to work on a new thing, be it a bugfix or a feature.

If the change will _never_ be released to the public, you can continue
building on your private branch.

                     O   public
                      \
 ...---x---x---x---X---S---x---x   private

On the other hand, if it will _eventually_ be merged to the public, then
you fork a topic branch from public, build it on that branch:

                       a---a---a---a  feature-A
                      /
                     O---o---o---o  public
                      \
 ...---x---x---x---X---S---x---x   private

and merge it to public:

                       a---a---a---a  feature-A
                      /             \   
                     O---o---o---o---*  public
                      \
 ...---x---x---x---X---S---x---x   private

I depicted such a topic as "feature-A" that consists of four commits 'a'
in the pictures above.  Since you made the initial merge S, open source
world may have added three commits 'o' to improve your system and you also
have a few more commits 'x' that are secret.

The private branch can take the result of the work done on the topic
branch, either by merging public wholesale:

                       a---a---a---a  feature-A
                      /             \   
                     O---o---o---o---*  public
                      \               \ 
 ...---x---x---x---X---S---x---x-------*  private

in which case you will also get the open source improvements 'o', or by
merging that particular topic alone:

                       a---a---a---a...    feature-A
                      /             \  . 
                     O---o---o---o---*  .  public
                      \                  .
 ...---x---x---x---X---S---x---x----------*  private

in which case your private branch will not (yet) have the three 'o' open
source improvements.

Note that you may consider some of your features "differentiating edge"
that you may want to keep private for one year and then release it to the
public.  In such a case, you can even choose to merge feature-A to private
first:

                       a---a---a---a...    feature-A
                      /                . 
                     O---o---o---o      .  public
                      \                  .
 ...---x---x---x---X---S---x---x----------*  private

and after a year, you can merge it to public to arrive at the previous
picture.

Re: Private/public branches/repos workflow

[Jerome Martin]

Junio,

Thanks a lot for your lengthy email and nice ASCII art making it a lot
clearer to me why I was running in circles with the wrong idea on
merge semantics :-) I think I understand clearly what you suggest, and
the part about discipline is the conclusion I was drawing myself when
answering Seth above.

I understand there is no silver bullet, and that an orphan branch +
discipline (regarding merges) + clever usage/planning of development
branches is as good as the system might get with that 'hidden' history
restriction. However, I am still wondering how (un?)practical this is
in real life.

It is probably just me not being fluent enough with git, but I still
cannot help feeling the whole workflow as being awkward and
error-prone for a daily usage. But hey, you seem to say that it is
already used around a lot, at least for the maintenance/development
branches case, so I guess it must be usable ;-)

Could you tell me how you would see usage of rebasing inside the
workflow you describe ? I am thinking about features (mistakenly)
implemented on top of the private branch that need later to be release
in the public branch. Can this be the shortcut to avoid lengthy
cherry-picks ?

On Thu, May 5, 2011 at 8:14 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Jerome Martin <tramjoe.merin@gmail.com> writes:
>
>> I have an application that lives in a git tree. That application is a
>> commercial product, but we have decided to dual-licence it and release
>> an open-source version of it, along with all further commits to it.
>> Preparatory work for this has been done, and I now have two (let's
>> ignore the other) branches in my git tree, 'public' and 'private'.
>>
>> The problem is, I cannot simply push the public branch on a public
>> repository, because the history contains a lot of stuff that are not
>> to be publicly released.
>
> To do this properly without heavy cherry-picking, you would need a lot of
> discipline.  Not just "you" as an integrator, but everybody in your
> development organization who has an access to the 'private' branch.
>
> What you have is something like this:
>
>                   O   public
>
>  ...---x---x---x---X   private
>
> where X is the tip of private branch that contains unreleasable stuff,
> while O is the tip of public branch, a redacted version fit for releasing
> to the general public.  The former has a lot of history behind it, commits
> marked with 'x' that you are ashamed of showing to the public ;-), while
> the latter does not have any history behind it (the initial code dump).
>
> Of course, if you build some feature that is not proprietary directly on
> top of X and make a commit 'a', and try to merge that to the public
> branch, the resulting history will become this, pulling all the history
> behind X:
>                    O----*   public
>                        /
>  ...---x---x---x---X---a   private
>
> Unlike "merge tracking" tacked on to some other systems, a merge in git is
> a statement you make: "I looked at all the histories behind all the
> commits I am merging, and decided that this tree I am recording as the
> merge suits the purpose of my branch better than any of the parents".
> Merging 'a' is not "The change between a's parent and a was cherry picked
> onto this branch" (i.e. what mergeinfo records for SVN).  In short, if you
> want to keep your x's private, you have to promise yourself that you will
> never ever merge private to public.
>
>        Side note.  This is exactly a same discipline for managing the
>        maintenance track for the released version and the development
>        track for the next release in the open source world.  You queue
>        fixes to the maintenance track, and you may merge the maintenance
>        to the development, but you never merge the development track to
>        the maintenance, because you do not want to pull new features from
>        the development branch into the bugfix only branch.
>
> First of all, you would do this once before making any changes:
>
>    $ git checkout private
>    $ git merge -s ours public
>
> This will create a new commit at the tip of private, that records that all
> the histories of public is now contained in private, but the recorded
> state (i.e. the tree) is identical to that of X.  Let's call this commit S
> (it is a merge that synchronizes two independent histories):
>
>                     O   public
>                      \
>  ...---x---x---x---X---S   private
>
> Then, from here on, your developers have a choice to make whenever
> starting to work on a new thing, be it a bugfix or a feature.
>
> If the change will _never_ be released to the public, you can continue
> building on your private branch.
>
>                     O   public
>                      \
>  ...---x---x---x---X---S---x---x   private
>
> On the other hand, if it will _eventually_ be merged to the public, then
> you fork a topic branch from public, build it on that branch:
>
>                       a---a---a---a  feature-A
>                      /
>                     O---o---o---o  public
>                      \
>  ...---x---x---x---X---S---x---x   private
>
> and merge it to public:
>
>                       a---a---a---a  feature-A
>                      /             \
>                     O---o---o---o---*  public
>                      \
>  ...---x---x---x---X---S---x---x   private
>
> I depicted such a topic as "feature-A" that consists of four commits 'a'
> in the pictures above.  Since you made the initial merge S, open source
> world may have added three commits 'o' to improve your system and you also
> have a few more commits 'x' that are secret.
>
> The private branch can take the result of the work done on the topic
> branch, either by merging public wholesale:
>
>                       a---a---a---a  feature-A
>                      /             \
>                     O---o---o---o---*  public
>                      \               \
>  ...---x---x---x---X---S---x---x-------*  private
>
> in which case you will also get the open source improvements 'o', or by
> merging that particular topic alone:
>
>                       a---a---a---a...    feature-A
>                      /             \  .
>                     O---o---o---o---*  .  public
>                      \                  .
>  ...---x---x---x---X---S---x---x----------*  private
>
> in which case your private branch will not (yet) have the three 'o' open
> source improvements.
>
> Note that you may consider some of your features "differentiating edge"
> that you may want to keep private for one year and then release it to the
> public.  In such a case, you can even choose to merge feature-A to private
> first:
>
>                       a---a---a---a...    feature-A
>                      /                .
>                     O---o---o---o      .  public
>                      \                  .
>  ...---x---x---x---X---S---x---x----------*  private
>
> and after a year, you can merge it to public to arrive at the previous
> picture.
>
>



-- 
Jérôme Martin

Re: Private/public branches/repos workflow

[Jerome Martin]

Thanks for the additional clarification on detached vs orphan.

I want to thank you guys again for being so patient and helpful. It
helps a lot, as for some reason, only reading the docs doesn't seem to
sink in as fast as I would like ;-)

Regarding the usage of an orphan branch, a coworker of mine suggested
an alternate way, which seems similar to me regarding the result, but
if you could take a look at it and tell me what you think, that would
be very helpful:

1) On the public server, init a blank repo, and commit there an export
of the public tree as initial import.
2) On the devel machines, use git-remote to track that repo's master
as a local 'public' branch. That public branch now has no history
other than the initial import, and is in the same repo as both my
private branch and my public-branch-with-full-history (not to be used
for future work).
3) From there, use this 'public' branch as if it was created using
--orphan from my local public-branch-with-full-history.

Are there any essential properties that set apart a 'public' branch
created this way from one creating using your suggested --orphan
method ?

On Thu, May 5, 2011 at 11:10 PM, Seth Robertson <in-gitvger@baka.org> wrote:
>
> In message <BANLkTikFdV=nGW-L+3NVzP58uPiS5ui-9A@mail.gmail.com>, Jerome Martin
> writes:
>
>    One thing I do not understand clearly is why with a detached (I guess this
>    is synonym to orphan ? At least when creating it ?) branch cannot be then
>    merged with the branch that was initially copied to create it and just get
>    the additional commits. Instead, it gets the whole history. Being used to
>    using merge between my public/private branches, I am afraid that using that
>    orphan branch might lead to disaster eventually.
>
> Junio appears to have answered much of these followup questions so I
> will focus on a slight misunderstand you exhibit here.
>
> An orphan branch is very different from a detached head.  A detached
> head is not on any branch.  Commits on detached heads cannot typically
> be referenced by any symbolic name and will eventually go away through
> garbage collection.  In general you want to avoid detached heads for
> any read/write activity.
>
> An orphan branch, on the other hand, is one which simply doesn't have
> a parent.  The first commit you make in a new repository is also
> orphan, though it is not talked about as such.  The point of this
> orphan branch is that since it does not have a parental link from your
> private branch, if you publish the orphan it will not bring along any
> of the private commits.
>
> As Junio stated/suggested, as long as you are careful to never merge
> from private into public, and instead either cherry-pick or do
> development from public and merge that into private, you should be
> safe.  However, a good safety test might be to have an intermediate
> repository be part of your public push process, and have a
> post-receive (or other process) script run which looks for magic
> cookies in that intermediate repo showing that private data has leaked
> before proceeding to push that validated repo to a publicly accessible
> repo.
>
>                                        -Seth Robertson
>



-- 
Jérôme Martin

Re: Private/public branches/repos workflow

[Junio C Hamano]

Jerome Martin <tramjoe.merin@gmail.com> writes:

> Could you tell me how you would see usage of rebasing inside the
> workflow you describe ? I am thinking about features (mistakenly)
> implemented on top of the private branch that need later to be release
> in the public branch. Can this be the shortcut to avoid lengthy
> cherry-picks ?

Mistakes and changes of plans can happen, but it's not a big deal.

In the git project itself, even the "master" branch that corresponds to
your "private" branch is managed with topic branch workflow, so it is
entirely possible that a topic that was originally meant only for the next
release turns out to be better backported to the maintenance track.

If you have a private "feature-A" topic with many commits:

                     O   public
                      \
 ...---x---x---x---X---S  private
                        \
                         a---b---...---z  feature-A

and later it turns out that it is better to release them also to the
public, instead of cherry-picking the commits to the public, you could at
that point do something like this:

    $ git checkout -b public-feature-A feature-A
    $ git rebase --onto public private

which will first create a new branch "public-feature-A" that points at the
same tip of "feature-A", and then rebases the commits on that branch that
were made since the topic was forked from "private" on top of the tip of
the "public" branch, resulting in:

                       a'--b'--...--z' public-feature-A
                      /
                     O   public
                      \
 ...---x---x---x---X---S  private
                        \
                         a---b---...---z  feature-A

and then you may choose when to merge that to the "public" branch.
By the time you do this, perhaps the original "feature-A" may have already
been merged to "private" after fully tested and audited, like this:

                       a'--b'--...--z' public-feature-A
                      /
                     O   public
                      \
 ...---x---x---x---X---S---x---x---x-----*  private
                        \               / 
                         a---b---...---z  feature-A

and then after you merged "public-feature-A" to "public":

                       a'--b'--...--z' public-feature-A
                      /              \  
                     O---o---...---o--*   public
                      \
 ...---x---x---x---X---S---x---...---x---*  private
                        \               / 
                         a---b---...---z  feature-A

merging "public" back to "private" hopefully may have too many conflicts
between the feature-A and public-feature-A branches, as they are supposed
to be patch equivalent, to produce this:


                       a'--b'--...--z' public-feature-A
                      /              \  
                     O---o---...---o--*---.   public
                      \                     \
 ...---x---x---x---X---S---x---...---x---*---*   private
                        \               / 
                         a---b---...---z  feature-A

After this you could even garbage collect the two topic branches ;-).

Re: Private/public branches/repos workflow

[Jerome Martin]

Again, thanks for the help :-)

On Fri, May 6, 2011 at 6:10 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Jerome Martin <tramjoe.merin@gmail.com> writes:
>
>> Could you tell me how you would see usage of rebasing inside the
>> workflow you describe ? I am thinking about features (mistakenly)
>> implemented on top of the private branch that need later to be release
>> in the public branch. Can this be the shortcut to avoid lengthy
>> cherry-picks ?
>
> Mistakes and changes of plans can happen, but it's not a big deal.
>
> In the git project itself, even the "master" branch that corresponds to
> your "private" branch is managed with topic branch workflow, so it is
> entirely possible that a topic that was originally meant only for the next
> release turns out to be better backported to the maintenance track.
>
> If you have a private "feature-A" topic with many commits:
>
>                     O   public
>                      \
>  ...---x---x---x---X---S  private
>                        \
>                         a---b---...---z  feature-A
>
> and later it turns out that it is better to release them also to the
> public, instead of cherry-picking the commits to the public, you could at
> that point do something like this:
>
>    $ git checkout -b public-feature-A feature-A
>    $ git rebase --onto public private
>
> which will first create a new branch "public-feature-A" that points at the
> same tip of "feature-A", and then rebases the commits on that branch that
> were made since the topic was forked from "private" on top of the tip of
> the "public" branch, resulting in:
>
>                       a'--b'--...--z' public-feature-A
>                      /
>                     O   public
>                      \
>  ...---x---x---x---X---S  private
>                        \
>                         a---b---...---z  feature-A
>
> and then you may choose when to merge that to the "public" branch.
> By the time you do this, perhaps the original "feature-A" may have already
> been merged to "private" after fully tested and audited, like this:
>
>                       a'--b'--...--z' public-feature-A
>                      /
>                     O   public
>                      \
>  ...---x---x---x---X---S---x---x---x-----*  private
>                        \               /
>                         a---b---...---z  feature-A
>
> and then after you merged "public-feature-A" to "public":
>
>                       a'--b'--...--z' public-feature-A
>                      /              \
>                     O---o---...---o--*   public
>                      \
>  ...---x---x---x---X---S---x---...---x---*  private
>                        \               /
>                         a---b---...---z  feature-A
>
> merging "public" back to "private" hopefully may have too many conflicts
> between the feature-A and public-feature-A branches, as they are supposed
> to be patch equivalent, to produce this:
>
>
>                       a'--b'--...--z' public-feature-A
>                      /              \
>                     O---o---...---o--*---.   public
>                      \                     \
>  ...---x---x---x---X---S---x---...---x---*---*   private
>                        \               /
>                         a---b---...---z  feature-A
>
> After this you could even garbage collect the two topic branches ;-).
>



-- 
Jérôme Martin