Is there a way to exclude user-specified files or directories from participating in merges?

Is there a way to exclude user-specified files or directories from participating in merges?

[Brent Goodrick]

Suppose I create a git repo called central.git on a machine I will
call "central". In that central.git repo, I put these files:

  work.sh
  home.sh
  generic.sh

When I clone the central.git repo on to a different machine I will
call "work", I want this fileset to be pulled:

  work.sh
  generic.sh

But not the home.sh file.

Similarly, when I clone the central.git repo on a machine I will call
"home", I want this fileset to be pulled:

  home.sh
  generic.sh

But not the work.sh file.

What I think I need are two branches, one called "home_branch" and
"work_branch", but read on for the twist:

Say I'm working on editing the work machines fileset on the work repo
I had cloned originally from central.git, and commit a change to both
generic.sh and work.sh.  I do a git-push to an appropriate remote
branch I have set up on the central.git repo, so that I can do a
git-merge type of integration on the central machine in the
central.git repo into the other branches (i.e., into the home_branch),
specifically so that the home_branch gets updated with the change to
the generic.sh file.  However, I want the home_branch to be updated
with the change I made to generic.sh, but I don't ever want the
work.sh to show up in the home_branch that would occur during a normal
merge. Likewise, I would not ever want the home.sh file to participate
in merges from the work fileset back over to the home fileset (and
likewise I would not ever want to see the work.sh file show up on the
home_branch).

The above should apply for all files certain special directories. For
instance, if I were to have a work_files directory and a home_files
directory, then the the work_files is for the "work" machine (and
work_branch) and the "home_files" is for the "home" machine (and
home_branch).

How do I mark certain files and/or directories (via relative file
paths or with file globbing) on certain branches to be excluded from
being merged into all other (or a specified list of) branches?
Ideally I would want to only have to add some logic to the .git/config
file in the central.git repo that specifies the exclusions/exceptions,
and not have to remember to make corresponding changes into any of the
other repos that are cloned from it. Is this possible?

Also, is there a way to avoid the home.sh file from ever being added
to any file underneath the .git directory of the repo I cloned on the
"work" machine. I would not want there to be any risk that anyone with
network access to the "work" machine (say, a sysadmin with sufficient
privileges) to be able to see any form of the home.sh file since it
exists on some branch in the .git directory.

Thanks,
Brent

Re: Is there a way to exclude user-specified files or directories from participating in merges?

[Junio C Hamano]

Brent Goodrick <bgoodr@gmail.com> writes:

> Suppose I create a git repo called central.git on a machine I will
> call "central". In that central.git repo, I put these files:
>
>   work.sh
>   home.sh
>   generic.sh
>
> When I clone the central.git repo on to a different machine I will
> call "work", I want this fileset to be pulled:
>
>   work.sh
>   generic.sh
>
> But not the home.sh file.

You would have one common branch and one branch per deployment.

A common branch would host only common files (e.g. generic.sh file from
your example).  Per deployment branch, e.g. home, would branch from the
common branch (so it starts with some version of generic.sh) and may add
its own private files (e.g. home.sh).

And stick to the following two rules:

 - You make edits to common files only on the common branch.
 - You merge from common to deployment, never the other way.

So at work, you would have a checkout of your work "deployment branch",
and find needs to change things.  It is Ok to edit both work.sh and
generic.sh (without being able to edit both, it would be hard to verify if
the changes would work together) at this time, but don't commit the result
in the work branch.

Save the changes to work.sh away (e.g. "git diff work.sh >P.diff" and then
"git checkout HEAD work.sh"), switch to the common branch, and commit the
changes to the generic file.  Switch back to the deployment branch, merge
the common branch (to pick up the changes to home.sh), reapply the changes
specific to the deployment you saved earlier (e.g. "git apply P.diff"),
tne commit the result.

Re: Is there a way to exclude user-specified files or directories from participating in merges?

[Brent Goodrick]

On Tue, Feb 17, 2009 at 5:05 PM, Junio C Hamano <gitster@pobox.com> wrote:
> So at work, you would have a checkout of your work "deployment branch",
> and find needs to change things.  It is Ok to edit both work.sh and
> generic.sh (without being able to edit both, it would be hard to verify if
> the changes would work together) at this time, but don't commit the result
> in the work branch.
>
> Save the changes to work.sh away (e.g. "git diff work.sh >P.diff" and then
> "git checkout HEAD work.sh"), switch to the common branch, and commit the
> changes to the generic file.  Switch back to the deployment branch, merge
> the common branch (to pick up the changes to home.sh), reapply the changes
> specific to the deployment you saved earlier (e.g. "git apply P.diff"),
> tne commit the result.
>

Thanks. Well, I should have said in my initial request: "Without
manually forwarding changes from branch to branch and without having
to remember special rules about what I can and cannot merge into which
branch", since that is likely to get forgotten. :)

The answer I am hearing you say is that git doesn't have a way to
automatically exclude files akin to how rsync handles include/exclude.
 Is that what you are saying? Or, could the hook mechanism be
exploited to get this behavior?

bg

Re: Is there a way to exclude user-specified files or directories from participating in merges?

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Brent Goodrick <bgoodr@gmail.com> writes:
>
>> Suppose I create a git repo called central.git on a machine I will
>> call "central". In that central.git repo, I put these files:
>>
>>   work.sh
>>   home.sh
>>   generic.sh
>>
>> When I clone the central.git repo on to a different machine I will
>> call "work", I want this fileset to be pulled:
>>
>>   work.sh
>>   generic.sh
>>
>> But not the home.sh file.
>
> You would have one common branch and one branch per deployment.
>
> A common branch would host only common files (e.g. generic.sh file from
> your example).  Per deployment branch, e.g. home, would branch from the
> common branch (so it starts with some version of generic.sh) and may add
> its own private files (e.g. home.sh).
>
> And stick to the following two rules:
>
>  - You make edits to common files only on the common branch.
>  - You merge from common to deployment, never the other way.
>
> So at work, you would have a checkout of your work "deployment branch",
> and find needs to change things.  It is Ok to edit both work.sh and
> generic.sh (without being able to edit both, it would be hard to verify if
> the changes would work together) at this time, but don't commit the result
> in the work branch.
>
> Save the changes to work.sh away (e.g. "git diff work.sh >P.diff" and then
> "git checkout HEAD work.sh"), switch to the common branch, and commit the
> changes to the generic file.  Switch back to the deployment branch, merge
> the common branch (to pick up the changes to home.sh), reapply the changes
> specific to the deployment you saved earlier (e.g. "git apply P.diff"),
> tne commit the result.

By the way, earlier in a different thread, somebody wondered if being able
to make a commit on detached HEAD is a good thing, and this is a good
example of why it is convenient.  When I use the above "one-way merge,
never merging back" workflow in real life [*1*], I do not use a simple
"git diff work.sh >P.diff && git checkout HEAD work.sh" to save away the
tentative changes I made and verified on the deployment branch.  The above
is an oversimplified example.

For one, in real life projects, there are many files that are specific to
individual deployments, and for another, distinction between generic vs
deployment specific changes does not cleanly appear at file boundaries.

Instead, I would make a real commit, with full intention of discarding it
later.

                  T work
                 /
         ...o---o
               / 
       ...o---o common

I would go back to common branch ("git checkout common"), cherry-pick the
commit from the deployment branch ("git cherry-pick --no-commit work").
This would conflict heavily because the common branch does not have any
changes specific to the deployment branch (i.e. files that were modified
since the deployment forked from the generic, and files the deployment
added).  That is Ok.  I would use "git add -i" and editor to sift out the
deployment specific parts to discard, and commit only the parts of the
change T made that are truly generic:

                  T work
                 /
         ...o---o
               / 
       ...o---o---A common

Then I would go back to the deployment, and detach the head at commit
before the tentative commit T.  From here, I can merge the common branch
in.

                  T work
                 /
         ...o---o HEAD
               / 
       ...o---o---A common

"git merge common" would make something like this:

                  T work
                 /
         ...o---o---B HEAD
               /   /
       ...o---o---A common

I know that the remainder of the change (i.e. difference between B and T)
should be the parts that are specific to the deployment.  After reading
through the output of "git diff HEAD work" to make sure that it has all
deployment specific changes I made (and nothing that should have been in
the commit A on common), I would:

	git read-tree -m -u work && git commit

to grow the history of detached HEAD.

                  T work
                 /
         ...o---o---B---C HEAD
               /   /
       ...o---o---A common

After this step, just wrap it up with:

	git branch -f work && git checkout work

to reach the final state:

                  T
                 /
         ...o---o---B---C work
               /   /
       ...o---o---A common

The tentative commit T becomes dangling but we know "diff T C" are empty
and records the good state we verified when we made T.

This is one case I still use the plumbing "read-tree -m -u".


[Footnote]

*1* http://gitster.livejournal.com/26540.html

Re: Is there a way to exclude user-specified files or directories from participating in merges?

[Junio C Hamano]

Brent Goodrick <bgoodr@gmail.com> writes:

> Thanks. Well, I should have said in my initial request: "Without
> manually forwarding changes from branch to branch and without having
> to remember special rules about what I can and cannot merge into which
> branch", since that is likely to get forgotten. :)
>
> The answer I am hearing you say is that git doesn't have a way to
> automatically exclude files akin to how rsync handles include/exclude.
>  Is that what you are saying? Or, could the hook mechanism be
> exploited to get this behavior?

A merge is defined as a whole tree operation simply because there is no
sane way to support repeated merges (even a single direction merges)
otherwise.  What I explained was one (note that I am not saying "one true"
here) workflow that naturally supports "common version and multiple
variants" pattern that logically follows the definition of what a merge
is.

I would imagine you could define a custom merge strategy that knows to
ignore changes you made to work.sh file when you merge from work to common
(and home.sh file when you merge from home to common) to implement what
you would want, but for one thing the resulting history would not make
sense (e.g. a merge from work to central would appear as if it reverts all
the changes work made to certain files).  It would be Ok if you were using
git as a mere backup+sneakernet medium (in such a case you would not care
what the history would show you), but that is not the intended target of
git, so there is no such built-in support.

Also such a custom merge strategy would be very project specific and as
your project grows and/or as you add more deployments, I suspect its rules
will have to become a lot more complicated.  I haven't even thought about
what should happen in such a merge strategy when you try to merge work to
home.

Compared to that, the two simple rules "commit chagnes to generic things
only to the generic branch" and "merge only from generic to specific" will
not grow as your project grows complexity.

Re: Is there a way to exclude user-specified files or directories from participating in merges?

[Brent Goodrick]

On Tue, Feb 17, 2009 at 5:58 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Compared to that, the two simple rules "commit changes to generic things
> only to the generic branch" and "merge only from generic to specific" will
> not grow as your project grows in complexity.

I now I see the wisdom of the above statement.  You've given me a lot
of approaches to think about and try. Thanks for your help!

bg

Re: Is there a way to exclude user-specified files or directories from participating in merges?

[Sitaram Chamarty]

On 2009-02-18, Junio C Hamano <gitster@pobox.com> wrote:
> And stick to the following two rules:
>
>  - You make edits to common files only on the common branch.
>  - You merge from common to deployment, never the other way.
>
> So at work, you would have a checkout of your work "deployment branch",
> and find needs to change things.  It is Ok to edit both work.sh and
> generic.sh (without being able to edit both, it would be hard to verify if
> the changes would work together) at this time, but don't commit the result
> in the work branch.
>
> Save the changes to work.sh away (e.g. "git diff work.sh >P.diff" and then
> "git checkout HEAD work.sh"), switch to the common branch, and commit the
> changes to the generic file.  Switch back to the deployment branch, merge
> the common branch (to pick up the changes to home.sh), reapply the changes
> specific to the deployment you saved earlier (e.g. "git apply P.diff"),
> tne commit the result.

[I did read your followup also; my question applies to both
versions of the technique]

Let me explain where I'm coming from: this is very often
needed when you maintain customer specific branches, and the
workflows in both your posts in this thread so far are too
complex for, err, me <sheepish grin> :-)

Would it not be easier to do something like this?  (I suck
at 2-d drawing, even line... but this should still be
understandable)

(W = work, T = temporary, C = common)

  - make granular commits and test etc, from W to T

        O---a---b+1---c---2---d---3
        W is pointing at commit O
        T is pointing at commit 3
        b+1 is a commit that contains both types of changes

  - use rebase -i (including split commits if needed, as
    described in 'git help rebase') to put all the changes
    that go to master before the ones that only go to work.

        O---a---b---c---d---1---2---3

  - (retest if needed)

  - cherry pick the first set of changes to common (in this
    example, a, b, c, d will become a', etc on common)

  - merge from common to work (x, y, etc are some other
    changes that went into common since the last time you
    merged)

        O---x---y---a'---b'---c'---d'
        W is now pointing at d'

  - cherry pick the stuff that remains

        O---x---y---a'---b'---c'---d'---1'---2'---3'
        W is now pointing at 3'

Re: Is there a way to exclude user-specified files or directories from participating in merges?

[Junio C Hamano]

Sitaram Chamarty <sitaramc@gmail.com> writes:

> Let me explain where I'm coming from: this is very often needed when you
> maintain customer specific branches, and the workflows in both your
> posts in this thread so far are too complex for, err, me <sheepish grin>
> :-)
>
> Would it not be easier to do something like this?  (I suck at 2-d
> drawing, even line... but this should still be understandable)

What you drew is a detailed discussion on a technique to use to group
together common part and customer specific part, and I think it is Ok to
do whatever you feel comfortable with.  It is essentially the same as my
"in real life, 'git diff >P.diff' is not how I would do this" example,
just going into more detail on what you would do to sift 'common only' vs
'specific to work branch' apart, and I think what you are doing is sane.

But if you wrote it as a draft of a document to explain how-to to new
people, I think you need to clarify a few things.

It is unclear in your description how the "common" branch progressed in
the whole process, and how the resulting history looks.  I can guess that
you meant commits marked with alphabet letters are of common kind and
numbers are of work kind, but you do not want to force readers to guess.

It also is not quite clear that you are using a temporary branch in
addition to common and work, and where in your sequence you are doing "git
checkout" to switch branches.

Re: Is there a way to exclude user-specified files or directories from participating in merges?

[Sitaram Chamarty]

On 2009-02-18, Junio C Hamano <gitster@pobox.com> wrote:
> Sitaram Chamarty <sitaramc@gmail.com> writes:
>
> > Let me explain where I'm coming from: this is very often needed when you
> > maintain customer specific branches, and the workflows in both your
> > posts in this thread so far are too complex for, err, me <sheepish grin>
> > :-)
> >
> > Would it not be easier to do something like this?  (I suck at 2-d
> > drawing, even line... but this should still be understandable)
> 
> What you drew is a detailed discussion on a technique to use to group
> together common part and customer specific part, and I think it is Ok to
> do whatever you feel comfortable with.  It is essentially the same as my
> "in real life, 'git diff >P.diff' is not how I would do this" example,
> just going into more detail on what you would do to sift 'common only' vs
> 'specific to work branch' apart, and I think what you are doing is sane.

Thanks -- I mainly wanted confirmation that one does *not*
have to do diff/patch/apply or plumbing commands to achieve
what the original poster was asking.

> But if you wrote it as a draft of a document to explain how-to to new
> people, I think you need to clarify a few things.

> It is unclear in your description how the "common" branch progressed in
> the whole process, and how the resulting history looks.  I can guess that
> you meant commits marked with alphabet letters are of common kind and
> numbers are of work kind, but you do not want to force readers to guess.

> It also is not quite clear that you are using a temporary branch in
> addition to common and work, and where in your sequence you are doing "git
> checkout" to switch branches.

I did not write it in that light then, but I will do so now,
and if you have a few minutes to critique it that would be
great.  If it's crap, sorry for the noise.

----->8-----

The following document explains how to maintain a 'common'
branch, with one or more 'customer specific' branches that
hang off of the common branch.  The inspiration was a
question about maintaining 'work' and 'home' configurations
which differ perhaps slightly from a 'common' configuration,
which leads to the same sort of situation.

The basic rules are best described by Junio in
http://permalink.gmane.org/gmane.comp.version-control.git/110489

>  - You make edits to common files only on the common branch.
>  - You merge from common to deployment, never the other way.

This is one way to follow those rules.

We use the following terminology:

'common' is the branch representing the base product.  All
regular customers get this version.

'special' is a special branch for a specific customer.  This
customer needs changes unique to his environment, which
should not be merged back into the common branch.  However,
this branch must regularly get the benefit of changes in the
mainline 'common' branch.  (This is the genesis of the two
rules above).

If you have more than one special customer the same logic
will apply for each of them separately, although if you have
too many such customers you may also want to look at "Never
merging back" (http://gitster.livejournal.com/26540.html)
for an additional tip on this.

Also, we assume the actual development and testing needs to
be done on one branch, so you will have a mix of 'common'
and 'special' changes all together.

The history looks like this in the beginning:

    -----o          <- special branch (current)
        /
    ---o            <- common branch

Before you start, make a temporary branch TEMP from
'special'; leave special where it was.

    git checkout -b TEMP special

You now make some changes for the special customer, which
also involve some 'common' changes that need to be ported
back to the common branch.  The topology now looks like
this, where A, B, C, are changes that logically belong on
'common', and 1, 2, 3, are changes that are specific to this
customer.

Notice that the 2 sets of changes are intermixed because
that is how the development happened, and that there is even
one commit where common and special changes are mixed
(perhaps you realised this only later).

    -----o--A--B1--2--3--C          <- "TEMP" branch (current)
        /
    ---o                            <- common

Meanwhile, just to make things interesting, the common
branch has also had some other, unrelated changes which you
eventually want on the special branch as well.

    -----o--A--B1--2--3--C          <- "TEMP" branch (current)
        /                                            
    ---o--X--Y                      <- common

The first thing to do is to tease the tangled commits apart
using rebase.

Using 'git rebase -i special', get the topology into this
shape.  Note that we have split the "B1" commit into B and 1
separately.  'git help rebase' has a very simple and clear
section on splitting commits, so I will not detail that
here.

    -----o--A--B--C--1--2--3        <- "TEMP" branch (current)
        /                                              
    ---o--X--Y                      <- common

At this point you may want to retest, just to be sure.

Now switch to the 'common' branch and cherry pick the
changes that belong on 'common'.  The simplest way to cherry
pick is gitk.

    git checkout common
    gitk
    # and cherry pick first A, then B, then C, in order

    -----o--A--B--C--1--2--3        <- "TEMP" branch
        /                                              
    ---o--X--Y--A'--B'--C'          <- common (current)

Now merge from 'common' to 'special'

    git checkout special
    git merge common

           A--B--C--1--2--3     <- TEMP
          /
     ----o-----------------o    <- special (current)
        /                 /
    ---o--X--Y--A'--B'--C'

Now all we need is to get commits 1, 2, and 3 onto
'special'; this is easiest done by rebasing TEMP first...

    git rebase special TEMP

                             1'--2'--3'     <- TEMP (current)
                            /
     ----o-----------------o    <- special
        /                 /
    ---o--X--Y--A'--B'--C'

...and then making special equal to TEMP

    git checkout special
    git merge TEMP

                             1'--2'--3'     <- special (current)
                            /
     ----o-----------------o
        /                 /
    ---o--X--Y--A'--B'--C'

Now you don't need TEMP anymore, and can delete it:

    git branch -d TEMP