"git overlay" - command for overlaying branches

"git overlay" - command for overlaying branches

[Oliver Bandel]

Hello,

I'm adressing the problem with files separated from the main branch(es),
which currently might be (more or less) solved with either submodules or subtrees.
I want to suggest a new command here.

As usecase-example I assume a project that has 'branch_a' and 'branch_b'
with some files may be identical, some different between them.
I assume that the (classical) way the files are handled
by belonging to those branches is intended.

Then say later I want to add more files to these branches,
but don't want to commit them in either of these existing branches.
Instead a branch 'branch_addons' is created, which solely contains
files that are used in 'branch_a' as well as 'branch_b'.

Assume there is a command 'git overlay' which overlays one branch on top of another branch,
so that the working dir contains files of both branches (with the overlayed on-top
having priority in hiding all the other files of same name in the working dir).
The files in the working dir are regarded as being members of the branches,
they are coming from initially (with the (last) overlayed branch as priority).

This, I guess, would solve the use case of the submodules/subtree problem.

Example:

Assuming only the files of the according branches are in the working dir.

branch_a has these files:
  a1.txt
  a2.txt
  common.txt


branch_b has these files:
  b1.txt
  b2.txt
  common.txt


branch_addons has these files:

  addon_1.txt
  addon_2.txt



CASE_1:
When sitting in 'branch_a' and typing 'git overlay branch_addons' I see these files:

  a1.txt
  a2.txt
  common.txt
  addon_1.txt
  addon_2.txt


CASE_2:
Would I instead be sitting in 'branch_b' and typing 'git overlay branch_addons' I see these files:

  b1.txt
  b2.txt
  common.txt
  addon_1.txt
  addon_2.txt

Changing files and then adding/committing them would apply add/commit on the branches,
where these files are comming from.

This means, in CASE_1, adding/committing changes of files from 'branch_a' would work on 'branch_a',
adding/committing changes of files from 'branch_addons' would work on 'branch_addons'.

In CASE_2, adding/committing changes of files from 'branch_b' would work on 'branch_b',
adding/committing changes of files from 'branch_addons' would work on 'branch_addons'.


If 'branch_addons' would also contain a file common.txt, then this file would be
residing in the working dir (hiding all other common.txt) and
changing/adding/committing it, would result in the 'branch_addons' common.txt
being changed.

A 'git unoverlay' or 'git unoverlay branch_addons' or 'git overlay --rm'
or 'git overlay --off' would disable the overlay, which means only the main branch
(here 'branch_a' or 'branch_b') would be checked out.

A switch of the branch (checkout/switch) might automatically un-overlay all of
the overlayed branches, or might be forbidden until an explicit un-overlay
would be done. (maybe configurable behaviour)

I assume that more than one overlay would be possible.
Of course the order of applying the overlays determenines, what branches the
files in the working dir are associated with (and checked out from).


Other use cases of course could use programming files.
The main branch might contain some interface files (*.h, *.mli, ...)
and the overlays the implementation (*.c, *.ml, ...).

Then switching the implementation for the same interface
(assuming no interface files are in the overlay-branches)
would just be one git command.


Additionally it would be good to have an easy way to move a file from one branch to another
branch (for example from one of the main branches to an overlay or vice versa).
If common.txt in the above example does not differ between 'branch_a' and
'branch_b', it might make sense to move it to maybe a branch 'branch_common', which
might be used as overlay as well (or as a base branch instead, making branhc_a and 'branch_b'
possible candidates for being overlayed).


I suggest considering such an overlay feature for git in the future.
If there are problems lurking in the background here
(for example when considering merges, rebasing and such stuff),
or if anything would fit in nicely, I can't see right now.


Ciao,
  Oliver

P.S.:

  Instead of "overlay" I first had in mind "mount" and "unmount" as name,
  but the name "overlay" is better choice, even though "unoverlay" is somewhat intricate.

Re: "git overlay" - command for overlaying branches

[Michal Suchánek]

On Fri, Nov 24, 2023 at 05:39:12PM +0100, Oliver Bandel wrote:
> Hello,
> 
> I'm adressing the problem with files separated from the main branch(es),
> which currently might be (more or less) solved with either submodules or subtrees.
> I want to suggest a new command here.
> 
> As usecase-example I assume a project that has 'branch_a' and 'branch_b'
> with some files may be identical, some different between them.
> I assume that the (classical) way the files are handled
> by belonging to those branches is intended.
> 
> Then say later I want to add more files to these branches,
> but don't want to commit them in either of these existing branches.
> Instead a branch 'branch_addons' is created, which solely contains
> files that are used in 'branch_a' as well as 'branch_b'.

Now you can merge branch_addons onto branch_a and branch_b and be done.

Unfortunately, git does not provide 'theirs' merge strategy while 'ours'
is provided - somewhat asymmetrical.

Thanks

Michal

Re: "git overlay" - command for overlaying branches

[Oliver Bandel]

Quoting  Michal Suchánek <msuchanek@suse.de> (snt: 2023-11-24 21:59 +0100 CET) (rcv: 2023-11-24 21:59 +0100 CET):
> On Fri, Nov 24, 2023 at 05:39:12PM +0100, Oliver Bandel wrote:
> > Hello,
> > 
> > I'm adressing the problem with files separated from the main branch(es),
> > which currently might be (more or less) solved with either submodules or subtrees.
> > I want to suggest a new command here.
> > 
> > As usecase-example I assume a project that has 'branch_a' and 'branch_b'
> > with some files may be identical, some different between them.
> > I assume that the (classical) way the files are handled
> > by belonging to those branches is intended.
> > 
> > Then say later I want to add more files to these branches,
> > but don't want to commit them in either of these existing branches.
> > Instead a branch 'branch_addons' is created, which solely contains
> > files that are used in 'branch_a' as well as 'branch_b'.
> 
> Now you can merge branch_addons onto branch_a and branch_b and be done.

Maybe my example or the explanation was not clear enough or stopped to early.

After a merge, I may have added files from one branch to another branch
and some files might be changed during the merge operation and this is
persistent then.

But the overlay would be a temporal situation and so no "pollution" with files from
other branches would occur and no changes of the files with same names
(some were just temporarily hidden).

In an overlay situation, all involved branches are checked out into the
working dir (the non-hidden files are checked out) at the same time.
All changes (change/add/commit) could then be done in one go (with 'git
add' and 'git commit' as in the usual way), and all commits will only
affect those branches where the changed files are comming from.
This means editing more than one branch at the same time,
committing to more than one branch at the same time,
but not merging them.

After un-overloading, the temporarily overlayed files from other
branches will be removed from the working dir by git.
No merge has been done, but changes to more than one branch might have
happened.

I hope it's more clear now, how that differs from normal merge-branches situation.

Ciao,
  Oliver

P.S.: Changing sources and tests together but only publishing the code would be
      easy this way.
      Just check out the sources, overlay the tests, do the work, commit
      the changes and after un-overlaying the sources branch has no
      "tests pollution", and no other tricks are needed then.

      Also useful, when working with LaTeX documents and you temporarily
      want to change the included extra files (or use the other layoutet
      old version together with the current one) from a different
      branch, but don't want the changes be permanently in your
      currently preferred main branch (nevertheless change the main-doc
      and the extra files in one go).

      For those people who commit their passwords or private keys into git,
      this also could increase security ;-)
      For testing the code, the secrets are just overlayed.

Re: "git overlay" - command for overlaying branches

[Michal Suchánek]

On Sat, Nov 25, 2023 at 12:51:06AM +0100, Oliver Bandel wrote:
> Quoting  Michal Suchánek <msuchanek@suse.de> (snt: 2023-11-24 21:59 +0100 CET) (rcv: 2023-11-24 21:59 +0100 CET):
> > On Fri, Nov 24, 2023 at 05:39:12PM +0100, Oliver Bandel wrote:
> > > Hello,
> > > 
> > > I'm adressing the problem with files separated from the main branch(es),
> > > which currently might be (more or less) solved with either submodules or subtrees.
> > > I want to suggest a new command here.
> > > 
> > > As usecase-example I assume a project that has 'branch_a' and 'branch_b'
> > > with some files may be identical, some different between them.
> > > I assume that the (classical) way the files are handled
> > > by belonging to those branches is intended.
> > > 
> > > Then say later I want to add more files to these branches,
> > > but don't want to commit them in either of these existing branches.
> > > Instead a branch 'branch_addons' is created, which solely contains
> > > files that are used in 'branch_a' as well as 'branch_b'.
> > 
> > Now you can merge branch_addons onto branch_a and branch_b and be done.
> 
> Maybe my example or the explanation was not clear enough or stopped to early.
> 
> After a merge, I may have added files from one branch to another branch
> and some files might be changed during the merge operation and this is
> persistent then.
> 
> But the overlay would be a temporal situation and so no "pollution" with files from
> other branches would occur and no changes of the files with same names
> (some were just temporarily hidden).
> 
> In an overlay situation, all involved branches are checked out into the
> working dir (the non-hidden files are checked out) at the same time.
> All changes (change/add/commit) could then be done in one go (with 'git
> add' and 'git commit' as in the usual way), and all commits will only
> affect those branches where the changed files are comming from.
> This means editing more than one branch at the same time,
> committing to more than one branch at the same time,
> but not merging them.
> 
> After un-overloading, the temporarily overlayed files from other
> branches will be removed from the working dir by git.
> No merge has been done, but changes to more than one branch might have
> happened.
> 
> I hope it's more clear now, how that differs from normal merge-branches situation.

It's not clear why would anyone do that, though.

> 
> Ciao,
>   Oliver
> 
> P.S.: Changing sources and tests together but only publishing the code would be
>       easy this way.

And why would you do that?

Even if you wanted the git archive has option to exclude directories so
you could publish separate releases from one branch.

>       Just check out the sources, overlay the tests, do the work, commit
>       the changes and after un-overlaying the sources branch has no
>       "tests pollution", and no other tricks are needed then.

Since you detail how 'do the work' would commit the changes to both
branches at the same time that means that specific revision of the code
is tied to specific revision of the tests, different revisions would not
work together. If the overlay branch is used with multiple different
branches it will diverge, and will be compatible only with one of them
at any time.

Merging a specific revision of the overlay branch solves this problem.
It ties the correct revision of the shared branch together with the
correct revision of the separate files.

It's the same with the interface-implementation example you gave
earlier.

>       Also useful, when working with LaTeX documents and you temporarily
>       want to change the included extra files (or use the other layoutet
>       old version together with the current one) from a different
>       branch, but don't want the changes be permanently in your
>       currently preferred main branch (nevertheless change the main-doc
>       and the extra files in one go).

Don't really understand what you are trying to do here.

>       For those people who commit their passwords or private keys into git,
>       this also could increase security ;-)
>       For testing the code, the secrets are just overlayed.

Or not put those into project git in the first place. When the secrets
are in an 'overlay branch' in the same repository they are published
anyway.

Thanks

Michal

Re: "git overlay" - command for overlaying branches

[Oliver Bandel]

What I basically want to do is:

- Working on more than one branch at the same time.
- Having all checked-out branches together in one working dir.
- When adding/committing, all checked-out files will be added/committed
  to those branches, from where they were checked out.

This way I could edit multiple branches simultaneously independently without the need of merging.
Merging can be done later on, if needed.

Possible alternatives?

For the multiple checkouts I could do a workaround and use a local bare-repo
and check out each branch of interest into a seperate directory.
I could then edit the files from the different dirs.

But for the overlaying thing, to get the files into one dir
(with the automatic hiding of equally named files from branches that
were overlayed before) it would need something like a FUSE-based
tool, so that all the branches could be blended into one working dir.

A "git overlay" would make this easier.
No extra bare-repo needed, no additional tool for blending the dirs
needed.

Ciao,
  Oliver