Feature Enhancement Idea.

Feature Enhancement Idea.

[Deon George]

Hi,

I'm not sure if this is the right place, but I thought I'd post my
idea and maybe somebody will either redirect me to the right place, or
give me that "that wont happen".

Im fairly new to GIT (wish I had discovered it long ago), and I really
like using it - great work guys/garls :)

My idea is to enhance GIT to support (I'll call it) development
"layers". The current design of GIT is that the working repository and
working directory assume that all files belong together in the same
project. I would like to see GIT go 3D and support layers, so that
files (and/or file content) can belong to multiple repositories (or
considered unique projects), even though the working tree presents all
files as if they were one.

To explain further...

Lets say, I am not the primary developer of a project, however I am a
"module"/"plugin"/"addon" contributor to a project - ie: my primary
involvement is to write additions to existing projects. (EG: I
write/support a driver in the kernel (that is not included in the base
tree), or I write an "addon" to an existing application, using that
existing applications "module" capability).

As part of me developing my "module" (or modules if I develop more
than one), I need the working tree to have the "base" and "my bits". I
want to manage both the changes I make to "my bits", and also record
any changes that need to be made to the "base" (so that "my module(s)"
will work). Normally, the changes to the "base" would be submitted
upstream (and hopefully accepted), while I would normally be
responsible for the packaging and change control of "my bits".

If the upstream chooses not to accept my contributions (ie: my changes
never appear in base), then whenever I package "my bits", GIT would
also include the base components as well.

I know I can achieve some of this by using GIT branches (I've been
doing that so far), however, GIT branches has a few limitations that I
am sure that GIT "layers" would overcome... Ultimately, I believe GIT
can handle my layers idea - and it should be possible to have multiple
layers (where multiple components could come from different GIT
repositories), however, they are all "checked out" into the one
working tree.

Thinking about this further, there would be two possible layer
scenarios (I think GIT can handle both).
* File autonomy (most cases)
This is where files (by filename) belong to different projects. EG: If
my working tree had files "a, b & c", "1, 2 & 3", and "X Y & Z". Files
"abc" could be layer one (the base), files "123" could be layer two
(dependant on base) and files XYZ could be layer three (which could be
dependant on base OR dependant on layer two).

Whenever modifications were done to any file, GIT would know which
layer owns the file and GIT processing is done as normal. Upstream
pulls from any layer should not normally generated any conflicts,
unless there are filename clashes between layers (and in this
situation the layer hierarchy should be considered authoritative, with
the conflict needed to be resolved in a lower layer.)

* Content autonomy
This is were some content in files belongs to "my work" (eg: Modifying
a Makefile to compile my work when the base is compiled). In this
situation, I may have two outcomes - I either want the changes to
flagged for upstream (to hopefully be included), or I may want to keep
the changes with my work, because I know it upstream would never
accept them (or it isnt appropriate).

In either cases, upstream pulls should be considered authoritative and
any conflicts I would need to resolve as normal commit (either wanting
them to be resubmittable for upstream, or commiting them as part of my
work.)

Like I mentioned, I can achieve some of this by the use of branches
already, however, where it comes complicated, is when:
* I commit a change to the wrong branch (and thus upstream will never
see my enhancements),
* I want to identify changes to one layer (that I went to send to
upstream for review), without including the other layers (because it
probably isnt relevant)
* I want to work on more than one layer (I need to be diligent about
pulling and merging)

An example of usage might be:
* git clone ... (or git init) -layer "A"
* git checkout -b mywork -layer "A"
* git clone ... (or git init) -layer "B" -dependson "A"
* git checkout -b mymodule -layer "B"
* add/remove/edit files
* git add file x -layer "A"
* git add file y -layer "B"
* git rm file z -layer "A"
* git commit (as usual)
* git tag "V2.8" -layer "A"
* git tag "mymodule V1.0" -layer "B"

git diff -layer "A" mywork.. would show my changes that I would want
sent upstream (without mymodule commits)
git archive -layer "B" would package up my module for distribution
(without layer "A")
git archive -layer "A" would package up my version of layer A (without
layer "B")

Could this be included as part of GITs functionality (or is it
possible already) ?

...deon

Re: Feature Enhancement Idea.

[Christian Couder]

Hi,

On Wed, Sep 23, 2009 at 8:17 AM, Deon George <deon.george@gmail.com> wrote:
> Hi,
>
> I'm not sure if this is the right place, but I thought I'd post my
> idea and maybe somebody will either redirect me to the right place, or
> give me that "that wont happen".
>
> Im fairly new to GIT (wish I had discovered it long ago), and I really
> like using it - great work guys/garls :)
>
> My idea is to enhance GIT to support (I'll call it) development
> "layers". The current design of GIT is that the working repository and
> working directory assume that all files belong together in the same
> project. I would like to see GIT go 3D and support layers, so that
> files (and/or file content) can belong to multiple repositories (or
> considered unique projects), even though the working tree presents all
> files as if they were one.

Perhaps you could have a look at "git replace" that is now in the master branch.
It could be improved to provide different "views" of a single repository.
I don't think that alone it would provide everything you want though.

Best regards,
Christian.

Re: Feature Enhancement Idea.

[Johan Herland]

On Wednesday 23 September 2009, Deon George wrote:
> Could this be included as part of GITs functionality (or is it
> possible already) ?

Have a look at git submodules ('git help submodule'), or the git-subtree 
script that has been discussed on this list a couple of times [1].

[1] http://alumnit.ca/~apenwarr/log/?m=200904#30 and 
http://github.com/apenwarr/git-subtree


...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net

Re: Feature Enhancement Idea.

[Deon George]

2009/9/23 Johan Herland <johan@herland.net>:
> Have a look at git submodules ('git help submodule'), or the git-subtree
> script that has been discussed on this list a couple of times [1].

git submodule looks like it will do a little of what I want - I'll do
some more reading on it to see exactly how it works. Thanks for the
tip.

My initial look at it seems to miss an important feature that my layer
idea would provide. It looks like git submodule assumes that
everything in a subdirectory belongs to a repository - with my layer
idea, I would want any layer to share the same directory structure and
files (or content) belonging to a distinct layer's repository.

IE:
the base might provide
plugins
plugins/README
modules
modules/README
lib/common.php

and a layer might provide
plugins/a.php
modules/a.php
lib/a.php

another layer might provide
plugins/b.php
modules/b.php
lib/b.php

If I was a C developer, I'd have a go at creating it - but I'm just a
php developer :)
...deon

Re: Feature Enhancement Idea.

[Ciprian Dorin, Craciun]

On Wed, Sep 23, 2009 at 2:12 PM, Deon George <deon.george@gmail.com> wrote:
> 2009/9/23 Johan Herland <johan@herland.net>:
>> Have a look at git submodules ('git help submodule'), or the git-subtree
>> script that has been discussed on this list a couple of times [1].
>
> git submodule looks like it will do a little of what I want - I'll do
> some more reading on it to see exactly how it works. Thanks for the
> tip.
>
> My initial look at it seems to miss an important feature that my layer
> idea would provide. It looks like git submodule assumes that
> everything in a subdirectory belongs to a repository - with my layer
> idea, I would want any layer to share the same directory structure and
> files (or content) belonging to a distinct layer's repository.
>
> IE:
> the base might provide
> plugins
> plugins/README
> modules
> modules/README
> lib/common.php
>
> and a layer might provide
> plugins/a.php
> modules/a.php
> lib/a.php
>
> another layer might provide
> plugins/b.php
> modules/b.php
> lib/b.php
>
> If I was a C developer, I'd have a go at creating it - but I'm just a
> php developer :)
> ...deon


    Practically you want something like unionfs [1] but for git. Right?

    But probably you could settle for something like Hg Queues [2]. Is
there similar for Git?

    Ciprian Craciun.

    [1] http://en.wikipedia.org/wiki/UnionFS
    [2] http://mercurial.selenic.com/wiki/MqExtension

Re: Feature Enhancement Idea.

[Junio C Hamano]

Deon George <deon.george@gmail.com> writes:

> Like I mentioned, I can achieve some of this by the use of branches
> already, however, where it comes complicated, is when:
> * I commit a change to the wrong branch (and thus upstream will never
> see my enhancements),
> * I want to identify changes to one layer (that I went to send to
> upstream for review), without including the other layers (because it
> probably isnt relevant)
> * I want to work on more than one layer (I need to be diligent about
> pulling and merging)

I think you are getting ahead of yourself by coming up with "layers" as a
solution, without really spelling out and then thinking through what
problem you are trying to solve.

I think the primary scenario your itch comes from is (I am writing this
down to make sure I understand where you are trying to go):

 - you would want to build your changes on top of somebody else's code;

 - while doing so it is often more convenient if you do not have to think
   about which pieces of changes should go to upstream, and which other
   pieces of changes should stay local;

 - git allows you to do this with topic branch workflow, with selective
   adding with "add -p", stashing and branch switching.

 - You can however misidentify which bits should go to upstream and commit
   to a wrong branch.  You want to reduce the chance of this mistake.

Am I following you Ok so far?

Now, how does "layers" (I understand by this word you mean the concept of
"layering a worktree from one branch on top of layering a worktree from
another") solve that problem?  If you come up with a nice way to identify,
and have the tool help you identify, "these bits belong to that layer",
wouldn't that approach also apply to existing topic branch workflow by
having the same logic of that magic tool to help you say "these bits
should go to that topic branch"?

When the granularity of the change is per-file (your "autonomous" case),
"git checkout" to switch back to your topic would already solve half of
that issue.

Suppose you have mainline (origin/master) and your customization topic
(custom), and file F is something that does not and will never exist in
the mainline.  You use your master branch as your integration testing
branch.  With traditional topic branch workflow, you would

    $ git checkout custom
    ... work on F and other files, test the changes to your satisfaction
    $ git commit -m 'I am satisfied on my custom work'
    $ git checkout master
    $ git rebase origin/master ;# keep up with the upstream
    $ git cherry-pick ... ;# some commits in custom that should go to upstream
    $ git merge custom ;# test merge to be thrown away!

    ... now you are on the integrated result in 'master'.
    ... test, notice breakages in F, and edit it to improve it
    ... the changes to F belongs to 'custom', not for upstream!
    $ git checkout custom ;# this will take the changes to F with you
    $ git commit -m 'I should have done this to F instead' F

    ... go back to do more work
    $ git checkout master
    ... loop forever ;-)

and after you are satisfied, you would reset the merge from custom away
and offer the master that is combination of what was in the origin/master
plus the "upstream worthy" bits you cherry-picked ones from your custom
branch.

When the granularity is sub-file, "checkout -m custom" or "stash, checkout
custom, then unstash" would help carrying the changes across branches.
You would need to identify which changes you would want to commit to
"custom" and which changes you would want to give to upstream by making a
commit on "master" when you go back there again.  The middle part of the
above workflow will be slightly modified for a file G that is shared
between origin/master and custom branches, perhaps something like:

    ... now you are on the integrated result in 'master'.
    ... test, notice breakages in G, and edit it to improve it
    ... the changes to G belongs to 'custom', not for upstream!
    $ git checkout -m custom ;# this will take the changes to G with you
    $ git add -p G ;# pick changes that only belong to custom
    $ git commit -m 'I should have done this to G instead'

If the "layers" logic somehow can help you automate the process of sifting
your changes into these two categories (perhaps it may scan the file and
knows which function belongs to "custom" alone, I dunno and care about the
details at this level of discussion), wouldn't that same logic help you
the same way?

Past attempts that made into the toolchest are "add -p" (and "stash -p"
that uses the same mechanism) which is interactive and not automated.
Maybe you can build some logic to automate it further (e.g. "changes to
this function should automatically be excluded from "git add" while on
'master', but should automatically be included in "git add" while on
'custom'), and it may turn out be a good addition to the toolchest.

It also may make a good addition to the toolchest to have a tool to
further simplify the branch switching explicitly initiated by the end user
in the above illustrations.  In some cases, such as the "autonomous" case
above, you do not _have_ to go back to custom branch from the work-flow
point of view (you would be committing a totally untested work to custom
branch, but as long as you will revisit and retest the changes in its
context, it is not a major crime at all).  It would be useful if you can
say, while still on 'master' branch, "I just made this fix to file F that
belongs to 'custom'; commit that change alone to 'custom' branch".
Written as a shell script, roughly, it would look something like:

    #!/bin/sh
    target_branch="$1"
    base=$(git rev-parse --verify "$target_branch") || exit
    shift ;# all others are paths
    GIT_INDEX_FILE=.git/temp
    export GIT_INDEX_FILE
    git read-tree -m "$1"
    git add "$@"
    c=$(echo "side commit" | git commit-tree $(git write-tree) -p "$base")
    git update-ref "refs/heads/$target_branch" "$c"

But I do not see how the concept of "layering a worktree from one branch
on top of layering a worktree from another", which is the implication I
get from the word "git layers", helps any of that.  It seems an orthogonal
concept that we can do without to solve the problem you are describing.

Re: Feature Enhancement Idea.

[Johan Herland]

On Wednesday 23 September 2009, Ciprian Dorin, Craciun wrote:
>     Practically you want something like unionfs [1] but for git. Right?
> 
>     But probably you could settle for something like Hg Queues [2]. Is
> there similar for Git?

TopGit, StGit, Guilt?

Not sure they (or Hg Queues) solve Deon's problem though...


...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net

Re: Feature Enhancement Idea.

[Deon George]

Junio C Hamano wrote:

> I think the primary scenario your itch comes from is (I am writing this
> down to make sure I understand where you are trying to go):
>
>  - you would want to build your changes on top of somebody else's code;

Yes - more correctly, I want to enhance an application, where I am (or
different person - it doesnt matter) is responsible for that code.
Thus I want to source code manage my enhancement, and (for whatever
reason), the other components of the application dont want my part in
their "base".

>  - while doing so it is often more convenient if you do not have to think
>    about which pieces of changes should go to upstream, and which other
>    pieces of changes should stay local;

Not strictly true, but yes - I am aware that my modules might be
completely autonomous to the base code (thus I source control manage
everything that my module requires), or I discover bugs in the base
code, that I believe upstream should have. There may also be a case,
where I want to modify some code in the base, but I know that it
shouldnt never be included in the base code (and thus I would need my
commit to be stored in my SCM, not the base's SCM) - eg: Modifying a
Makefile...

So while I would commit to a layer, I should make a conscious decision
of where the commit should go (I own the commit work, or I want to
send the commit upstream as a contribution enhancement/fix) - because
the default assumption may be wrong.

>
>  - git allows you to do this with topic branch workflow, with selective
>    adding with "add -p", stashing and branch switching.

I dont know about topic branches "git add -p"? (I havent got that
advanced yet, looks like I need to do some more research on the
features of GIT :)

> Am I following you Ok so far?

Yes, but one more important criteria - my working tree is made up of
files (or content) managed in more than once source control management
repository - I use an SCM to manage and track changes, and I use
multiple repository because any one owner of one repository may not
want the code from another as part of their "base". IE: It could be I
write a custom driver for a widget device, that Linus doesnt want that
code in the base kernel (for whatever reason) and I maintain that part
of the code, or (in my case), I want to maintain (and encourage others
to contribute) to a module (or set of modules) for an existing web
application framework.

Additionally, each layer should be able to switch between its own
branches (ie: My driver 1.0 works well with kernel "N", now Linus
publishes "N+1", I want to switch that (base) layer to "N+1" and
test/fix my module to work with that release - and then commit/tag it
as such - ie: my driver (now 1.1) works with "N+1"). Further a bug is
now found in my 1.0 driver, so I want to switch back the kernel layer
to "N", and my driver layer to 1.0, make the fix, and release
1.0.1)... Kinda get the idea Im working towards...?

Additionally, it should not necessarily be "one base" and "my layer",
it may be multiple layers (and hence multiple repositories), but all
layers make up an application with multiple features, each
individually source control managed. Each layer would have a
dependency to another (otherwise they could be developed in their own
repository entirely - and the dependency may be needed to be know for
merge conflict resolution (where do the conflicts need to be
commited), when pulling from the upstream owners of that layer)

Thus, when I package up my module - it shouldnt include the base
(since the base may already be installed), and the base developers
want to keep their application more like a framework - so they are
responsible for the core workings of the code, and the module
developers are responsible for their components.

To achieve what I want today, I think I would need to have:

master
master-mycontribs (branched from master, and merged from master)
mymodule (branched from master-mycontribs and merge from there - this
is pushed to my SCM repository)
mymodule-working (where I test and commit - and merge back to my module)

(this comes stuck, when I want to switch master to a previous version...)

And thus, the diff between "master" and "master-mycontribs", are my
contributions to go upstream. The diff between "master" and "mymodule"
is what I would package to make my module as an enhancement to anyone
already using "master" (at release "N").

If upstream accepts my enhancements, then "master" =
"master-mycontribs" (and master is now release "N+1"), and the next
release of my module (1.2) would still be the diff between
"master-mycontribs" and "mymodule" (but would be excluding my code
that I added to master-mycontribs (since master now equals that),
because the end user who is already running N+1 has those
enhancements.

I'll research git "topics" and see if that does what I am dreaming of :)

...deon

Re: Feature Enhancement Idea.

[Jakub Narebski]

Deon George <deon.george@gmail.com> writes:

> I'm not sure if this is the right place, but I thought I'd post my
> idea and maybe somebody will either redirect me to the right place, or
> give me that "that won't happen".
> 
> Im fairly new to GIT (wish I had discovered it long ago), and I really
> like using it - great work guys/garls :)
> 
> My idea is to enhance GIT to support (I'll call it) development
> "layers". The current design of GIT is that the working repository and
> working directory assume that all files belong together in the same
> project. I would like to see GIT go 3D and support layers, so that
> files (and/or file content) can belong to multiple repositories (or
> considered unique projects), even though the working tree presents all
> files as if they were one.

[cut very long description]

> Could this be included as part of GITs functionality (or is it
> possible already) ?

First, I assume there that you do not allow for the same file to
belong to different repositories.

Second, if all parts that you want to belong to other repository are
in separate subdirectories, and all files in those subdirectories
belong to this other repository, you can try either submodules 
(git-submodule), or subtree (subtree merge, or third-party git-subtree
helper).  Note also that this assume that you want to have 'master'
repository which indirectly or directly has al the files.


Third, if the above isn't what you want, then you can manually
intermingle working directories of different git repositories
(probably requiring decouplig of bare git repository (git-dir)
from working area (work-tree)).  Git repository know what files
it tracks, so you would only need to take care to ignore files
that belong to other repositories.

If it is to manual for you, and to error prone, you are welcome
to come up with set of scripts implementing "layers" feature you
want.  That is how initial version of submodule feature was done...

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Feature Enhancement Idea.

[Deon George]

Jakub Narebski wrote:

> First, I assume there that you do not allow for the same file to
> belong to different repositories.

Well, no, I cant see why a file cannot belong to both repositories.
I'm sure it would be easier if a file did belong to a unique
repository, but given that GIT operates at a content layer (from what
I've read anyway), I cant see why it couldnt belong to more than one.

> you can try either submodules
> (git-submodule), or subtree (subtree merge, or third-party git-subtree
> helper).

I had a quick look at submodule - I dont think it helps me anyway,
since I want the flexibility for files to belong to many
sub-directories - not all under 1 sub directory hierarchy.

> Third, if the above isn't what you want, then you can manually
> intermingle working directories of different git repositories
> (probably requiring decouplig of bare git repository (git-dir)
> from working area (work-tree)).

Ahh, now this sounds like it might be what I want to do - I think I'll
explore this. I can see that it would provide file level autonomy
only, but as a starting point I think it will help heaps...

I'll work on it as time permits and come up with some scripts to
provide what I am after (need to learn more on the workings of GIT
first - everyday I learn a new feature :)

Thanks for your idea...

...deon

Re: Feature Enhancement Idea.

[Eric Raible]

Deon George <deon.george <at> gmail.com> writes:

> > Third, if the above isn't what you want, then you can manually
> > intermingle working directories of different git repositories
> > (probably requiring decouplig of bare git repository (git-dir)
> > from working area (work-tree)).
> 
> Ahh, now this sounds like it might be what I want to do - I think I'll
> explore this. I can see that it would provide file level autonomy
> only, but as a starting point I think it will help heaps...

Proof of concept:

export GIT_WORK_TREE=.

git --git-dir=.git1 init
git --git-dir=.git1 add view1-file
git --git-dir=.git1 commit -m"view1 initial"

git --git-dir=.git2 init
git --git-dir=.git2 add view1-file
git --git-dir=.git2 commit -m"view2 initial"

git --git-dir=.git1 log
git --git-dir=.git2 log