"groups of files" in Git?

"groups of files" in Git?

[Nikolay Shustov]

Hi,
I have been recently struggling with migrating my development workflow
from Perforce to Git, all because of the following thing:

I have to work on several features in the same code tree parallel, in
the same Perforce workspace. The major reason why I cannot work on one
feature then on another is just because I have to make sure that the
changes in the related areas of the product play together well.

With Perforce, I can have multiple changelists opened, that group the
changed files as needed.

With Git I cannot seem to finding the possibility to figure out how to
achieve the same result. And the problem is that putting change sets
on different Git branches (or workdirs, or whatever Git offers that
makes the changes to be NOT in the same source tree) is not a viable
option from me as I would have to re-build code as I re-integrate the
changes between the branches (or whatever changes separation Git
feature is used).
Build takes time and resources and considering that I have to do it on
multiple platforms (I do cross-platform development) it really
denominates the option of not having multiple changes in the same code
tree.

Am I ignorant about some Git feature/way of using Git that would help?
Is it worth considering adding to Git a feature like "group of files"
that would offer some virtutal grouping of the locally changed files
in the checked-out branch?

Thanks in advance,
- Nikolay

Re: "groups of files" in Git?

[Stefan Beller]

On Tue, Jul 11, 2017 at 8:45 AM, Nikolay Shustov
<nikolay.shustov@gmail.com> wrote:
> Hi,
> I have been recently struggling with migrating my development workflow
> from Perforce to Git, all because of the following thing:
>
> I have to work on several features in the same code tree parallel, in
> the same Perforce workspace. The major reason why I cannot work on one
> feature then on another is just because I have to make sure that the
> changes in the related areas of the product play together well.

So in that case the features are not independent, but related to each other?
In that case you want to have these things in the same working tree as
well as in the same branch.

Take a look at git.git itself, for example:

    git clone git://github.com/git/git
    git log --oneline --graph

You will see a lot of "Merge X into master/maint" commits, but then
you may want to dive into each feature by:

    git log --oneline e83e71c5e1

for example and then you'll see lots of commits (that were developed
in the same branch), but that are closely related. However they are
different enough to be in different commits. (different features, as
I understand)

> With Perforce, I can have multiple changelists opened, that group the
> changed files as needed.
>
> With Git I cannot seem to finding the possibility to figure out how to
> achieve the same result. And the problem is that putting change sets
> on different Git branches (or workdirs, or whatever Git offers that
> makes the changes to be NOT in the same source tree) is not a viable
> option from me as I would have to re-build code as I re-integrate the
> changes between the branches (or whatever changes separation Git
> feature is used).

you would merge the branches and then run the tests/integration. Yes that
seems cumbersome.

> Build takes time and resources and considering that I have to do it on
> multiple platforms (I do cross-platform development) it really
> denominates the option of not having multiple changes in the same code
> tree.
>
> Am I ignorant about some Git feature/way of using Git that would help?
> Is it worth considering adding to Git a feature like "group of files"
> that would offer some virtutal grouping of the locally changed files
> in the checked-out branch?

The way of Git is that a commit (snapshot) by definition describes a
set of files (The set of all files in the project). So If you need two features
there at the same time, you probably want it in the same commit.

If they are different enough such that you could have them independently,
but really want to test them together, your testing may need to become
more elaborate (test a merge of all feature branches) I would think.

>
> Thanks in advance,
> - Nikolay

Re: "groups of files" in Git?

[Nikolay Shustov]

> The way of Git is that a commit (snapshot) by definition describes a
> set of files (The set of all files in the project). So If you need two features
> there at the same time, you probably want it in the same commit.

Thank you, but if I wanted these two features to be in the same
commit, I would have no reasons to see them as two distinctive groups.
I mean, groups of uncommitted files.

The general problem of not having multiple features in the same code
tree is the cost of doing multiple builds and integration testing
runs.
Now I imagine there could be workaround of having two features
developed at different branches and then merging them into 3rd branch
for building/testing; however this introduces overhead of maintaining
at lest two code trees: one for "dirty changes" where I do the code
changes that are not guaranteed to be even build-able and another
"build/test" code tree. Plus merging the changes from one to another.
A bit too much, IMHO.

On Tue, Jul 11, 2017 at 1:18 PM, Stefan Beller <sbeller@google.com> wrote:
> On Tue, Jul 11, 2017 at 8:45 AM, Nikolay Shustov
> <nikolay.shustov@gmail.com> wrote:
>> Hi,
>> I have been recently struggling with migrating my development workflow
>> from Perforce to Git, all because of the following thing:
>>
>> I have to work on several features in the same code tree parallel, in
>> the same Perforce workspace. The major reason why I cannot work on one
>> feature then on another is just because I have to make sure that the
>> changes in the related areas of the product play together well.
>
> So in that case the features are not independent, but related to each other?
> In that case you want to have these things in the same working tree as
> well as in the same branch.
>
> Take a look at git.git itself, for example:
>
>     git clone git://github.com/git/git
>     git log --oneline --graph
>
> You will see a lot of "Merge X into master/maint" commits, but then
> you may want to dive into each feature by:
>
>     git log --oneline e83e71c5e1
>
> for example and then you'll see lots of commits (that were developed
> in the same branch), but that are closely related. However they are
> different enough to be in different commits. (different features, as
> I understand)
>
>> With Perforce, I can have multiple changelists opened, that group the
>> changed files as needed.
>>
>> With Git I cannot seem to finding the possibility to figure out how to
>> achieve the same result. And the problem is that putting change sets
>> on different Git branches (or workdirs, or whatever Git offers that
>> makes the changes to be NOT in the same source tree) is not a viable
>> option from me as I would have to re-build code as I re-integrate the
>> changes between the branches (or whatever changes separation Git
>> feature is used).
>
> you would merge the branches and then run the tests/integration. Yes that
> seems cumbersome.
>
>> Build takes time and resources and considering that I have to do it on
>> multiple platforms (I do cross-platform development) it really
>> denominates the option of not having multiple changes in the same code
>> tree.
>>
>> Am I ignorant about some Git feature/way of using Git that would help?
>> Is it worth considering adding to Git a feature like "group of files"
>> that would offer some virtutal grouping of the locally changed files
>> in the checked-out branch?
>
> The way of Git is that a commit (snapshot) by definition describes a
> set of files (The set of all files in the project). So If you need two features
> there at the same time, you probably want it in the same commit.
>
> If they are different enough such that you could have them independently,
> but really want to test them together, your testing may need to become
> more elaborate (test a merge of all feature branches) I would think.
>
>>
>> Thanks in advance,
>> - Nikolay

Re: "groups of files" in Git?

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> On Tue, Jul 11, 2017 at 8:45 AM, Nikolay Shustov
>
>> With Git I cannot seem to finding the possibility to figure out how to
>> achieve the same result. And the problem is that putting change sets
>> on different Git branches (or workdirs, or whatever Git offers that
>> makes the changes to be NOT in the same source tree) is not a viable
>> option from me as I would have to re-build code as I re-integrate the
>> changes between the branches (or whatever changes separation Git
>> feature is used).
>
> you would merge the branches and then run the tests/integration. Yes that
> seems cumbersome.

Sometimes the need to make trial merge for testing cannot be avoided
and having branches for separate topics is the only sensible
approach, at least in the Git world.

Imagine your project has two components that are interrelated, say,
the server and the client, that have to work well with each other.
In addition, you want to make sure your updated server works well
with existing clients, and vice versa.

One way that naturally maps this scenario to the development
workflow is to have a server-update topic and a client-update topic
branches, and separate changes to update each side with their own
commits:

             s---s---S    server-update topic
            /
    ---o---o----o----M    mainline
            \
             c---c---C    client-update topic

And during the development of these *-update topics, you try three
merges:

 (1) Merge S to the mainline M and test the whole thing, to make sure
     that existing client will still be able to talk with the
     updated server.

 (2) Merge C to the mainline M and test the whole thing, to make
     sure that updated clients will still be able to talk with the
     existing server.

 (3) Merge both S and C to the mainline M and test the whole thing,
     to make sure the updated ones talk to each other.

If there is no significant development going on on the mainline in
the meantime, (1) and (2) can be done by trying out S and C alone
without making a trial merge with M.  The same for (3)---it can be
just a trial merge between S and C without updates that happened on
the mainline.

I'd love to hear from folks in Perforce or other world how they
address this scenario with their system.

Re: "groups of files" in Git?

[Nikolay Shustov]

For me the roadblock for multiple iterations through merging of the
different parts (S, C, then C+S) is the time that will be spent on
rebuilding the mainline.
That's why I would like to have C+S in the same source tree then run
tests for S, tests for C (if they can be run standalone) and C+S
tests, then tests for whatever other pieces may be affected. (As I
mentioned, there are more layers than client + server in my situation,
e.g. client  + transport + server).

I am not really try to ignite the holy war between Perforce and Git
(and why would one???), but if you are interested in the answer on how
you'd do your scenario in Perforce, it would be: "use shelved
changelists".
In Perforce, you could "shelve" the changelist, similar to "stash" in
Git, but the difference is that the Perforce shelved changes are
accessible across clients. I.e. the other developer can "unshelve"
these pending changes to its sandbox (to the same or the different
branch) so that sandbox would get the pending changes as well. That
would be like the developer made these changes himself. Whatever
automated/manual process is involved, it is typical to run "a trial
build/tests" on shelved changelist (i.e. uncommitted yet files) to
verify the quality of changes.
Git achieves the same through the ease of manipulation with branches
and I like the way it does it much more.

My question was about how to robustly handle "multiple pending
commits" which in Perforce are represented by concept of pending
changelists.

On Thu, Jul 13, 2017 at 2:22 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> On Tue, Jul 11, 2017 at 8:45 AM, Nikolay Shustov
>>
>>> With Git I cannot seem to finding the possibility to figure out how to
>>> achieve the same result. And the problem is that putting change sets
>>> on different Git branches (or workdirs, or whatever Git offers that
>>> makes the changes to be NOT in the same source tree) is not a viable
>>> option from me as I would have to re-build code as I re-integrate the
>>> changes between the branches (or whatever changes separation Git
>>> feature is used).
>>
>> you would merge the branches and then run the tests/integration. Yes that
>> seems cumbersome.
>
> Sometimes the need to make trial merge for testing cannot be avoided
> and having branches for separate topics is the only sensible
> approach, at least in the Git world.
>
> Imagine your project has two components that are interrelated, say,
> the server and the client, that have to work well with each other.
> In addition, you want to make sure your updated server works well
> with existing clients, and vice versa.
>
> One way that naturally maps this scenario to the development
> workflow is to have a server-update topic and a client-update topic
> branches, and separate changes to update each side with their own
> commits:
>
>              s---s---S    server-update topic
>             /
>     ---o---o----o----M    mainline
>             \
>              c---c---C    client-update topic
>
> And during the development of these *-update topics, you try three
> merges:
>
>  (1) Merge S to the mainline M and test the whole thing, to make sure
>      that existing client will still be able to talk with the
>      updated server.
>
>  (2) Merge C to the mainline M and test the whole thing, to make
>      sure that updated clients will still be able to talk with the
>      existing server.
>
>  (3) Merge both S and C to the mainline M and test the whole thing,
>      to make sure the updated ones talk to each other.
>
> If there is no significant development going on on the mainline in
> the meantime, (1) and (2) can be done by trying out S and C alone
> without making a trial merge with M.  The same for (3)---it can be
> just a trial merge between S and C without updates that happened on
> the mainline.
>
> I'd love to hear from folks in Perforce or other world how they
> address this scenario with their system.

Re: "groups of files" in Git?

[Junio C Hamano]

Nikolay Shustov <nikolay.shustov@gmail.com> writes:

> I am not really try to ignite the holy war between Perforce and Git
> (and why would one???), but if you are interested in the answer on how
> you'd do your scenario in Perforce, it would be: "use shelved
> changelists".

Oh, that was not my intention, either.  My interest was to see if
there is a good solution that we could steal from other world.

> In Perforce, you could "shelve" the changelist, similar to "stash" in
> Git, but the difference is that the Perforce shelved changes are
> accessible across clients. I.e. the other developer can "unshelve"
> these pending changes to its sandbox (to the same or the different
> branch) so that sandbox would get the pending changes as well. That
> would be like the developer made these changes himself. Whatever
> automated/manual process is involved, it is typical to run "a trial
> build/tests" on shelved changelist (i.e. uncommitted yet files) to
> verify the quality of changes.
> Git achieves the same through the ease of manipulation with branches
> and I like the way it does it much more.

Thanks.  Shelving and letting others unshelve is like keeping the
changes in separate branches and privately share them among
developers, so they sound pretty much equivalent features to me.

> My question was about how to robustly handle "multiple pending
> commits" which in Perforce are represented by concept of pending
> changelists.

And in Git, they are represented by concept of commits that are not
yet pushed out to the public repository to become the final history
carved in stone.

Re: "groups of files" in Git?

[Igor Djordjevic]

On 13/07/2017 23:20, Junio C Hamano wrote:
> Nikolay Shustov <nikolay.shustov@gmail.com> writes:
>> My question was about how to robustly handle "multiple pending
>> commits" which in Perforce are represented by concept of pending
>> changelists.
> 
> And in Git, they are represented by concept of commits that are not
> yet pushed out to the public repository to become the final history
> carved in stone.

If I may, I don`t think "multiple pending commits" is the issue here 
(as that is indeed what a private branch is), but more something like 
"multiple branches pending/live merge branch", or something.

To illustrate, let`s say this is our starting position:

(1)      o---o---o (featureA)
        /         \
    ---o---o---o---M (master, HEAD)
        \         /
         o---o---o (featureB)


We`re currently on commit "M", being a merge commit between our 
"master" and two feature branches.

Now, what seems lacking, while still possible through a series of 
steps, is an easy (single step) way to modify current state and 
commit the change to the _feature branch_, while still being on the 
"master" branch, still having everything merged in.

So after I make a "featureA" related change while on "M", to be able 
to issue a single command, for example:
 
    $ git commit --branch=featureA

... or:
 
    $ git commit -b featureA 
 
..., where "featureA" would need to be one of the parents of the 
current commit we are at (commit "M", in our case), and get a 
situation like this:

(2)      o---o---o---A (featureA)
        /             \
    ---o---o---o-------M' (master, HEAD)
        \             /
         o---o---o---/ (featureB)


Here, "A" is a new commit/change I`ve just made (while still being on 
the "master" branch), and it is automatically commited to related 
"featureA" branch, with merge commit "M" now recreated into "M'" to 
hold the new "featureA" commit "A" as well.

I guess it would be a kind of alias to doing:

    $ git checkout featureA
    $ git add ...
    $ git commit
    $ git checkout master
    $ git reset --hard HEAD^
    $ git merge featureA featureB

... or something, where last merge step would need to remember 
previous merge commit "M" parent branches and merge them again to 
produce an updated "M'" merge commit.

In the same manner, it should be possible to drop a commit from the 
feature branch in a single step, for example returning to the state 
as shown in (1), or even "port" it from one branch to the other, like
this (without a need for it to be the last commit, even):

(3)      o---o---o---\ (featureA)
        /             \
    ---o---o---o-------M' (master, HEAD)
        \             /
         o---o---A'--o (featureB)


Something like "rebase on steroids", lol, keeping the HEAD where it 
is, and its merge commit beneath updated.

This indeed seems similar to Mercurial`s patch "queues", except being 
much better as everything is still version controlled at all times, 
no additional tools needed to version control the patches (unless 
that`s already been addressed in Mercurial as well, dunno).
 
And it still seems to be following Git`s "multiple commits per 
feature, single feature per branch" spirit, just allowing for 
easier/faster branch integration testing.

p.s. Even if my short sample might be flawed in one way or the other, 
it should show the essence of the functionality we`re discussing 
here, I think.

Regards,
Buga

Re: "groups of files" in Git?

[Igor Djordjevic]

Just a small update/fixup:

On 14/07/2017 00:39, Igor Djordjevic wrote:
> I guess it would be a kind of alias to doing:
> 
>     $ git checkout featureA
>     $ git add ...
>     $ git commit
>     $ git checkout master
>     $ git reset --hard HEAD^
>     $ git merge featureA featureB
> 

This should, in fact, be:    
    
    $ git checkout featureA
    $ git commit
    $ git checkout master
    $ git reset --hard HEAD^
    $ git merge <HEAD@{1} parents>
    
(removed "git add" step, as that is needed for proposed single step 
solution as well, as a usual step preceding the commit; also replaced 
concrete branch names in the last step with a more generic 
description, better communicating real intent)

> In the same manner, it should be possible to drop a commit from the 
> feature branch in a single step, for example returning to the state 
> as shown in (1), or even "port" it from one branch to the other, like
> this (without a need for it to be the last commit, even):
> 
> (3)      o---o---o---\ (featureA)
>         /             \
>     ---o---o---o-------M' (master, HEAD)
>         \             /
>          o---o---A'--o (featureB)

Here, the diagram should look like this:

(3)      o---o---o---\ (featureA)
        /             \
    ---o---o---o-------M'' (master, HEAD)
        \             /
         o---o---A''-o (featureB)

(replaced leftover M' from the previous diagram with M'' to show it`s 
yet another (updated) merge commit, different from both M and M' in 
terms of SHA1, yet the contents would probably, but not necessarily, 
be the same for all three; same for leftover A', replaced with A'')

Regards,
Buga

Re: "groups of files" in Git?

[Igor Djordjevic]

Eh, yet another one, sorry:

On 14/07/2017 01:32, Igor Djordjevic wrote:
>     
>     $ git checkout featureA
>     $ git commit
>     $ git checkout master
>     $ git reset --hard HEAD^
>     $ git merge <HEAD@{1} parents>

The last line should stress <HEAD@{1} *parent branches*>, as we`re 
not merging exact parent commits the previous merge commit was made 
of, but updated tips of the branches the previous merge commit was 
made of... or something along those lines :)

RE: "groups of files" in Git?

[Randall S. Becker]

-----Original Message-----
On July 11, 2017 11:45 AM Nikolay Shustov wrote:
>I have been recently struggling with migrating my development workflow from Perforce to Git, all because of the following thing:
>I have to work on several features in the same code tree parallel, in the same Perforce workspace. The major reason why I cannot
>work on one feature then on another is just because I have to make sure that the changes in the related areas of the product play together well.
>With Perforce, I can have multiple changelists opened, that group the changed files as needed.

>With Git I cannot seem to finding the possibility to figure out how to achieve the same result. And the problem is
>that putting change sets on different Git branches (or workdirs, or whatever Git offers that makes the changes to
>be NOT in the same source tree) is not a viable option from me as I would have to re-build code as I re-integrate
>the changes between the branches (or whatever changes separation Git feature is used).
>Build takes time and resources and considering that I have to do it on multiple platforms (I do cross-platform development) it really denominates the option of not having multiple changes in the same code tree.

Change sets are core git functionality. When you commit, you commit a group of changes across multiple files, not single file at a time, like most legacy SCM systems. Individual features are managed typically managed using topic branches that can be switched (using checkout) rapidly, which in your case will cause a localized rebuild based on what files were swapped.

If you need something slightly different than topic branches, do multiple clones off a base integration branch. This will give you multiple working source trees. Switch each clone to its own branch and work with them locally. If you need to move changes from one branch to another, commit, push on one branch, and pull merge to the other branch.

You could also use stash to accomplish similar things, but I wouldn't.

Cheers,
Randall

Re: "groups of files" in Git?

[Junio C Hamano]

Nikolay Shustov <nikolay.shustov@gmail.com> writes:

> I have to work on several features in the same code tree parallel, in
> the same Perforce workspace. The major reason why I cannot work on one
> feature then on another is just because I have to make sure that the
> changes in the related areas of the product play together well.
>
> With Perforce, I can have multiple changelists opened, that group the
> changed files as needed.
>
> With Git I cannot seem to finding the possibility to figure out how to
> achieve the same result. And the problem is that putting change sets
> on different Git branches (or workdirs, or whatever Git offers that
> makes the changes to be NOT in the same source tree) is not a viable
> option from me ...

Naturally.  If these separate changes need to work together, it is
way too inconvenient if these changes do not appear in a single
unified working tree to be built and tested.

> Is it worth considering adding to Git a feature like "group of files"
> that would offer some virtutal grouping of the locally changed files
> in the checked-out branch?

Let's step back and let me make sure if I understand you correctly.
You want to work on a system with two distinct areas (say, the
frontend and the backend), that have to work together, but you want
to make two commits, one for each area.  You make changes for both
areas in your working tree, build and test them to make sure the
whole thing works well together, and at the end, you make two
commits.  

In your real project, you may be doing more than two areas and more
than two commits, but is the above a good degenerative case that
shows the basic idea?  If not, then please disregard all of the
following.

You can make partial commits in Git.  In the simplest case, you may
have two separate files backend.py and frontend.py, you make edits
to both files and then make two commits:

	$ git commit backend.py
	$ git commit frontend.py

Changes to some files may contain both changes for the backend and
for the frontend that does not allow you to separate commits at file
boundary, and Git even lets you handle such a case.  If you have the
third file in addition to the above two, called common.py, you could
instead

	$ git add backend.py
	$ git add -p common.py

to prepare the index to contain only the changes for the backend
part ("add -p" lets you interactively pick and choose the hunks
relevant to the backend part), and conclude the commit for the
backend part with

	$ git commit ;# no paths arguments

and then when all the remaining changes are for the frontend part,
you can follow it with

	$ git commit -a

to make another commit for the frontend part.

A short answer to your question, provided if I understood you
correctly, is "no, there is no way to say 'backend.py, backend-2.py,
...' are the backend things and call it a filegroup", accompanied by
"a filegroup would only be effective when changes align with file
boundary".

And if your response is "but most of the time changes align with
file boundary", a counter-response is "and most of the time changes
align with directory boundary (in well structured project, at
least), so you can do 'git commit backend/' for all the backend part
without having to name all the paths anyway".

There is an experimental "attributes-limited pathspec" feature in
recent versions of Git, which lets you assign arbitrary sets of
labels to each paths, and using that you may be able to do

	$ git commit ':(attr:filegroup=backend).'

and I suspect that would be the closest thing you would want (read
about 'pathspec' in 'git help glossary')

Re: "groups of files" in Git?

[Nikolay Shustov]

Thank you for the explanation. The example about backend and frontend
is relevant even though I normally have to deal with more layers at
the same time.

However, in my case I have the thing that you have already tried to
address, partially: the changes always align with file boundaries BUT
not with directory boundaries. Imagine you have the stack of backend,
data transport and frontend layers. The feature has to touch all three
layers thus resulting in the changes in the apparently different
directories. Thus, making the distinction by the pathspec (if I
understood it right from reading the documentation) would not help.

The attributes could be a solution, if I could:
1. create attribute designated to the feature
2. "mark" uncommitted files in different directory with that attribute
3. filter the list of unchanged files with such attribute
4. create commit for the files only with the certain attribute

You've kindly demonstrated that #4 is doable; however I could not
clearly get for the Git documentation if #1 - #3 are achievable...
Could you point me to the right place in the documentation, please?


On Tue, Jul 11, 2017 at 1:27 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Nikolay Shustov <nikolay.shustov@gmail.com> writes:
>
>> I have to work on several features in the same code tree parallel, in
>> the same Perforce workspace. The major reason why I cannot work on one
>> feature then on another is just because I have to make sure that the
>> changes in the related areas of the product play together well.
>>
>> With Perforce, I can have multiple changelists opened, that group the
>> changed files as needed.
>>
>> With Git I cannot seem to finding the possibility to figure out how to
>> achieve the same result. And the problem is that putting change sets
>> on different Git branches (or workdirs, or whatever Git offers that
>> makes the changes to be NOT in the same source tree) is not a viable
>> option from me ...
>
> Naturally.  If these separate changes need to work together, it is
> way too inconvenient if these changes do not appear in a single
> unified working tree to be built and tested.
>
>> Is it worth considering adding to Git a feature like "group of files"
>> that would offer some virtutal grouping of the locally changed files
>> in the checked-out branch?
>
> Let's step back and let me make sure if I understand you correctly.
> You want to work on a system with two distinct areas (say, the
> frontend and the backend), that have to work together, but you want
> to make two commits, one for each area.  You make changes for both
> areas in your working tree, build and test them to make sure the
> whole thing works well together, and at the end, you make two
> commits.
>
> In your real project, you may be doing more than two areas and more
> than two commits, but is the above a good degenerative case that
> shows the basic idea?  If not, then please disregard all of the
> following.
>
> You can make partial commits in Git.  In the simplest case, you may
> have two separate files backend.py and frontend.py, you make edits
> to both files and then make two commits:
>
>         $ git commit backend.py
>         $ git commit frontend.py
>
> Changes to some files may contain both changes for the backend and
> for the frontend that does not allow you to separate commits at file
> boundary, and Git even lets you handle such a case.  If you have the
> third file in addition to the above two, called common.py, you could
> instead
>
>         $ git add backend.py
>         $ git add -p common.py
>
> to prepare the index to contain only the changes for the backend
> part ("add -p" lets you interactively pick and choose the hunks
> relevant to the backend part), and conclude the commit for the
> backend part with
>
>         $ git commit ;# no paths arguments
>
> and then when all the remaining changes are for the frontend part,
> you can follow it with
>
>         $ git commit -a
>
> to make another commit for the frontend part.
>
> A short answer to your question, provided if I understood you
> correctly, is "no, there is no way to say 'backend.py, backend-2.py,
> ...' are the backend things and call it a filegroup", accompanied by
> "a filegroup would only be effective when changes align with file
> boundary".
>
> And if your response is "but most of the time changes align with
> file boundary", a counter-response is "and most of the time changes
> align with directory boundary (in well structured project, at
> least), so you can do 'git commit backend/' for all the backend part
> without having to name all the paths anyway".
>
> There is an experimental "attributes-limited pathspec" feature in
> recent versions of Git, which lets you assign arbitrary sets of
> labels to each paths, and using that you may be able to do
>
>         $ git commit ':(attr:filegroup=backend).'
>
> and I suspect that would be the closest thing you would want (read
> about 'pathspec' in 'git help glossary')
>

Re: "groups of files" in Git?

[Stefan Beller]

On Tue, Jul 11, 2017 at 11:10 AM, Nikolay Shustov
<nikolay.shustov@gmail.com> wrote:
> Thank you for the explanation. The example about backend and frontend
> is relevant even though I normally have to deal with more layers at
> the same time.
>
> However, in my case I have the thing that you have already tried to
> address, partially: the changes always align with file boundaries BUT
> not with directory boundaries. Imagine you have the stack of backend,
> data transport and frontend layers. The feature has to touch all three
> layers thus resulting in the changes in the apparently different
> directories. Thus, making the distinction by the pathspec (if I
> understood it right from reading the documentation) would not help.
>
> The attributes could be a solution, if I could:
> 1. create attribute designated to the feature
> 2. "mark" uncommitted files in different directory with that attribute

1+2 should be answered by the gitattributes man page
https://git-scm.com/docs/gitattributes


> 3. filter the list of unchanged files with such attribute

This sounds like one of
  "git status :(attr:backend) ."
  "git status :(exclude,attr:backend) ."

> 4. create commit for the files only with the certain attribute
>
> You've kindly demonstrated that #4 is doable; however I could not
> clearly get for the Git documentation if #1 - #3 are achievable...
> Could you point me to the right place in the documentation, please?
>

Re: "groups of files" in Git?

[Nikolay Shustov]

Thank you for #3.
As for 1+2, the documentation says:

"Each line in gitattributes file is of form:

pattern attr1 attr2 ...

...
When the pattern matches the path in question, the attributes listed
on the line are given to the path."

My understanding is that to have the bunch of the files in the
separate directories having the same attribute, I would have to, for
each file, create a separate gitattributes line with the exact
paths/filename and needed attribute. Is it would you are suggesting or
I misunderstood the idea?


On Tue, Jul 11, 2017 at 2:19 PM, Stefan Beller <sbeller@google.com> wrote:
> On Tue, Jul 11, 2017 at 11:10 AM, Nikolay Shustov
> <nikolay.shustov@gmail.com> wrote:
>> Thank you for the explanation. The example about backend and frontend
>> is relevant even though I normally have to deal with more layers at
>> the same time.
>>
>> However, in my case I have the thing that you have already tried to
>> address, partially: the changes always align with file boundaries BUT
>> not with directory boundaries. Imagine you have the stack of backend,
>> data transport and frontend layers. The feature has to touch all three
>> layers thus resulting in the changes in the apparently different
>> directories. Thus, making the distinction by the pathspec (if I
>> understood it right from reading the documentation) would not help.
>>
>> The attributes could be a solution, if I could:
>> 1. create attribute designated to the feature
>> 2. "mark" uncommitted files in different directory with that attribute
>
> 1+2 should be answered by the gitattributes man page
> https://git-scm.com/docs/gitattributes
>
>
>> 3. filter the list of unchanged files with such attribute
>
> This sounds like one of
>   "git status :(attr:backend) ."
>   "git status :(exclude,attr:backend) ."
>
>> 4. create commit for the files only with the certain attribute
>>
>> You've kindly demonstrated that #4 is doable; however I could not
>> clearly get for the Git documentation if #1 - #3 are achievable...
>> Could you point me to the right place in the documentation, please?
>>

Re: "groups of files" in Git?

[Lars Schneider]

> On 11 Jul 2017, at 17:45, Nikolay Shustov <nikolay.shustov@gmail.com> wrote:
> 
> Hi,
> I have been recently struggling with migrating my development workflow
> from Perforce to Git, all because of the following thing:
> 
> I have to work on several features in the same code tree parallel, in
> the same Perforce workspace. The major reason why I cannot work on one
> feature then on another is just because I have to make sure that the
> changes in the related areas of the product play together well.
> 
> With Perforce, I can have multiple changelists opened, that group the
> changed files as needed.
> 
> With Git I cannot seem to finding the possibility to figure out how to
> achieve the same result. And the problem is that putting change sets
> on different Git branches (or workdirs, or whatever Git offers that
> makes the changes to be NOT in the same source tree) is not a viable
> option from me as I would have to re-build code as I re-integrate the
> changes between the branches (or whatever changes separation Git
> feature is used).
> Build takes time and resources and considering that I have to do it on
> multiple platforms (I do cross-platform development) it really
> denominates the option of not having multiple changes in the same code
> tree.
> 
> Am I ignorant about some Git feature/way of using Git that would help?
> Is it worth considering adding to Git a feature like "group of files"
> that would offer some virtutal grouping of the locally changed files
> in the checked-out branch?

Interesting question that came up at my workplace, too.

Here is what I suggested:
1. Keep working on a single branch and make commits for all features
2. If you make a commit, prefix the commit message with the feature name
3. After you are done with a feature create a new feature branch based on
   your combined feature branch. Use `git rebase -i` [1] to remove all
   commits that are not relevant for the feature. Alternatively you could
   cherry pick the relevant commits [2] if this is faster.

I wonder what others think about this solution. Maybe there is a better
solution that I overlooked?

- Lars

[1] https://robots.thoughtbot.com/git-interactive-rebase-squash-amend-rewriting-history
[2] http://think-like-a-git.net/sections/rebase-from-the-ground-up/cherry-picking-explained.html

Re: "groups of files" in Git?

[Nikolay Shustov]

Thank you for the idea, however I am having troubles with basically
maintaining the uncommitted groups of files: I would prefer the clear
distinction that "those files belong to feature A" and "these files
belong to feature B", before I commit anything. Committing separately
every change for feature A and for feature B would probably a good
option unless I have many changes and then cherry-picking the proper
commits to create a single changeset for the integration would become
a nightmare.

On Tue, Jul 11, 2017 at 1:39 PM, Lars Schneider
<larsxschneider@gmail.com> wrote:
>
>> On 11 Jul 2017, at 17:45, Nikolay Shustov <nikolay.shustov@gmail.com> wrote:
>>
>> Hi,
>> I have been recently struggling with migrating my development workflow
>> from Perforce to Git, all because of the following thing:
>>
>> I have to work on several features in the same code tree parallel, in
>> the same Perforce workspace. The major reason why I cannot work on one
>> feature then on another is just because I have to make sure that the
>> changes in the related areas of the product play together well.
>>
>> With Perforce, I can have multiple changelists opened, that group the
>> changed files as needed.
>>
>> With Git I cannot seem to finding the possibility to figure out how to
>> achieve the same result. And the problem is that putting change sets
>> on different Git branches (or workdirs, or whatever Git offers that
>> makes the changes to be NOT in the same source tree) is not a viable
>> option from me as I would have to re-build code as I re-integrate the
>> changes between the branches (or whatever changes separation Git
>> feature is used).
>> Build takes time and resources and considering that I have to do it on
>> multiple platforms (I do cross-platform development) it really
>> denominates the option of not having multiple changes in the same code
>> tree.
>>
>> Am I ignorant about some Git feature/way of using Git that would help?
>> Is it worth considering adding to Git a feature like "group of files"
>> that would offer some virtutal grouping of the locally changed files
>> in the checked-out branch?
>
> Interesting question that came up at my workplace, too.
>
> Here is what I suggested:
> 1. Keep working on a single branch and make commits for all features
> 2. If you make a commit, prefix the commit message with the feature name
> 3. After you are done with a feature create a new feature branch based on
>    your combined feature branch. Use `git rebase -i` [1] to remove all
>    commits that are not relevant for the feature. Alternatively you could
>    cherry pick the relevant commits [2] if this is faster.
>
> I wonder what others think about this solution. Maybe there is a better
> solution that I overlooked?
>
> - Lars
>
> [1] https://robots.thoughtbot.com/git-interactive-rebase-squash-amend-rewriting-history
> [2] http://think-like-a-git.net/sections/rebase-from-the-ground-up/cherry-picking-explained.html
>

Re: "groups of files" in Git?

[Lars Schneider]

> On Tue, Jul 11, 2017 at 1:39 PM, Lars Schneider
> <larsxschneider@gmail.com> wrote:
>> 
>>> On 11 Jul 2017, at 17:45, Nikolay Shustov <nikolay.shustov@gmail.com> wrote:
>>> 
>>> Hi,
>>> I have been recently struggling with migrating my development workflow
>>> from Perforce to Git, all because of the following thing:
>>> 
>>> I have to work on several features in the same code tree parallel, in
>>> the same Perforce workspace. The major reason why I cannot work on one
>>> feature then on another is just because I have to make sure that the
>>> changes in the related areas of the product play together well.
>>> 
>>> With Perforce, I can have multiple changelists opened, that group the
>>> changed files as needed.
>>> 
>>> With Git I cannot seem to finding the possibility to figure out how to
>>> achieve the same result. And the problem is that putting change sets
>>> on different Git branches (or workdirs, or whatever Git offers that
>>> makes the changes to be NOT in the same source tree) is not a viable
>>> option from me as I would have to re-build code as I re-integrate the
>>> changes between the branches (or whatever changes separation Git
>>> feature is used).
>>> Build takes time and resources and considering that I have to do it on
>>> multiple platforms (I do cross-platform development) it really
>>> denominates the option of not having multiple changes in the same code
>>> tree.
>>> 
>>> Am I ignorant about some Git feature/way of using Git that would help?
>>> Is it worth considering adding to Git a feature like "group of files"
>>> that would offer some virtutal grouping of the locally changed files
>>> in the checked-out branch?
>> 
>> Interesting question that came up at my workplace, too.
>> 
>> Here is what I suggested:
>> 1. Keep working on a single branch and make commits for all features
>> 2. If you make a commit, prefix the commit message with the feature name
>> 3. After you are done with a feature create a new feature branch based on
>>   your combined feature branch. Use `git rebase -i` [1] to remove all
>>   commits that are not relevant for the feature. Alternatively you could
>>   cherry pick the relevant commits [2] if this is faster.
>> 
>> I wonder what others think about this solution. Maybe there is a better
>> solution that I overlooked?
>> 
>> - Lars
>> 
>> [1] https://robots.thoughtbot.com/git-interactive-rebase-squash-amend-rewriting-history
>> [2] http://think-like-a-git.net/sections/rebase-from-the-ground-up/cherry-picking-explained.html
>> 

> On 11 Jul 2017, at 19:54, Nikolay Shustov <nikolay.shustov@gmail.com> wrote:
> 
> Thank you for the idea, however I am having troubles with basically
> maintaining the uncommitted groups of files: I would prefer the clear
> distinction that "those files belong to feature A" and "these files
> belong to feature B", before I commit anything. Committing separately
> every change for feature A and for feature B would probably a good
> option unless I have many changes and then cherry-picking the proper
> commits to create a single changeset for the integration would become
> a nightmare.

I see. Why so complicated with gitattributes then?

How about this:
Let's say you start working on featureX that affects file1 and file2
and featureY that affects file8 and file9

1. Create aliases to add the files:
   $ git config --local alias.featx 'add file1 file2'
   $ git config --local alias.featy 'add file8 file9'

2. Work on the features. Whenever you have something ready for featureX
   run this:
   $ git featx
   $ git commit

   Whenever you have something ready for featureY run this:
   $ git featy
   $ git commit

Wouldn't that work?

- Lars

Re: "groups of files" in Git?

[Nikolay Shustov]

Thank you, this could work, but if I am adding new file to the
feature/removing the existing file from the feature, aliases usage for
"add" doesn't help much.
I would really need to have the lists of files... and attributes look
more promising.

On Tue, Jul 11, 2017 at 4:20 PM, Lars Schneider
<larsxschneider@gmail.com> wrote:
>> On Tue, Jul 11, 2017 at 1:39 PM, Lars Schneider
>> <larsxschneider@gmail.com> wrote:
>>>
>>>> On 11 Jul 2017, at 17:45, Nikolay Shustov <nikolay.shustov@gmail.com> wrote:
>>>>
>>>> Hi,
>>>> I have been recently struggling with migrating my development workflow
>>>> from Perforce to Git, all because of the following thing:
>>>>
>>>> I have to work on several features in the same code tree parallel, in
>>>> the same Perforce workspace. The major reason why I cannot work on one
>>>> feature then on another is just because I have to make sure that the
>>>> changes in the related areas of the product play together well.
>>>>
>>>> With Perforce, I can have multiple changelists opened, that group the
>>>> changed files as needed.
>>>>
>>>> With Git I cannot seem to finding the possibility to figure out how to
>>>> achieve the same result. And the problem is that putting change sets
>>>> on different Git branches (or workdirs, or whatever Git offers that
>>>> makes the changes to be NOT in the same source tree) is not a viable
>>>> option from me as I would have to re-build code as I re-integrate the
>>>> changes between the branches (or whatever changes separation Git
>>>> feature is used).
>>>> Build takes time and resources and considering that I have to do it on
>>>> multiple platforms (I do cross-platform development) it really
>>>> denominates the option of not having multiple changes in the same code
>>>> tree.
>>>>
>>>> Am I ignorant about some Git feature/way of using Git that would help?
>>>> Is it worth considering adding to Git a feature like "group of files"
>>>> that would offer some virtutal grouping of the locally changed files
>>>> in the checked-out branch?
>>>
>>> Interesting question that came up at my workplace, too.
>>>
>>> Here is what I suggested:
>>> 1. Keep working on a single branch and make commits for all features
>>> 2. If you make a commit, prefix the commit message with the feature name
>>> 3. After you are done with a feature create a new feature branch based on
>>>   your combined feature branch. Use `git rebase -i` [1] to remove all
>>>   commits that are not relevant for the feature. Alternatively you could
>>>   cherry pick the relevant commits [2] if this is faster.
>>>
>>> I wonder what others think about this solution. Maybe there is a better
>>> solution that I overlooked?
>>>
>>> - Lars
>>>
>>> [1] https://robots.thoughtbot.com/git-interactive-rebase-squash-amend-rewriting-history
>>> [2] http://think-like-a-git.net/sections/rebase-from-the-ground-up/cherry-picking-explained.html
>>>
>
>> On 11 Jul 2017, at 19:54, Nikolay Shustov <nikolay.shustov@gmail.com> wrote:
>>
>> Thank you for the idea, however I am having troubles with basically
>> maintaining the uncommitted groups of files: I would prefer the clear
>> distinction that "those files belong to feature A" and "these files
>> belong to feature B", before I commit anything. Committing separately
>> every change for feature A and for feature B would probably a good
>> option unless I have many changes and then cherry-picking the proper
>> commits to create a single changeset for the integration would become
>> a nightmare.
>
> I see. Why so complicated with gitattributes then?
>
> How about this:
> Let's say you start working on featureX that affects file1 and file2
> and featureY that affects file8 and file9
>
> 1. Create aliases to add the files:
>    $ git config --local alias.featx 'add file1 file2'
>    $ git config --local alias.featy 'add file8 file9'
>
> 2. Work on the features. Whenever you have something ready for featureX
>    run this:
>    $ git featx
>    $ git commit
>
>    Whenever you have something ready for featureY run this:
>    $ git featy
>    $ git commit
>
> Wouldn't that work?
>
> - Lars
>
>

Re: "groups of files" in Git?

[Igor Djordjevic]

For starters, let me say that I consider myself a mere advanced 
beginner Git user, and I haven`t used Perforce ever before (did some 
reading now), but still, for what it`s worth, here are my thoughts, 
please bare with me :)

Do feel free to correct me if I miss something.

On 11/07/2017 19:39, Lars Schneider wrote:
> 
>> On 11 Jul 2017, at 17:45, Nikolay Shustov <nikolay.shustov@gmail.com> wrote:
>>
>> Hi,
>> I have been recently struggling with migrating my development workflow
>> from Perforce to Git, all because of the following thing:
>>
>> I have to work on several features in the same code tree parallel, in
>> the same Perforce workspace. The major reason why I cannot work on one
>> feature then on another is just because I have to make sure that the
>> changes in the related areas of the product play together well.
>>
>> With Perforce, I can have multiple changelists opened, that group the
>> changed files as needed.
>>
>> With Git I cannot seem to finding the possibility to figure out how to
>> achieve the same result. And the problem is that putting change sets
>> on different Git branches (or workdirs, or whatever Git offers that
>> makes the changes to be NOT in the same source tree) is not a viable
>> option from me as I would have to re-build code as I re-integrate the
>> changes between the branches (or whatever changes separation Git
>> feature is used).
>> Build takes time and resources and considering that I have to do it on
>> multiple platforms (I do cross-platform development) it really
>> denominates the option of not having multiple changes in the same code
>> tree.
>>
>> Am I ignorant about some Git feature/way of using Git that would help?
>> Is it worth considering adding to Git a feature like "group of files"
>> that would offer some virtutal grouping of the locally changed files
>> in the checked-out branch?
> 
> Interesting question that came up at my workplace, too.
> 
> Here is what I suggested:
> 1. Keep working on a single branch and make commits for all features
> 2. If you make a commit, prefix the commit message with the feature name
> 3. After you are done with a feature create a new feature branch based on
>    your combined feature branch. Use `git rebase -i` [1] to remove all
>    commits that are not relevant for the feature. Alternatively you could
>    cherry pick the relevant commits [2] if this is faster.
> 
> I wonder what others think about this solution. Maybe there is a better
> solution that I overlooked?
> 
> - Lars
> 
> [1] https://robots.thoughtbot.com/git-interactive-rebase-squash-amend-rewriting-history
> [2] http://think-like-a-git.net/sections/rebase-from-the-ground-up/cherry-picking-explained.html

This "single branch, related commits" approach is exactly what came 
to my mind as well.

But, isn`t Perforce "changelist" an "atomic" group of changes - like 
"commit" in Git, "changeset" in Team Foundation Version Control, 
etc...?

If so, it would mean that this "multiple pending changelists" flow 
would/should be translated to "multiple pending commits" in Git, 
where in the end _a single_ Perforce "changelist" is _a single_ Git 
"commit".

Might be this is where the confusion is coming from, trying to fit 
natural Git "multiple commits per feature" (but "single feature per 
branch") concept into a "single changelist per feature" Perforce 
concept, described/required here?

I don`t think there is a firm concept of such "multiple pending 
commits" in Git, but the point is the author is having multiple 
changelists/commits, and it actively updates them (the _same ones_), 
until they`re ready to be merged (submitted, in Perforce)... which 
you can do in Git, and might be quite easily :)

So... In Git, you can create a separate commit for each changelist 
you have in Perforce (all these commits being on the same branch, as 
desired). Then, when you would have wanted to "update" the pending 
Perforce changelist (not sure what the corresponding command is in 
Perforce), you would just `git commit` your current state with 
additional "--squash" or "--fixup" parameters (depending on if you 
would like to add more description to existing/original commit 
message, or not), and the original commit SHA1.

In the end, when everything is tested together and you would like to 
commit features separately (like submitting changelists in Perforce), 
you would just need to `git rebase -i --autosquash` your branch, 
where Git would squash all your "update" commits (fixup/squash ones, 
that is) to the original/initial ones you made as per your 
changelists/features. No need for manual rearranging, cherry-picking, 
or whatever.

An example flow, with two "changelists" for two features (I`ll be 
using capital letters A, B, C... instead of commit SHA1, for 
simplicity):

	... do some "Feature 1" work...
	$ git commit -m "Feature 1"
	... do some "Feature 2" work...
	$ git commit -m "Feature 2"
	... do some "Feature 1" work...
	$ git commit --fixup A
	... do some "Feature 1" work...
	$ git commit --fixup A
	... do some "Feature 2" work...
	$ git commit --squash B
	... do some "Feature 1" work...
	$ git commit --fixup A
	... do some "Feature 1" work...
	$ git commit --squash A
	... do some "Feature 2" work...
	$ git commit --fixup B


Branch history would look something like this (H is latest commit):

	H fixup! Feature 2
	G squash! Feature 1
	F fixup! Feature 1
	E squash! Feature 2
	D fixup! Feature 1
	C fixup! Feature 1
	B Feature 2
	A Feature 1


When you finally do `git rebase -i --autosquash A^`, you should get a 
list like this[1]:

	pick A Feature 1
	fixup C fixup! Feature 1
	fixup D fixup! Feature 1
	fixup F fixup! Feature 1
	squash G squash! Feature 1
	pick B Feature 2
	squash E squash! Feature 2
	fixup H fixup! Feature 2


Once rebase is finished, you`ll end up with branch history looking 
like this:

	B' Feature 2
	A' Feature 1

... where commits A, C, D, F and G have been squashed into a single 
commit A', and commits B, E and H have been squashed into a single 
commit B'. These two single commits should correspond to your two 
Perforce changelists.

Now you can merge your commits separately, as desired ("submit" the 
"changelists").

You can even first rearrange/split/squash them further, or make 
separate branches out of them, whatever you find appropriate - you 
can do whatever you like to them while they`re your local commits 
("pending changelists"), before making them live/visible for other 
users as well (merge them to a public branch, "submit changelist").

p.s. Doesn`t the flow required here look similar to Mercurial patch
"queues" approach (again, resembling "quilt" functionality)? If so, 
"Guilt"[2] may be an option here as well... if the described flow 
can`t be altered a bit to align better with Git itself, might be 
profiting on the side of overall workflow simplicity ;)

[1] Having commits automatically grouped/ordered, you can even 
    replace some "fixup" and "squash" with "reword", for example, so 
    those commits are kept as separate ones, providing you a chance 
    to edit their messages.
[2] http://repo.or.cz/w/guilt.git

Regards,
Buga

Re: "groups of files" in Git?

[Nikolay Shustov]

Thank you for the detailed explanation, it looks like merging the
commits would be helpful in my case. And I think it is a very good
analogy that Perforce changelists are like multiple pending committs,
if Git were supporting such.

What it won't be achieving by using commits in this schema is the
following thing I can do in Perforce:
In the uncommitted Perforce changelists I can revert the changed file
to the original state and move the files between the changelists.
Quite often, while working on something, in the middle I would decide
to isolate changes to a certain set of files to a separate changelsit
- but then I might change my mind. It is all flexible until I actually
commit my Perforce changelist, after which it becomes very much as
committed changes in any other source control.
This is actual flexibility I am looking for achieving in Git.


On Tue, Jul 11, 2017 at 6:46 PM, Igor Djordjevic
<igor.d.djordjevic@gmail.com> wrote:
> For starters, let me say that I consider myself a mere advanced
> beginner Git user, and I haven`t used Perforce ever before (did some
> reading now), but still, for what it`s worth, here are my thoughts,
> please bare with me :)
>
> Do feel free to correct me if I miss something.
>
> On 11/07/2017 19:39, Lars Schneider wrote:
>>
>>> On 11 Jul 2017, at 17:45, Nikolay Shustov <nikolay.shustov@gmail.com> wrote:
>>>
>>> Hi,
>>> I have been recently struggling with migrating my development workflow
>>> from Perforce to Git, all because of the following thing:
>>>
>>> I have to work on several features in the same code tree parallel, in
>>> the same Perforce workspace. The major reason why I cannot work on one
>>> feature then on another is just because I have to make sure that the
>>> changes in the related areas of the product play together well.
>>>
>>> With Perforce, I can have multiple changelists opened, that group the
>>> changed files as needed.
>>>
>>> With Git I cannot seem to finding the possibility to figure out how to
>>> achieve the same result. And the problem is that putting change sets
>>> on different Git branches (or workdirs, or whatever Git offers that
>>> makes the changes to be NOT in the same source tree) is not a viable
>>> option from me as I would have to re-build code as I re-integrate the
>>> changes between the branches (or whatever changes separation Git
>>> feature is used).
>>> Build takes time and resources and considering that I have to do it on
>>> multiple platforms (I do cross-platform development) it really
>>> denominates the option of not having multiple changes in the same code
>>> tree.
>>>
>>> Am I ignorant about some Git feature/way of using Git that would help?
>>> Is it worth considering adding to Git a feature like "group of files"
>>> that would offer some virtutal grouping of the locally changed files
>>> in the checked-out branch?
>>
>> Interesting question that came up at my workplace, too.
>>
>> Here is what I suggested:
>> 1. Keep working on a single branch and make commits for all features
>> 2. If you make a commit, prefix the commit message with the feature name
>> 3. After you are done with a feature create a new feature branch based on
>>    your combined feature branch. Use `git rebase -i` [1] to remove all
>>    commits that are not relevant for the feature. Alternatively you could
>>    cherry pick the relevant commits [2] if this is faster.
>>
>> I wonder what others think about this solution. Maybe there is a better
>> solution that I overlooked?
>>
>> - Lars
>>
>> [1] https://robots.thoughtbot.com/git-interactive-rebase-squash-amend-rewriting-history
>> [2] http://think-like-a-git.net/sections/rebase-from-the-ground-up/cherry-picking-explained.html
>
> This "single branch, related commits" approach is exactly what came
> to my mind as well.
>
> But, isn`t Perforce "changelist" an "atomic" group of changes - like
> "commit" in Git, "changeset" in Team Foundation Version Control,
> etc...?
>
> If so, it would mean that this "multiple pending changelists" flow
> would/should be translated to "multiple pending commits" in Git,
> where in the end _a single_ Perforce "changelist" is _a single_ Git
> "commit".
>
> Might be this is where the confusion is coming from, trying to fit
> natural Git "multiple commits per feature" (but "single feature per
> branch") concept into a "single changelist per feature" Perforce
> concept, described/required here?
>
> I don`t think there is a firm concept of such "multiple pending
> commits" in Git, but the point is the author is having multiple
> changelists/commits, and it actively updates them (the _same ones_),
> until they`re ready to be merged (submitted, in Perforce)... which
> you can do in Git, and might be quite easily :)
>
> So... In Git, you can create a separate commit for each changelist
> you have in Perforce (all these commits being on the same branch, as
> desired). Then, when you would have wanted to "update" the pending
> Perforce changelist (not sure what the corresponding command is in
> Perforce), you would just `git commit` your current state with
> additional "--squash" or "--fixup" parameters (depending on if you
> would like to add more description to existing/original commit
> message, or not), and the original commit SHA1.
>
> In the end, when everything is tested together and you would like to
> commit features separately (like submitting changelists in Perforce),
> you would just need to `git rebase -i --autosquash` your branch,
> where Git would squash all your "update" commits (fixup/squash ones,
> that is) to the original/initial ones you made as per your
> changelists/features. No need for manual rearranging, cherry-picking,
> or whatever.
>
> An example flow, with two "changelists" for two features (I`ll be
> using capital letters A, B, C... instead of commit SHA1, for
> simplicity):
>
>         ... do some "Feature 1" work...
>         $ git commit -m "Feature 1"
>         ... do some "Feature 2" work...
>         $ git commit -m "Feature 2"
>         ... do some "Feature 1" work...
>         $ git commit --fixup A
>         ... do some "Feature 1" work...
>         $ git commit --fixup A
>         ... do some "Feature 2" work...
>         $ git commit --squash B
>         ... do some "Feature 1" work...
>         $ git commit --fixup A
>         ... do some "Feature 1" work...
>         $ git commit --squash A
>         ... do some "Feature 2" work...
>         $ git commit --fixup B
>
>
> Branch history would look something like this (H is latest commit):
>
>         H fixup! Feature 2
>         G squash! Feature 1
>         F fixup! Feature 1
>         E squash! Feature 2
>         D fixup! Feature 1
>         C fixup! Feature 1
>         B Feature 2
>         A Feature 1
>
>
> When you finally do `git rebase -i --autosquash A^`, you should get a
> list like this[1]:
>
>         pick A Feature 1
>         fixup C fixup! Feature 1
>         fixup D fixup! Feature 1
>         fixup F fixup! Feature 1
>         squash G squash! Feature 1
>         pick B Feature 2
>         squash E squash! Feature 2
>         fixup H fixup! Feature 2
>
>
> Once rebase is finished, you`ll end up with branch history looking
> like this:
>
>         B' Feature 2
>         A' Feature 1
>
> ... where commits A, C, D, F and G have been squashed into a single
> commit A', and commits B, E and H have been squashed into a single
> commit B'. These two single commits should correspond to your two
> Perforce changelists.
>
> Now you can merge your commits separately, as desired ("submit" the
> "changelists").
>
> You can even first rearrange/split/squash them further, or make
> separate branches out of them, whatever you find appropriate - you
> can do whatever you like to them while they`re your local commits
> ("pending changelists"), before making them live/visible for other
> users as well (merge them to a public branch, "submit changelist").
>
> p.s. Doesn`t the flow required here look similar to Mercurial patch
> "queues" approach (again, resembling "quilt" functionality)? If so,
> "Guilt"[2] may be an option here as well... if the described flow
> can`t be altered a bit to align better with Git itself, might be
> profiting on the side of overall workflow simplicity ;)
>
> [1] Having commits automatically grouped/ordered, you can even
>     replace some "fixup" and "squash" with "reword", for example, so
>     those commits are kept as separate ones, providing you a chance
>     to edit their messages.
> [2] http://repo.or.cz/w/guilt.git
>
> Regards,
> Buga

Re: "groups of files" in Git?

[Junio C Hamano]

Nikolay Shustov <nikolay.shustov@gmail.com> writes:

> Thank you for the detailed explanation, it looks like merging the
> commits would be helpful in my case. And I think it is a very good
> analogy that Perforce changelists are like multiple pending committs,
> if Git were supporting such.
>
> What it won't be achieving by using commits in this schema is the
> following thing I can do in Perforce:
> In the uncommitted Perforce changelists I can revert the changed file
> to the original state and move the files between the changelists.
> Quite often, while working on something, in the middle I would decide
> to isolate changes to a certain set of files to a separate changelsit
> - but then I might change my mind. It is all flexible until I actually
> commit my Perforce changelist, after which it becomes very much as
> committed changes in any other source control.
> This is actual flexibility I am looking for achieving in Git.

I actually think we already have such a flexibility.  Unlike
Perforce, Git is distributed, and the most important aspect of the
distinction is that what happens _in_ your local Git repository may
be called "committed" in Git lingo, but not visible to the public.

You can consider these commits you make in your repository "pending"
when you think of your workflow in Perforce terms, until you merge
and push out the result, which roughly corresponds to "submitting"
in Perforce lingo.

Once you start treating your local commits that you haven't pushed
out as changes that are still "pending" when observed from the
outside world, you'd realize that you have as much flexibilty, if
not more, to dice and slice them with the local tools like "rebase
-i", "add -p", etc., as you would have in your Perforce workflow,
I would think.

Re: "groups of files" in Git?

[Nikolay Shustov]

Thank you, but I am not sure I quite understand the idea.
Could you please elaborate on it for the following example?

I have two Perforce changelists ("A" and "B") that group uncommitted
sets of files (paths to each of files could be different):

changelist A:
file1
file2

changelist B:
file3
file4

In Perforce, I am able to do the following:
- move files between changelists (e.g. file1 could be moved to changelist B)
- add new files to changeslit (e.g. changelist B can get additional file5)
- revert file changes which would effectively remove file from the
changelst (e.g. revert file2 will remove it from changelist A)

How would I do it with sets of files that would belong to Git commit?


On Thu, Jul 13, 2017 at 2:09 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Nikolay Shustov <nikolay.shustov@gmail.com> writes:
>
>> Thank you for the detailed explanation, it looks like merging the
>> commits would be helpful in my case. And I think it is a very good
>> analogy that Perforce changelists are like multiple pending committs,
>> if Git were supporting such.
>>
>> What it won't be achieving by using commits in this schema is the
>> following thing I can do in Perforce:
>> In the uncommitted Perforce changelists I can revert the changed file
>> to the original state and move the files between the changelists.
>> Quite often, while working on something, in the middle I would decide
>> to isolate changes to a certain set of files to a separate changelsit
>> - but then I might change my mind. It is all flexible until I actually
>> commit my Perforce changelist, after which it becomes very much as
>> committed changes in any other source control.
>> This is actual flexibility I am looking for achieving in Git.
>
> I actually think we already have such a flexibility.  Unlike
> Perforce, Git is distributed, and the most important aspect of the
> distinction is that what happens _in_ your local Git repository may
> be called "committed" in Git lingo, but not visible to the public.
>
> You can consider these commits you make in your repository "pending"
> when you think of your workflow in Perforce terms, until you merge
> and push out the result, which roughly corresponds to "submitting"
> in Perforce lingo.
>
> Once you start treating your local commits that you haven't pushed
> out as changes that are still "pending" when observed from the
> outside world, you'd realize that you have as much flexibilty, if
> not more, to dice and slice them with the local tools like "rebase
> -i", "add -p", etc., as you would have in your Perforce workflow,
> I would think.
>
>

Re: "groups of files" in Git?

[Fredrik Gustafsson]

Hi,
I will choose a bit of a less diplomatic path here. Instead of trying to
tell you how you can make git fit your needs, I would say that you
shouldn't. I've two arguments:

1.
It's always painful when you try to use a tool in some way it's not
intended or used to work. If you're doing something different than
anyone else using that tool, you're probably doing something wrong!

I doubt that your case is so special, so my suggestion is to either use
git the way most people use it, with one branch for each feature, or do
not use git at all, since perforce seems to be better with your
workstyle.

2.
Git is a snapshot based SCM system. That means that each commit unique
identify a version of the code. With your system (as well as any time
you're not commiting all changed files) the commit is never tested.
You've no idea of actually knowing if your two changes is dependent or
not. Of course you can guess, but it's still a guess and in your current
work way with perforce you have no way of knowing if your changesets
have a dependency between eachother or not since you never test them
individually.

--

Please let me know if you feel that I've missed something.

I can see four solutions:

1.
Now I would suggest that you have each feature in a commit and simply
run your tests every few commits so you don't have to run it for each
commit.

2.
Improve your build and test time. I'm sure there's things here to
improve.

3.
Continue to use perforce. If I recall correctly perforce has even a git
integration today.

4.
Use integration branches in git and run the tests on that branch. This
can be easy todo if you write some scripts for it.

Good luck!

-- 
Fredrik Gustafsson

phone: +46 733-608274
e-mail: iveqy@iveqy.com
website: http://www.iveqy.com

Re: "groups of files" in Git?

[astian]

Nikolay Shustov wrote:
> With Perforce, I can have multiple changelists opened, that group the
> changed files as needed.
>
> With Git I cannot seem to finding the possibility to figure out how to
> achieve the same result. And the problem is that putting change sets
> on different Git branches (or workdirs, or whatever Git offers that
> makes the changes to be NOT in the same source tree) is not a viable
> option from me as I would have to re-build code as I re-integrate the
> changes between the branches (or whatever changes separation Git
> feature is used).
> Build takes time and resources and considering that I have to do it on
> multiple platforms (I do cross-platform development) it really
> denominates the option of not having multiple changes in the same code
> tree.
>
> Am I ignorant about some Git feature/way of using Git that would help?
> Is it worth considering adding to Git a feature like "group of files"
> that would offer some virtutal grouping of the locally changed files
> in the checked-out branch?

I never used Perforce and I'm not even sure I understand your problem,
but I thought I'd mention something that nobody else seems to have yet
(unless I missed it):

First, one thing that seems obvious to me from your description is that
these "parallel features" you work on are obviously interdependent,
therefore I would rather consider the whole thing as a single feature.
Therefore, it makes sense to me to work in a single "topic branch".

This doesn't preclude one from separating the changes in logically
sensible pieces.  Indeed this is par for the course in Git and people do
it all the time by dividing the bulk of changes into a carefully chosen
series of commits.

I think the most common way of doing this is to simply work on the whole
thing and once you're happy with it you use "git rebase --interative" in
order to "prettify" your history.

But, and here comes the part I think nobody mentioned yet, if your
feature work is considerably large or spans a considerably long time it
may be undesirable to postpone all that work until the very end (perhaps
by then you already forgot important information, or perhaps too many
changes have accumulated so reviewing them all becomes significantly
less efficient).  In that case, one solution is to use a "patch
management system" which will let you do that work incrementally (going
back and forth as needed).

If you know mercurial, this is "hg mq".  I don't think Git has any such
system built-in, but I know there are at least these external tools that
integrate with Git:
https://git.wiki.kernel.org/index.php/Interfaces,_frontends,_and_tools#Patch-management_Interface_layers

Feel free to ignore this if I totally misunderstood your use case.

Cheers.

Re: "groups of files" in Git?

[Nikolay Shustov]

Thank you taking time to think about my issue (I am actually grateful
to everyone in this e-mail thread, who did). I looked into Mercurian
MQ and it doesn't seem to fit what I need - from what I understood, it
speaks about the committed changes and those are not amendable. This
is kinda not what I need.

On Tue, Jul 11, 2017 at 6:27 PM, astian <astian@eclipso.at> wrote:
> Nikolay Shustov wrote:
>> With Perforce, I can have multiple changelists opened, that group the
>> changed files as needed.
>>
>> With Git I cannot seem to finding the possibility to figure out how to
>> achieve the same result. And the problem is that putting change sets
>> on different Git branches (or workdirs, or whatever Git offers that
>> makes the changes to be NOT in the same source tree) is not a viable
>> option from me as I would have to re-build code as I re-integrate the
>> changes between the branches (or whatever changes separation Git
>> feature is used).
>> Build takes time and resources and considering that I have to do it on
>> multiple platforms (I do cross-platform development) it really
>> denominates the option of not having multiple changes in the same code
>> tree.
>>
>> Am I ignorant about some Git feature/way of using Git that would help?
>> Is it worth considering adding to Git a feature like "group of files"
>> that would offer some virtutal grouping of the locally changed files
>> in the checked-out branch?
>
> I never used Perforce and I'm not even sure I understand your problem,
> but I thought I'd mention something that nobody else seems to have yet
> (unless I missed it):
>
> First, one thing that seems obvious to me from your description is that
> these "parallel features" you work on are obviously interdependent,
> therefore I would rather consider the whole thing as a single feature.
> Therefore, it makes sense to me to work in a single "topic branch".
>
> This doesn't preclude one from separating the changes in logically
> sensible pieces.  Indeed this is par for the course in Git and people do
> it all the time by dividing the bulk of changes into a carefully chosen
> series of commits.
>
> I think the most common way of doing this is to simply work on the whole
> thing and once you're happy with it you use "git rebase --interative" in
> order to "prettify" your history.
>
> But, and here comes the part I think nobody mentioned yet, if your
> feature work is considerably large or spans a considerably long time it
> may be undesirable to postpone all that work until the very end (perhaps
> by then you already forgot important information, or perhaps too many
> changes have accumulated so reviewing them all becomes significantly
> less efficient).  In that case, one solution is to use a "patch
> management system" which will let you do that work incrementally (going
> back and forth as needed).
>
> If you know mercurial, this is "hg mq".  I don't think Git has any such
> system built-in, but I know there are at least these external tools that
> integrate with Git:
> https://git.wiki.kernel.org/index.php/Interfaces,_frontends,_and_tools#Patch-management_Interface_layers
>
> Feel free to ignore this if I totally misunderstood your use case.
>
> Cheers.
>
>

Re: "groups of files" in Git?

[Igor Djordjevic]

Hi astian,

On 12/07/2017 00:27, astian wrote:
> Nikolay Shustov wrote:
>> With Perforce, I can have multiple changelists opened, that group the
>> changed files as needed.
>>
>> With Git I cannot seem to finding the possibility to figure out how to
>> achieve the same result. And the problem is that putting change sets
>> on different Git branches (or workdirs, or whatever Git offers that
>> makes the changes to be NOT in the same source tree) is not a viable
>> option from me as I would have to re-build code as I re-integrate the
>> changes between the branches (or whatever changes separation Git
>> feature is used).
>> Build takes time and resources and considering that I have to do it on
>> multiple platforms (I do cross-platform development) it really
>> denominates the option of not having multiple changes in the same code
>> tree.
>>
>> Am I ignorant about some Git feature/way of using Git that would help?
>> Is it worth considering adding to Git a feature like "group of files"
>> that would offer some virtutal grouping of the locally changed files
>> in the checked-out branch?
> 
> I never used Perforce and I'm not even sure I understand your problem,
> but I thought I'd mention something that nobody else seems to have yet
> (unless I missed it):
> 
> First, one thing that seems obvious to me from your description is that
> these "parallel features" you work on are obviously interdependent,
> therefore I would rather consider the whole thing as a single feature.
> Therefore, it makes sense to me to work in a single "topic branch".
> 
> This doesn't preclude one from separating the changes in logically
> sensible pieces.  Indeed this is par for the course in Git and people do
> it all the time by dividing the bulk of changes into a carefully chosen
> series of commits.
> 
> I think the most common way of doing this is to simply work on the whole
> thing and once you're happy with it you use "git rebase --interative" in
> order to "prettify" your history.
> 
> But, and here comes the part I think nobody mentioned yet, if your
> feature work is considerably large or spans a considerably long time it
> may be undesirable to postpone all that work until the very end (perhaps
> by then you already forgot important information, or perhaps too many
> changes have accumulated so reviewing them all becomes significantly
> less efficient).  In that case, one solution is to use a "patch
> management system" which will let you do that work incrementally (going
> back and forth as needed).
> 
> If you know mercurial, this is "hg mq".  I don't think Git has any such
> system built-in, but I know there are at least these external tools that
> integrate with Git:
> https://git.wiki.kernel.org/index.php/Interfaces,_frontends,_and_tools#Patch-management_Interface_layers
> 
> Feel free to ignore this if I totally misunderstood your use case.
> 
> Cheers
This message actually creeped me out the first time I read it, after 
writing an e-mail reply of my own[1].

The tone it`s written in, the points you make, and even the 
conclusion about "hg mg" -- as if you were reading my mind.

Yours was sent a bit before mine, but I guess we were writing it at 
the same time as well... Just spooky, lol.

That said, I totally understand what you`re talking about, and I gave 
an example of the desired (yet missing?) Git work flow here[2] :)

[1] https://public-inbox.org/git/6e4096fd-cbab-68f0-7a23-654382cb810e@gmail.com/
[2] https://public-inbox.org/git/27a3c650-5843-d446-1f59-64fabe5434a3@gmail.com/

Regards,
Buga