An alternate model for preparing partial commits

An alternate model for preparing partial commits

[Robert Anderson]

Seems to me the concept of the "index" is a half-baked version of what
I really want, which is the ability to factor a working tree's changes
into its constituent parts in preparation for committing them.  The
index provides some very nice facilities to factor out changes in a
working tree into a "staging area", but the fundamental flaw of this
in my view is that this "staging area" is not instantiated as a tree,
so it cannot be compiled and/or tested before committing.

Consider a facility where the state you want to commit next is built
up in the current working directory, and the original set of changes
exists in some proto-space like the index currently inhabits, where
you can query and manipulate that state, but it isn't instantiated in
your working tree.

Imagine a session like this:

You've got a couple of conflated changes in your working tree, that
you think you can break up into two orthogonal changes, each of which
will compile and pass a set of tests you've got.  You think.  You'd
like to verify the build and test before you commit each piece.

git prep

where "prep" means "prepare commit".  Don't get hung up on command or
option names I'm using as placeholders, I just made that up without
much deep thought about what to call it.

Now my tree appears clean (and git diff returns nothing).  I can now
start adding the changes I had in my working tree that I want to
include in the next commit, using git add (which would know I am in
the "prep" mode).  I can examine those original working dir changes I
am choosing from with:

git diff --prep

which, at this point, shows the same output that "git diff" did before
I ran "git prep."  Now I want to add some subset of my original
changes:

git add newfile.c
git add -i
<add a couple of hunks of the changes from file modfile.c>

Now I have a working tree state that I think I want to commit.  I can
examine it with:

git diff

and I can compile and test it.  Yep, it works and passes my test suite
(an option I did not have if I had added these changes to the index).
So now I want to commit:

git commit -a -m "made change A"

I think the commit should probably "pop" the rest of the changes I did
not commit back into the working directory.  If I want to pull another
subset of changes again, I can repeat the process with another "git
prep".

Does this idea resonate with anyone else?

Thanks,
Bob Anderson

Re: An alternate model for preparing partial commits

[Björn Steinbrink]

On 2008.06.26 23:50:06 -0700, Robert Anderson wrote:
> Seems to me the concept of the "index" is a half-baked version of what
> I really want, which is the ability to factor a working tree's changes
> into its constituent parts in preparation for committing them.  The
> index provides some very nice facilities to factor out changes in a
> working tree into a "staging area", but the fundamental flaw of this
> in my view is that this "staging area" is not instantiated as a tree,
> so it cannot be compiled and/or tested before committing.
> 
> Consider a facility where the state you want to commit next is built
> up in the current working directory, and the original set of changes
> exists in some proto-space like the index currently inhabits, where
> you can query and manipulate that state, but it isn't instantiated in
> your working tree.
> 
> Imagine a session like this:
> 
> You've got a couple of conflated changes in your working tree, that
> you think you can break up into two orthogonal changes, each of which
> will compile and pass a set of tests you've got.  You think.  You'd
> like to verify the build and test before you commit each piece.
> 
> git prep
> 
> where "prep" means "prepare commit".  Don't get hung up on command or
> option names I'm using as placeholders, I just made that up without
> much deep thought about what to call it.
> 
> Now my tree appears clean (and git diff returns nothing).  I can now
> start adding the changes I had in my working tree that I want to
> include in the next commit, using git add (which would know I am in
> the "prep" mode).  I can examine those original working dir changes I
> am choosing from with:
> 
> git diff --prep
> 
> which, at this point, shows the same output that "git diff" did before
> I ran "git prep."  Now I want to add some subset of my original
> changes:
> 
> git add newfile.c
> git add -i
> <add a couple of hunks of the changes from file modfile.c>
> 
> Now I have a working tree state that I think I want to commit.  I can
> examine it with:
> 
> git diff
> 
> and I can compile and test it.  Yep, it works and passes my test suite
> (an option I did not have if I had added these changes to the index).
> So now I want to commit:
> 
> git commit -a -m "made change A"
> 
> I think the commit should probably "pop" the rest of the changes I did
> not commit back into the working directory.  If I want to pull another
> subset of changes again, I can repeat the process with another "git
> prep".
> 
> Does this idea resonate with anyone else?

Hm, I use "stash" for that purpose, which leads to kind of the reverse
of your approach. So I do sth. like this:

 - hack hack hack
 - Notice that I want to make two commits out of what I have in my
   working tree
 - git add -p -- stage what I want in the first commit
 - git commit -m tmp -- temporary commit
 - git stash -- stash away what doesn't belong in the first commit
 - git reset HEAD^ -- drop the temporary commit, with the changes kept
   in the working tree
 - test, fix bugs, read the diff, whatever
 - git commit -- this time for good
 - git stash apply -- get back the changes for the second commit

Instead of using reset, you could also use "commit --amend" (I actually
used to do that), but that needs you to do "git diff HEAD^" to see the
full changes, and (IMHO) makes it a harder sometimes to review your
stuff, because you now have three places where the changes for one
commit might reside (HEAD, index and working tree).

Björn

[PATCH/RFC] stash: introduce 'stash save --keep-index' option

[SZEDER Gábor]

'git stash save' saves local modifications to a new stash, and runs 'git
reset --hard' to revert them to a clean index and work tree.  When the
'--keep-index' option is specified, after that 'git reset --hard' the
previous contents of the index is restored and the work tree is updated
to match the index.  This option is useful if the user wants to commit
only parts of his local modifications, but wants to test those parts
before committing.

Also add support for the completion of the new option, and add an
example use case to the documentation.

Signed-off-by: SZEDER Gábor <szeder@ira.uka.de>
---

On Fri, Jun 27, 2008 at 09:10:14AM +0200, Björn Steinbrink wrote:
> Hm, I use "stash" for that purpose, which leads to kind of the reverse
> of your approach. So I do sth. like this:
> 
>  - hack hack hack
>  - Notice that I want to make two commits out of what I have in my
>    working tree
>  - git add -p -- stage what I want in the first commit
>  - git commit -m tmp -- temporary commit
>  - git stash -- stash away what doesn't belong in the first commit
>  - git reset HEAD^ -- drop the temporary commit, with the changes kept
>    in the working tree
>  - test, fix bugs, read the diff, whatever
>  - git commit -- this time for good
>  - git stash apply -- get back the changes for the second commit
I used to do the same, so I have added a '--keep-index' option to 'git
stash save' to simplify this workflow.  Have a look at the use case
added to the documentation to see, how you could spare the temporary
commit and the 'reset HEAD^'.

RFC, because I'm not quite confident with using plumbing like 'git
read-tree'...  and there are no tests.


 Documentation/git-stash.txt            |   22 +++++++++++++++++++++-
 contrib/completion/git-completion.bash |   13 ++++++++++++-
 git-stash.sh                           |   22 ++++++++++++++++++----
 3 files changed, 51 insertions(+), 6 deletions(-)

diff --git a/Documentation/git-stash.txt b/Documentation/git-stash.txt
index baa4f55..936864f 100644
--- a/Documentation/git-stash.txt
+++ b/Documentation/git-stash.txt
@@ -36,12 +36,15 @@ is also possible).
 OPTIONS
 -------
 
-save [<message>]::
+save [--keep-index] [<message>]::
 
 	Save your local modifications to a new 'stash', and run `git-reset
 	--hard` to revert them.  This is the default action when no
 	subcommand is given. The <message> part is optional and gives
 	the description along with the stashed state.
++
+If the `--keep-index` option is used, all changes already added to the
+index are left intact.
 
 list [<options>]::
 
@@ -169,6 +172,23 @@ $ git stash apply
 ... continue hacking ...
 ----------------------------------------------------------------
 
+Testing partial commits::
+
+You can use `git stash save --keep-index` when you want to make two or
+more commits out of the changes in the work tree, and you want to test
+each change before committing:
++
+----------------------------------------------------------------
+... hack hack hack ...
+$ git add --patch foo
+$ git stash save --keep-index
+$ build && run tests
+$ git commit -m 'First part'
+$ git stash apply
+$ build && run tests
+$ git commit -a -m 'Second part'
+----------------------------------------------------------------
+
 SEE ALSO
 --------
 linkgit:git-checkout[1],
diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
index ebf7cde..9a15500 100755
--- a/contrib/completion/git-completion.bash
+++ b/contrib/completion/git-completion.bash
@@ -1139,8 +1139,19 @@ _git_show ()
 _git_stash ()
 {
 	local subcommands='save list show apply clear drop pop create'
-	if [ -z "$(__git_find_subcommand "$subcommands")" ]; then
+	local subcommand="$(__git_find_subcommand "$subcommands")"
+	if [ -z "$subcommand" ]; then
 		__gitcomp "$subcommands"
+	else
+		local cur="${COMP_WORDS[COMP_CWORD]}"
+		case "$subcommand,$cur" in
+		save,--*)
+			__gitcomp "--keep-index"
+			;;
+		*)
+			COMPREPLY=()
+			;;
+		esac
 	fi
 }
 
diff --git a/git-stash.sh b/git-stash.sh
index 4938ade..92531a2 100755
--- a/git-stash.sh
+++ b/git-stash.sh
@@ -86,6 +86,13 @@ create_stash () {
 }
 
 save_stash () {
+	keep_index=
+	case "$1" in
+	--keep-index)
+		keep_index=t
+		shift
+	esac
+
 	stash_msg="$1"
 
 	if no_changes
@@ -104,6 +111,13 @@ save_stash () {
 	git update-ref -m "$stash_msg" $ref_stash $w_commit ||
 		die "Cannot save the current status"
 	printf 'Saved working directory and index state "%s"\n' "$stash_msg"
+
+	git reset --hard
+
+	if test -n "$keep_index" && test -n $i_tree
+	then
+		git read-tree --reset -u $i_tree
+	fi
 }
 
 have_stash () {
@@ -153,7 +167,8 @@ apply_stash () {
 		die "$*: no valid stashed state found"
 
 	unstashed_index_tree=
-	if test -n "$unstash_index" && test "$b_tree" != "$i_tree"
+	if test -n "$unstash_index" && test "$b_tree" != "$i_tree" &&
+			test "$c_tree" != "$i_tree"
 	then
 		git diff-tree --binary $s^2^..$s^2 | git apply --cached
 		test $? -ne 0 &&
@@ -235,7 +250,7 @@ show)
 	;;
 save)
 	shift
-	save_stash "$*" && git-reset --hard
+	save_stash "$*"
 	;;
 apply)
 	shift
@@ -268,8 +283,7 @@ pop)
 	if test $# -eq 0
 	then
 		save_stash &&
-		echo '(To restore them type "git stash apply")' &&
-		git-reset --hard
+		echo '(To restore them type "git stash apply")'
 	else
 		usage
 	fi
-- 
1.5.6.1.95.ge713

Re: [PATCH/RFC] stash: introduce 'stash save --keep-index' option

[Junio C Hamano]

SZEDER Gábor <szeder@ira.uka.de> writes:

> 'git stash save' saves local modifications to a new stash, and runs 'git
> reset --hard' to revert them to a clean index and work tree.  When the
> '--keep-index' option is specified, after that 'git reset --hard' the
> previous contents of the index is restored and the work tree is updated
> to match the index.  This option is useful if the user wants to commit
> only parts of his local modifications, but wants to test those parts
> before committing.

Please do not describe how it does, before what it does and what it is
good for.  Here is an example:

	When preparing for a partial commit (iow, committing only part of
	the changes you made in your work tree), you would use various
	forms of "git add" to prepare the index to a shape you think is
	appropriate for committing, and finally run "git commit".

	This workflow however has a flaw in that you are committing
	something that you could never have tested as a whole (and without
	any other modification) in your work tree.

	With the new "--keep-index" option, "git stash" takes a snapshot
	of your index and the work tree state, and updates the work tree
	to match your index, i.e. what you are about to commit.  This way,
	you can commit with confidence, knowing that you are committing
	what you saw (and hopefully tested) as a whole in your work tree.

	After making that initial commit, "git stash pop" will bring the
	work tree state back without touching the index, so you can start
	the next cycle of "git add" to prepare the second batch.

I do not know if your --keep-index implementation would actually allow the
above workflow; I haven't read the patch.

Re: An alternate model for preparing partial commits

[Robert Anderson]

On Fri, Jun 27, 2008 at 12:10 AM, Björn Steinbrink <B.Steinbrink@gmx.de> wrote:
> Hm, I use "stash" for that purpose, which leads to kind of the reverse
> of your approach. So I do sth. like this:
>  - hack hack hack
>  - Notice that I want to make two commits out of what I have in my
>   working tree
>  - git add -p -- stage what I want in the first commit
>  - git commit -m tmp -- temporary commit

This is a guess at the first commit?  I don't like it, but I'm still listening.

>  - git stash -- stash away what doesn't belong in the first commit
>  - git reset HEAD^ -- drop the temporary commit, with the changes kept
>   in the working tree

Now I have my guess at the first commit as my tree state, correct?
What happens when I decide I need a couple of hunks from another file
which I missed in my first guess, and is now in the stashed state?
How do I get those out of the stash and into the working tree?  If
there is no convenient way to do that, then this method is not
sufficient to cover the use case I am talking about.

Thanks,
Bob

Re: An alternate model for preparing partial commits

[Björn Steinbrink]

On 2008.06.27 09:54:43 -0700, Robert Anderson wrote:
> On Fri, Jun 27, 2008 at 12:10 AM, Björn Steinbrink <B.Steinbrink@gmx.de> wrote:
> > Hm, I use "stash" for that purpose, which leads to kind of the reverse
> > of your approach. So I do sth. like this:
> >  - hack hack hack
> >  - Notice that I want to make two commits out of what I have in my
> >   working tree
> >  - git add -p -- stage what I want in the first commit
> >  - git commit -m tmp -- temporary commit
> 
> This is a guess at the first commit?  I don't like it, but I'm still
> listening.

It's rather a work-around to stash away only the changes that are not in
the index. See the other reply to my mail for a patch that adds an
option to stash to do that without the commit/reset hack.

> >  - git stash -- stash away what doesn't belong in the first commit
> >  - git reset HEAD^ -- drop the temporary commit, with the changes kept
> >   in the working tree
> 
> Now I have my guess at the first commit as my tree state, correct?
> What happens when I decide I need a couple of hunks from another file
> which I missed in my first guess, and is now in the stashed state?
> How do I get those out of the stash and into the working tree?  If
> there is no convenient way to do that, then this method is not
> sufficient to cover the use case I am talking about.

git stash pop
eventually fix conflicts if you changed the working tree in the meantime
go back to the "git add -p" step

Björn

Re: An alternate model for preparing partial commits

[Robert Anderson]

On Fri, Jun 27, 2008 at 10:27 AM, Björn Steinbrink <B.Steinbrink@gmx.de> wrote:
>> Now I have my guess at the first commit as my tree state, correct?
>> What happens when I decide I need a couple of hunks from another file
>> which I missed in my first guess, and is now in the stashed state?
>> How do I get those out of the stash and into the working tree?  If
>> there is no convenient way to do that, then this method is not
>> sufficient to cover the use case I am talking about.
>
> git stash pop
> eventually fix conflicts if you changed the working tree in the meantime
> go back to the "git add -p" step
>
> Björn

But that pops the entire stash, right?  Inconvenient at best.

A good UI here would allow you to move pieces bidirectionally to/from
the stash at will until the desired, verifiable factorization of
changes has been achieved.

Thanks,
Bob

Re: An alternate model for preparing partial commits

[Johannes Sixt]

Robert Anderson schrieb:
> Seems to me the concept of the "index" is a half-baked version of what
> I really want, which is the ability to factor a working tree's changes
> into its constituent parts in preparation for committing them.  The
> index provides some very nice facilities to factor out changes in a
> working tree into a "staging area", but the fundamental flaw of this
> in my view is that this "staging area" is not instantiated as a tree,
> so it cannot be compiled and/or tested before committing.

I do this all the time. After I have made $N commits out of my worktree, I
usually

	$ git rebase -i HEAD~$N

and turn all 'pick's into 'edit's and 'squash's. Then I can compile and
test each commit, perhaps add some fixups, in isolation.

-- Hannes

Re: An alternate model for preparing partial commits

[Robert Anderson]

On Fri, Jun 27, 2008 at 1:35 AM, Johannes Sixt <j.sixt@viscovery.net> wrote:
> Robert Anderson schrieb:
>> Seems to me the concept of the "index" is a half-baked version of what
>> I really want, which is the ability to factor a working tree's changes
>> into its constituent parts in preparation for committing them.  The
>> index provides some very nice facilities to factor out changes in a
>> working tree into a "staging area", but the fundamental flaw of this
>> in my view is that this "staging area" is not instantiated as a tree,
>> so it cannot be compiled and/or tested before committing.
>
> I do this all the time. After I have made $N commits out of my worktree, I
> usually
>
>        $ git rebase -i HEAD~$N
>
> and turn all 'pick's into 'edit's and 'squash's. Then I can compile and
> test each commit, perhaps add some fixups, in isolation.
>
> -- Hannes

Hannes,

I do not have N commits.  I have a modified working tree from which I
would like to create N commits.  I would like to compile and test an
instantiation of each of those to-be-committed states before
committing them.

Thanks,
Bob

Re: An alternate model for preparing partial commits

[Petr Baudis]

  Hi,

On Thu, Jun 26, 2008 at 11:50:06PM -0700, Robert Anderson wrote:
> Seems to me the concept of the "index" is a half-baked version of what
> I really want, which is the ability to factor a working tree's changes
> into its constituent parts in preparation for committing them.  The
> index provides some very nice facilities to factor out changes in a
> working tree into a "staging area", but the fundamental flaw of this
> in my view is that this "staging area" is not instantiated as a tree,
> so it cannot be compiled and/or tested before committing.
> 
> Consider a facility where the state you want to commit next is built
> up in the current working directory, and the original set of changes
> exists in some proto-space like the index currently inhabits, where
> you can query and manipulate that state, but it isn't instantiated in
> your working tree.

  I wanted to suggest using commit and commit --amend, but I realized
that frankly, I don't understand quite what are you wanting to do.
Through the process, are you preparing a sequence of two commits at
once, or merely a single commit? With s/--prep/--cached/ and throwing
git prep away completely, it's not clear to me how would what you
present be different at all from just using index - could you point out
what is actually different in your workflow compared to the prep
workflow you propose?

-- 
				Petr "Pasky" Baudis
The last good thing written in C++ was the Pachelbel Canon. -- J. Olson

Re: An alternate model for preparing partial commits

[Robert Anderson]

On Fri, Jun 27, 2008 at 1:50 AM, Petr Baudis <pasky@suse.cz> wrote:
>  Hi,
>
> On Thu, Jun 26, 2008 at 11:50:06PM -0700, Robert Anderson wrote:
>> Seems to me the concept of the "index" is a half-baked version of what
>> I really want, which is the ability to factor a working tree's changes
>> into its constituent parts in preparation for committing them.  The
>> index provides some very nice facilities to factor out changes in a
>> working tree into a "staging area", but the fundamental flaw of this
>> in my view is that this "staging area" is not instantiated as a tree,
>> so it cannot be compiled and/or tested before committing.
>>
>> Consider a facility where the state you want to commit next is built
>> up in the current working directory, and the original set of changes
>> exists in some proto-space like the index currently inhabits, where
>> you can query and manipulate that state, but it isn't instantiated in
>> your working tree.
>
>  I wanted to suggest using commit and commit --amend, but I realized
> that frankly, I don't understand quite what are you wanting to do.

I think if you read the whole message the answer to your question is there.

> Through the process, are you preparing a sequence of two commits at
> once, or merely a single commit? With s/--prep/--cached/ and throwing
> git prep away completely, it's not clear to me how would what you
> present be different at all from just using index - could you point out
> what is actually different in your workflow compared to the prep
> workflow you propose?

As I said in my original message, twice, was that the index state is
never instantiated and therefore cannot be compiled or tested.

Thanks,
Bob

Re: An alternate model for preparing partial commits

[Johannes Schindelin]

Hi,

On Thu, 26 Jun 2008, Robert Anderson wrote:

> Seems to me the concept of the "index" is a half-baked version of what
> I really want, which is the ability to factor a working tree's changes
> into its constituent parts in preparation for committing them.

Half-baked is probably too strong a word.

What you are basically asking for is to have the working directory 
as staging area, and to be able to stash away changes that are not to be 
committed.

Now, this is not necessarily what everybody wants, which is why many 
people are fine with the index.

Having said that, I played with the idea of a "git stash -i", which would 
allow you to select the changes to stash away.  (And by extension, "git 
stash -e" using the "git add -e" command.)

Hmm?

Ciao,
Dscho

Re: An alternate model for preparing partial commits

[Miklos Vajna]

[-- Attachment #1: Type: text/plain, Size: 921 bytes --]

On Fri, Jun 27, 2008 at 02:33:33PM +0100, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> Having said that, I played with the idea of a "git stash -i", which would 
> allow you to select the changes to stash away.  (And by extension, "git 
> stash -e" using the "git add -e" command.)

If we are at it, git checkout -i is also something which may be useful,
like:

1) Do two unrelated changes in a file.

2) You realize one of them is unnecessary.

Currently what you can do is something like:

1) You stage the first hunk using git add -p

2) git commit

3) git checkout file

But this forces you to commit early, and to commit --amend later. It
would be nice to be able to completely drop a hunk without first
commiting.

(Feel free to point out if this is something bad, I just remember from
the past that darcs revert - which is like git checkout - had such an
interactive mode.)

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: An alternate model for preparing partial commits

[Robert Anderson]

On Fri, Jun 27, 2008 at 6:33 AM, Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
> On Thu, 26 Jun 2008, Robert Anderson wrote:
>
>> Seems to me the concept of the "index" is a half-baked version of what
>> I really want, which is the ability to factor a working tree's changes
>> into its constituent parts in preparation for committing them.
>
> Half-baked is probably too strong a word.

It is too subtle.  That the index state - which becomes the next
committed state - is not available for building or testing before
committing is a deep flaw.

> What you are basically asking for is to have the working directory
> as staging area, and to be able to stash away changes that are not to be
> committed.

Correct.

> Now, this is not necessarily what everybody wants, which is why many
> people are fine with the index.

But it is something they should want, and should have, if they care
about the quality of their commits.  Especially in the common case of
a project with development lines which have some sort of policy about
build/test requirements.  How do you ensure your commits obey that
policy if you cannot verify it?  That is why the index is not a
sufficient mechanism for preparing partial commits.  It's fine for
quick and dirty operation when the factorization of the conflated
changes is obvious and trivial.  It is not sufficient otherwise.

> Having said that, I played with the idea of a "git stash -i", which would
> allow you to select the changes to stash away.  (And by extension, "git
> stash -e" using the "git add -e" command.)
>
> Hmm?

I meant to mention that - at least in the model I described - this has
some overlap with "stash" and could possibly be folded into it.

In my ideal UI, changes (from all changes to hunk level) could be
moved back and forth between the stash and the working tree equally
easily.  Would git stash -i allow that?

For example, if I moved a couple of files into the stash, and then
realized I needed one hunk back, could I easily retrieve just that
from the stash?

Thanks,
Bob

Re: An alternate model for preparing partial commits

[Johannes Schindelin]

Hi,

On Fri, 27 Jun 2008, Robert Anderson wrote:

> On Fri, Jun 27, 2008 at 6:33 AM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
>
> > On Thu, 26 Jun 2008, Robert Anderson wrote:
> >
> >> Seems to me the concept of the "index" is a half-baked version of 
> >> what I really want, which is the ability to factor a working tree's 
> >> changes into its constituent parts in preparation for committing 
> >> them.
> >
> > Half-baked is probably too strong a word.
> 
> It is too subtle.  That the index state - which becomes the next 
> committed state - is not available for building or testing before 
> committing is a deep flaw.
> 
> > Now, this is not necessarily what everybody wants, which is why many 
> > people are fine with the index.
> 
> But it is something they should want, and should have, if they care
> about the quality of their commits.

This is too narrow-minded a view for me.

No longer interested,
Dscho

Re: An alternate model for preparing partial commits

[Robert Anderson]

On Fri, Jun 27, 2008 at 10:45 AM, Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
> On Fri, 27 Jun 2008, Robert Anderson wrote:
>
>> On Fri, Jun 27, 2008 at 6:33 AM, Johannes Schindelin
>> <Johannes.Schindelin@gmx.de> wrote:
>>
>> > On Thu, 26 Jun 2008, Robert Anderson wrote:
>> >
>> >> Seems to me the concept of the "index" is a half-baked version of
>> >> what I really want, which is the ability to factor a working tree's
>> >> changes into its constituent parts in preparation for committing
>> >> them.
>> >
>> > Half-baked is probably too strong a word.
>>
>> It is too subtle.  That the index state - which becomes the next
>> committed state - is not available for building or testing before
>> committing is a deep flaw.
>>
>> > Now, this is not necessarily what everybody wants, which is why many
>> > people are fine with the index.
>>
>> But it is something they should want, and should have, if they care
>> about the quality of their commits.
>
> This is too narrow-minded a view for me.
>
> No longer interested,
> Dscho
>

Here's a patch to match the local culture:  "It is incredible how
stupid the idea of the index is."

Clearly you should now be interested.

Thanks,
Bob

Re: An alternate model for preparing partial commits

[Robert Anderson]

On Fri, Jun 27, 2008 at 10:54 AM, Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
> On Fri, 27 Jun 2008, Robert Anderson wrote:
>
>> Here's a patch to match the local culture:  "It is incredible how stupid
>> the idea of the index is."
>>
>> Clearly you should now be interested.
>
> No, you're wrong.  I am totall uninterested in talking to you now.
>

Tongue in cheek, of course - a play on Linus' "narrow minded views"
about svn.  Not much of a sense of humor, eh?

You may consider the idea that the next committed state ought to be
available for build/test to be a "narrow-minded" view.  I find it to
be a manifestly sound view.  In fact, if you disagree I have little
respect for your technical judgment and therefore am not concerned if
you will not talk with me.  Which is too bad on both counts.

Thanks,
Bob

Re: An alternate model for preparing partial commits

[Dana How]

On Fri, Jun 27, 2008 at 10:49 AM, Robert Anderson <rwa000@gmail.com> wrote:
> On Fri, Jun 27, 2008 at 10:45 AM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
>> Hi,
>>
>> On Fri, 27 Jun 2008, Robert Anderson wrote:
>>
>>> On Fri, Jun 27, 2008 at 6:33 AM, Johannes Schindelin
>>> <Johannes.Schindelin@gmx.de> wrote:
>>>
>>> > On Thu, 26 Jun 2008, Robert Anderson wrote:
>>> >
>>> >> Seems to me the concept of the "index" is a half-baked version of
>>> >> what I really want, which is the ability to factor a working tree's
>>> >> changes into its constituent parts in preparation for committing
>>> >> them.
>>> >
>>> > Half-baked is probably too strong a word.
>>>
>>> It is too subtle.  That the index state - which becomes the next
>>> committed state - is not available for building or testing before
>>> committing is a deep flaw.
>>>
>>> > Now, this is not necessarily what everybody wants, which is why many
>>> > people are fine with the index.
>>>
>>> But it is something they should want, and should have, if they care
>>> about the quality of their commits.
>>
>> This is too narrow-minded a view for me.
>>
>> No longer interested,
>> Dscho
>>
>
> Here's a patch to match the local culture:  "It is incredible how
> stupid the idea of the index is."
>
> Clearly you should now be interested.
>
> Thanks,
> Bob

I guess I'm not interested in the over-generalizations.  ;-)

But the ability to use e.g. some stash-like feature (as suggested above)
to easily make the index-state (about to be committed) fully available
for compiling/processing/testing without losing edits not yet ready
for commit is an extra feature we would use here at least some of the time.
I will admit it's currently not the "itch" at the top of my list.

Thanks,
-- 
Dana L. How danahow@gmail.com +1 650 804 5991 cell

Re: An alternate model for preparing partial commits

[Stephen Sinclair]

Hello,

On Fri, Jun 27, 2008 at 1:14 PM, Robert Anderson <rwa000@gmail.com> wrote:
> On Fri, Jun 27, 2008 at 6:33 AM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
>> Now, this is not necessarily what everybody wants, which is why many
>> people are fine with the index.
>
> But it is something they should want, and should have, if they care
> about the quality of their commits.  Especially in the common case of
> a project with development lines which have some sort of policy about
> build/test requirements.  How do you ensure your commits obey that
> policy if you cannot verify it?  That is why the index is not a
> sufficient mechanism for preparing partial commits.  It's fine for
> quick and dirty operation when the factorization of the conflated
> changes is obvious and trivial.  It is not sufficient otherwise.

I just thought I'd throw in my $0.02 here.  There's something
fundamental I think I'm not getting about this argument: it seems to
be based on the premise that partial commits allow untested tree
states to enter the repository.  However, having gotten used to the
git way of things, I personally don't see the problem with allowing
bad commits, as long as they are not pushed to public.

That is, I use the git add -p command all the time when I realize I've
just done two things that should be committed separately.  Then I'll
git commit everything else, and go back and test, like so:

git checkout master
[hack hack..]
git add -p
git commit
git commit -a
[test..]
git checkout master^
[test..]
git checkout master

See? All tested.  If I find a problem during testing, I'll probably
commit it and then rebase master off my new commit, fixing any
conflicts I just introduced.

Frankly I hardly even use the stash; since history can be edited, I
feel like commits are all I need when working with git.  Anything I do
doesn't matter until I push to public anyways.  So it's up to me to
make sure I test everything before pushing, but otherwise I'm very
happy with the ability to commit half-baked ideas and then go back to
make sure they are usable (and tested!) before pushing.  This is what
local branches are for, isn't it?


Steve

Re: An alternate model for preparing partial commits

[David Jeske]

-- Stephen Sinclair wrote:
> However, having gotten used to the git way of things, I
> personally don't see the problem with allowing
> bad commits, as long as they are not pushed to public.
Of course developers need to be able to make bad-commits. If they can't, we
quickly end up back in the "don't checkin" mentality of CVS. I want to be able
to do bad-commits. Further, I recognize that even with the best of intentions
bad-commits will enter the mainline. What I think is missing is a mechanism for
re-summarizing a set of committs that does not rely on the fact that they've
never been published. See my other email on "supercede"..

Consider a big public tree that is using bisect, but discovers that there is a
bad commit in the mainline. I have no idea what you would do with current git
to fix this. However, if I could pick two endpoints and "supercede" them, I
would have a bunch of options:

- I could supercede 2-commits with 1, effectively making the bad-commit
disappear in the linear history. Users who already have the history, however,
would be unaffected, because the start/end endpoints are the same.

- I could supercede 2-commits with two different commits, one that makes the
test pass correctly, and one that trues-it-up with the next patch, that somehow
made all the tests pass correctly again. (possibly by hoisting a diff out of
the 2nd commit that actually fixed the first issue)

While repairing an old test build problem may seem like an infrequent issue,
this mechanism has a much more frequent and more important benefit... which is
that people can share changes and then later rebase them -- and others who have
a copy of that branch will receive automatic merge/rebase support. (assuming
there are not reasons for conflict)

Re: An alternate model for preparing partial commits

[Wincent Colaiuta]

El 27/6/2008, a las 22:45, David Jeske escribió:

> - I could supercede 2-commits with 1, effectively making the bad- 
> commit
> disappear in the linear history. Users who already have the history,  
> however,
> would be unaffected, because the start/end endpoints are the same.


You can't do that. The start might be the same but the end point would  
not, even if the file contents happened to be the same as they were  
before. This is because each commit is identified by its SHA-1 hash,  
and changing any ancestor anywhere in the DAG will have a trickle-down  
effect to all subsequent commits which stem from it, changing their  
SHA-1 hashes too. So you wind up with a different end point. This is  
by design (see below).

El 27/6/2008, a las 22:51, David Jeske escribió:

> What git can't do, is let me "supercede" the old DAG-subset, so  
> people I shared
> them with can get my new changes without hurting their world.  
> Currently git
> seems to rely on the idea that "if you accept changes into your tree  
> that will
> be later rebased, it's up to you to figure it out". I don't see why  
> that is the
> case.

It has to be this way because it is the only way to do distributed  
SCM. Imagine for a minute that one developer has a commit "feedface",  
fetched from another repo. If the commit or any parent (grand parent,  
great grandparent etc) is changed then the hash _has_ to be different.  
Otherwise you would have the untenable situation that two developers,  
two repos, could refer to a particular commit "feedface" and it  
wouldn't actually be the same thing. Distributed development would be  
unworkably flakey if you couldn't rely on the stability of each  
commit's SHA-1 hash, and that fact that it embodies not just that  
particular state in the history, but all prior states too.

It's a totally basic, fundamental, central, non-negotiable concept to  
Git's design. And not just an idiosyncrasy either; if you do a review  
of other distributed SCMs out there you'll find that _all_ of them  
have a similar underlying precept, as it's literally the only way to  
do working distributed SCM.

Wincent

Re: An alternate model for preparing partial commits

[David Jeske]

-- Stephen Sinclair wrote:
> However, having gotten used to the git way of things, I
> personally don't see the problem with allowing
> bad commits, as long as they are not pushed to public.
Of course developers need to be able to make bad-commits. If they can't, we
quickly end up back in the "don't checkin" mentality of CVS. I want to be able
to do bad-commits. Further, I recognize that even with the best of intentions
bad-commits will enter the mainline. What I think is missing is a mechanism for
re-summarizing a set of committs that does not rely on the fact that they've
never been published. See my other email on "supercede"..

Consider a big public tree that is using bisect, but discovers that there is a
bad commit in the mainline. I have no idea what you would do with current git
to fix this. However, if I could pick two endpoints and "supercede" them, I
would have a bunch of options:

- I could supercede 2-commits with 1, effectively making the bad-commit
disappear in the linear history. Users who already have the history, however,
would be unaffected, because the start/end endpoints are the same.

- I could supercede 2-commits with two different commits, one that makes the
test pass correctly, and one that trues-it-up with the next patch, that somehow
made all the tests pass correctly again. (possibly by hoisting a diff out of
the 2nd commit that actually fixed the first issue)

While repairing an old test build problem may seem like an infrequent issue,
this mechanism has a much more frequent and more important benefit... which is
that people can share changes and then later rebase them -- and others who have
a copy of that branch will receive automatic merge/rebase support. (assuming
there are not reasons for conflict)

Re: An alternate model for preparing partial commits

[Dmitry Potapov]

On Fri, Jun 27, 2008 at 10:14:05AM -0700, Robert Anderson wrote:
> 
> It is too subtle.  That the index state - which becomes the next
> committed state - is not available for building or testing before
> committing is a deep flaw.

And why is that? It is like saying that any editor that does not allow
you to compile the file without saving it first has a deep flaw.

In Git, commits are not the same as in CVS. You can commit any changes
and amend them any time later before you publish your changes. So, what
you normally do is to commit your changes and then run test on them.
The advantage of this approach is that you can continue to your normal
work while test are running, besides the tests are running in clear and
control environment, not in the developer working director where always
there is some garbage.

> > Now, this is not necessarily what everybody wants, which is why many
> > people are fine with the index.
> 
> But it is something they should want, and should have, if they care
> about the quality of their commits.

Those who care about quality should have a review process, and the
review process works best when all changes are slit in a small logical
steps. How do you propose to that without committing changes first?

> Especially in the common case of
> a project with development lines which have some sort of policy about
> build/test requirements.  How do you ensure your commits obey that
> policy if you cannot verify it?

You can verify it, but you do that _after_ you committed changes but
before you publish them. BTW, policy may include that it should be
compiled and tested on a few platforms, so you cannot do that in
your working directory anyway.

I think the source of your confusion is that you consider git commit
as cvs commit, while git commit in some sense may be closer to saving
files, while a better analogue for cvs commit will be git push to a
public repository.

Dmitry

Re: An alternate model for preparing partial commits

[Robert Anderson]

On Fri, Jun 27, 2008 at 7:14 PM, Dmitry Potapov <dpotapov@gmail.com> wrote:
> On Fri, Jun 27, 2008 at 10:14:05AM -0700, Robert Anderson wrote:
>>
>> It is too subtle.  That the index state - which becomes the next
>> committed state - is not available for building or testing before
>> committing is a deep flaw.
>
> And why is that? It is like saying that any editor that does not allow
> you to compile the file without saving it first has a deep flaw.

I don't believe it is like that.  It would be like that if you
intended for your on-tree disk to have a policy to always compile (for
example).  That is not a common use-case.

> In Git, commits are not the same as in CVS.

I have not suggested they were.

> You can commit any changes
> and amend them any time later before you publish your changes.

This is enforcing a two-step process where there only need be one the
vast majority of the time, to require that commit and publish be
separate operations.

> Those who care about quality should have a review process, and the
> review process works best when all changes are slit in a small logical
> steps. How do you propose to that without committing changes first?

I don't understand the question.  The entire point of the facility I
am proposing is to facilitate creating small clean changesets.  Go
back and read my original proposal, or Junio's statement of more or
less the same idea, to see how it is proposed to do this.

> You can verify it, but you do that _after_ you committed changes but
> before you publish them.

Again, this is requiring two steps when it is otherwise not required,
creating an inefficient workflow that is error prone.

> BTW, policy may include that it should be
> compiled and tested on a few platforms, so you cannot do that in
> your working directory anyway.

Huh?  I have done that every day for 15+ years.

> I think the source of your confusion is that you consider git commit
> as cvs commit

No.  I have experience with a wide array of source control systems.
CVS fits my mental model the least well.  git is pretty close, but it
is not there yet.  The current partial commit facility is the biggest
misfire, in my view.

I like that git's philosophy does not include a draconian policy of
not changing history.  That's fine, it's practical, and it's useful,
to cover common scenarios in which you'd like to quickly recover from
a mistake.  However, I am afraid that these facilities have been
abused and turned into something that they are not well-suited for,
i.e., the use of lines of development as both keepers of history and
of scratch spaces where you scribble around with temporary things, all
the while git having no clue which is intended.  That these ideas are
conflated is a mistake.  That's my opinion.  These activities ought to
occur in separate, logically distinct spaces in which they occur,
because they have different requirements and common use-cases.

Bob

Re: An alternate model for preparing partial commits

[Dmitry Potapov]

On Fri, Jun 27, 2008 at 07:57:02PM -0700, Robert Anderson wrote:
> On Fri, Jun 27, 2008 at 7:14 PM, Dmitry Potapov <dpotapov@gmail.com> wrote:
> > On Fri, Jun 27, 2008 at 10:14:05AM -0700, Robert Anderson wrote:
> >>
> >> It is too subtle.  That the index state - which becomes the next
> >> committed state - is not available for building or testing before
> >> committing is a deep flaw.
> >
> > And why is that? It is like saying that any editor that does not allow
> > you to compile the file without saving it first has a deep flaw.
> 
> I don't believe it is like that.  It would be like that if you
> intended for your on-tree disk to have a policy to always compile (for
> example).  That is not a common use-case.

Do you think commit only tested changes is a common policy among
Git users? I seriously doubt that. And as I said before, git commit
is probably closer to saving a file than to what cvs commit does.


> > You can commit any changes
> > and amend them any time later before you publish your changes.
> 
> This is enforcing a two-step process where there only need be one the
> vast majority of the time, to require that commit and publish be
> separate operations.

I don't see it is a problem. In fact, it saves time for me, because
I can work while tests are running, and considering the whole cycle
with different configurations may take 3 hours, I really want to do
something useful while it is running... Besides, I really believe
in review, so it cannot be one-step process anyway...

> 
> > Those who care about quality should have a review process, and the
> > review process works best when all changes are slit in a small logical
> > steps. How do you propose to that without committing changes first?
> 
> I don't understand the question. 

If you use CVS like workflow, you may have the policy of no commit
until your patches are reviewed. In case of Git, you do commit but
only push to fast-forward only branch after receiving okay (or the
maintainer pull your changes to it). With Git, commit is more like
saving file, except that you save not a single file but the whole
changeset with the commit message and relevant information.

> > You can verify it, but you do that _after_ you committed changes but
> > before you publish them.
> 
> Again, this is requiring two steps when it is otherwise not required,
> creating an inefficient workflow that is error prone.

I don't see why you think that is inefficient or error prone.

> 
> > BTW, policy may include that it should be
> > compiled and tested on a few platforms, so you cannot do that in
> > your working directory anyway.
> 
> Huh?  I have done that every day for 15+ years.

You have your working directory let's say on Linux, and you have to
test your changes on Windows. So what do you do? With Git, it is
simple as you commit your changes and then run testing automatically
on different platforms and they use exactly what you put into the
repo. So if everything is okay, you can publish.

> 
> > I think the source of your confusion is that you consider git commit
> > as cvs commit
> 
> No.  I have experience with a wide array of source control systems.
> CVS fits my mental model the least well.  git is pretty close, but it
> is not there yet.  The current partial commit facility is the biggest
> misfire, in my view.

Do you have your personal experience of using it, or it is just some
abstract considerations? Certainly, any feature may be misused but, in
general, it is handy and I have not had any problem with it. And, yes,
if I split a very complex patch then I will use stash to facilitate
me with this work, but it is rare.

> I like that git's philosophy does not include a draconian policy of
> not changing history.  That's fine, it's practical, and it's useful,
> to cover common scenarios in which you'd like to quickly recover from
> a mistake.  However, I am afraid that these facilities have been
> abused and turned into something that they are not well-suited for,
> i.e., the use of lines of development as both keepers of history and
> of scratch spaces where you scribble around with temporary things, all
> the while git having no clue which is intended.

I don't see why git should know it. The policy depends on workflow
and is usually enforced by hooks. I don't see why Git should care
about it deeply. It is like a word processor can be used for writing
a draft of a document or the final version for publication. Sure,
you can mark something as work-in-progress (use tags or comments),
but it is not something about Git should care deeply inside.

Dmitry

Fwd: An alternate model for preparing partial commits

[Robert Anderson]

On Fri, Jun 27, 2008 at 9:03 PM, Dmitry Potapov <dpotapov@gmail.com> wrote:
> Do you think commit only tested changes is a common policy among
> Git users?

I think there are various flavors of "tested" for which the answer is
certainly yes.  I think that a line of development which always
builds, for example, is very, very common.

> And as I said before, git commit
> is probably closer to saving a file than to what cvs commit does.

That is putting scm into the development process at a level which is
far too intrusively fine-grained for my workflow, and for the workflow
of any developer that I work with, which is dozens.  I do not need to
record every "save of file."  That is useless and intrusive to me.  It
is, however, useful to me to record (commit) at points significantly
more coarse grained than that:  points which represent logically
distinct changes to the code that compile, for example.  Until then I
prefer to forget about my scm and have it stay out of my way until it
is needed to create those desired recorded states.

>> This is enforcing a two-step process where there only need be one the
>> vast majority of the time, to require that commit and publish be
>> separate operations.
>
> I don't see it is a problem.

Fine, you like doing extra work for no benefit.  Enjoy yourself.

> I don't see why you think that is inefficient or error prone.

Plainly not.  This is why I will construct some examples.

> You have your working directory let's say on Linux, and you have to
> test your changes on Windows. So what do you do?

I rebuild on about a dozen platforms simultaneously on a cross-mounted disk.

>> No.  I have experience with a wide array of source control systems.
>> CVS fits my mental model the least well.  git is pretty close, but it
>> is not there yet.  The current partial commit facility is the biggest
>> misfire, in my view.
>
> Do you have your personal experience of using it

Yes.

> I don't see why git should know it.

Clearly not.  Example 1:  is it ok to publish?  If temporary commits
exist, git should stop you.  But git has no idea if temporary commits
exist or not.

Bob

Re: Fwd: An alternate model for preparing partial commits

[Dmitry Potapov]

On Fri, Jun 27, 2008 at 11:31:24PM -0700, Robert Anderson wrote:
> On Fri, Jun 27, 2008 at 9:03 PM, Dmitry Potapov <dpotapov@gmail.com> wrote:
> > Do you think commit only tested changes is a common policy among
> > Git users?
> 
> I think there are various flavors of "tested" for which the answer is
> certainly yes.  I think that a line of development which always
> builds, for example, is very, very common.

Well, I don't think it is very common. Almost always builds, yes. As I
said before, with Git commit means more like saving your changes in
logical steps. Making sure that they are good enough to be publish is
done later, and it usually includes testing and may include something
else like code review, and code review is much easier if your work is
recorded in small logical steps, and when you can share these changes
easily and continue your normal work while your changes are reviewed.

> >> This is enforcing a two-step process where there only need be one the
> >> vast majority of the time, to require that commit and publish be
> >> separate operations.
> >
> > I don't see it is a problem.
> 
> Fine, you like doing extra work for no benefit.  Enjoy yourself.

I don't see where you find this extra work. Maybe, pressing a button
to save the file is also extra work? As to benefit, they are real for
me as it saves a lot of time and allows me to work more natural, in
the same way as you edit document. Your first version of document
does not have to be perfect, you can review and improve it later,
instead you focus on main ideas you want to express, and later, you
will proofread and rearrange things, but it is very important to
being able to save your changes even if they are not perfect yet.
Extra work like pressing the save button is negligible comparing
to everything else.

> > You have your working directory let's say on Linux, and you have to
> > test your changes on Windows. So what do you do?
> 
> I rebuild on about a dozen platforms simultaneously on a cross-mounted disk.

Does it work for MS Visual Studio? The last time I tried, it did not
allow to build the project on a shared disk. Anyway, your compilation
time is going to be worse due to network latency, and simultaneously
building on different platforms will be more difficult to organize.
With Git, you get everything for free. You committed your changes
and they immediately available everywhere, and when all tests passed
you can push changes to the main branch, or send to the integrator a
request to pull, or to send a request to code review. It all depends
on your policy...

> > I don't see why git should know it.
> 
> Clearly not.  Example 1:  is it ok to publish?  If temporary commits
> exist, git should stop you.  But git has no idea if temporary commits
> exist or not.

There are no such thing as temporary commits in Git. There are changes
that are ready to be accepted to the main branch and those that are not.
It could be different reasons why these changes cannot be integrated to
the main branch now.  Maybe, you should receive feedback from someone
who does code review your changes.  Maybe, these changes deem too risky
to be integrated now. Maybe, they will clash with someone else's work,
which has a higher priority, maybe something else.  So, I don't see how
you expect Git to know all of that. It is like to expect that your word
processor to know what whether your document is good work publication or
not, and don't let you to send the document out if it is not.

Dmitry

Re: An alternate model for preparing partial commits

[Robert Anderson]

On Sat, Jun 28, 2008 at 5:35 AM, Dmitry Potapov <dpotapov@gmail.com> wrote:
>> Fine, you like doing extra work for no benefit.  Enjoy yourself.
>
> I don't see where you find this extra work.

You aren't trying to.  You're knee-jerk defending the status quo, as
is the case of 95% of reply mail sent to any SCM mailing list.

I worked in this model for years.  It is inconvenient and creates
redundant work.  It is a PITA.

Thanks,
Bob

Re: An alternate model for preparing partial commits

[Dmitry Potapov]

On Sat, Jun 28, 2008 at 08:53:04AM -0700, Robert Anderson wrote:
> On Sat, Jun 28, 2008 at 5:35 AM, Dmitry Potapov <dpotapov@gmail.com> wrote:
> >> Fine, you like doing extra work for no benefit.  Enjoy yourself.
> >
> > I don't see where you find this extra work.
> 
> You aren't trying to.  You're knee-jerk defending the status quo, as
> is the case of 95% of reply mail sent to any SCM mailing list.

Nice... and you are an arrogant and narrow-minded individual who
believes that he knows better anyone else, but you are incapable to
produce neither code nor any solid logical argument to support your
view. So you limit yourself to verbal masturbation about some Git
deficiency and blah-blah-blah, which no one but you can notice here.
And then the workflow that you call you call as "inconvenient and
creates redundant work" produced Git, which has become self-hosted
in three days and has been developed very rapidly ever since then.
You can do better with your workflow? Prove it! Until then, I have
no interest in any further communication on this subject with you.

Thanks,
Dmitry

Re: An alternate model for preparing partial commits

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> On Thu, 26 Jun 2008, Robert Anderson wrote:
>
>> Seems to me the concept of the "index" is a half-baked version of what
>> I really want, which is the ability to factor a working tree's changes
>> into its constituent parts in preparation for committing them.
>
> Half-baked is probably too strong a word.
>
> What you are basically asking for is to have the working directory 
> as staging area, and to be able to stash away changes that are not to be 
> committed.
>
> Now, this is not necessarily what everybody wants, which is why many 
> people are fine with the index.

I've always said that I am not in favor of any form of partial commits,
exactly for the reason Robert states, namely that you are not committing
what you had in your work tree as a whole.  I said so back when the only
form of partial commits were "git commit [-o] this-file".  I said it again
even when I introduced "add -i", that the interface goes backwards and
does not fix the issues associated with partial commits.

But I agree with you that calling the index half-baked is missing the
point.  The index is merely the lowest level of facility to stage what is
to be committed, and there is no half nor full bakedness to it.  The way
the current Porcelain layer uses it however could be improved and Robert
is allowed to call _that_ half-baked when he is in a foul mood (even then
I would rather prefer people to be civil on this list).

So I would welcome constructive proposals to make things better.

But before going into the discussion, to be fair, I would mention that
people who are used to partial commits (perhaps inherited from their
CVS/SVN habit) defend the practice by saying that they will want to make
commit series first (with unproven separation between commit boundaries
that is inherent to the practice of making partial commits) and it is not
problem for them that their commits are not tested at commit time, because
they will test each step afterwards after they are done committing.  They
can fix things up later with "rebase -i" if they find glitches.

The defense makes sense from the workflow point of view, in that batching
things up tends to make people more productive.  You think of the logical
separation first and make commits without having to wait for each step to
build and test (otherwise your train of thought would be interrupted), and
then you test the final resulting sequence as a separate phase.  Although
I imagine I would personally not be able to work that way comfortably, I
consider this a personal preference issue, and if some people are more
productive to work that way, it is fine to support the workflow.

But that is not a reason not to support other workflows.

> Having said that, I played with the idea of a "git stash -i", which would 
> allow you to select the changes to stash away.

I would actually go the other way.  I think the problem we are trying to
solve here in this thread is to support this (other) workflow:

    You keep working, and eventually build all the changes intermixed in
    your work tree, perhaps without any commit, or perhaps with a commit
    sequence that is only meant as snapshots and not as a logical
    sequence.  Your work tree state is in good shape right now (you do
    build and test at this "commit goal" state).  Now you would want to
    split the changes while making sure each step is good (i.e. builds and
    tests fine as well as the patch makes sense standalone).

One thing I think would make sense is to stash away _everything_ at this
point.  That would take you to the state before you started working.  Then
if we can selectively _unstash_ the parts that should logically be
committed first to bring them to your work tree, then you can inspect that
change against HEAD, test it, and when you are happy with it, you would
make your first commit in the final sequence.

Once you have capability to unstash selectively and make that first commit
in the final sequence like so, breaking up the remainder that is still in
your stash to a reasonable sequence of commits can be done with the same
workflow.  Unstash the next batch, inspect, test and be satisfied and then
commmit.  Lather, rinse and repeat.

Re: An alternate model for preparing partial commits

[Robert Anderson]

On Fri, Jun 27, 2008 at 11:15 AM, Junio C Hamano <gitster@pobox.com> wrote:
> I've always said that I am not in favor of any form of partial commits,
> exactly for the reason Robert states, namely that you are not committing
> what you had in your work tree as a whole.  I said so back when the only
> form of partial commits were "git commit [-o] this-file".  I said it again
> even when I introduced "add -i", that the interface goes backwards and
> does not fix the issues associated with partial commits.

We are seeing eye-to-eye here.

> But I agree with you that calling the index half-baked is missing the
> point.

What I said is that the index is a half-baked version of _what I
want_, which is the ability to do partial commits from testable
states.  Clearly the index is a way to do partial commits, but it does
not address the "untested state" issue, so it is only halfway there,
i.e., half-baked.

> The index is merely the lowest level of facility to stage what is
> to be committed, and there is no half nor full bakedness to it.  The way
> the current Porcelain layer uses it however could be improved and Robert
> is allowed to call _that_ half-baked when he is in a foul mood (even then
> I would rather prefer people to be civil on this list).

Fair enough.  I did not mean "half baked" to be particular provocative, fwiw.

> I would actually go the other way.  I think the problem we are trying to
> solve here in this thread is to support this (other) workflow:
>
>    You keep working, and eventually build all the changes intermixed in
>    your work tree, perhaps without any commit, or perhaps with a commit
>    sequence that is only meant as snapshots and not as a logical
>    sequence.  Your work tree state is in good shape right now (you do
>    build and test at this "commit goal" state).  Now you would want to
>    split the changes while making sure each step is good (i.e. builds and
>    tests fine as well as the patch makes sense standalone).
>
> One thing I think would make sense is to stash away _everything_ at this
> point.  That would take you to the state before you started working.  Then
> if we can selectively _unstash_ the parts that should logically be
> committed first to bring them to your work tree, then you can inspect that
> change against HEAD, test it, and when you are happy with it, you would
> make your first commit in the final sequence.

Exactly.  That was a good summary of the workflow I proposed in my
original email.

> Once you have capability to unstash selectively and make that first commit
> in the final sequence like so, breaking up the remainder that is still in
> your stash to a reasonable sequence of commits can be done with the same
> workflow.  Unstash the next batch, inspect, test and be satisfied and then
> commmit.  Lather, rinse and repeat.

Bingo.

Time permitting, I will propose a more well thought through UI for
this workflow.  If you like it, we can talk about how it might be
implemented.

Thanks,
Bob

Re: An alternate model for preparing partial commits

[Jeff King]

On Fri, Jun 27, 2008 at 11:15:44AM -0700, Junio C Hamano wrote:

> I've always said that I am not in favor of any form of partial commits,
> exactly for the reason Robert states, namely that you are not committing
> what you had in your work tree as a whole.  I said so back when the only
> form of partial commits were "git commit [-o] this-file".  I said it again
> even when I introduced "add -i", that the interface goes backwards and
> does not fix the issues associated with partial commits.

I do partial commits all the time. I used to use the "go back and clean
up and test each" method. But now with stash, I use the workflow
mentioned elsewhere in the thread:

  1. hack hack hack
  2. add -i; commit -m tmp
  3. stash
  4. reset HEAD^

which is really kind of awkward, since I didn't want to make that commit
in the first place.

I think of it as three ordered buckets: index, working tree, and stash.
You can move changes from any bucket to an adjacent bucket, but you can
only test what's in the working tree.

So we have too many changes in the working tree. Right now we can put
some of them in the index bucket. But that doesn't help with testing.
So what I do now is move good changes to the index bucket (and then
commit -- more on that in a second), and then everything else to the
stash bucket, and then reset the commit.

The extra commit / reset is annoying. We could do better than that
with the proposed "stash --keep-index". Then I just have to move my good
changes into the index and say "ok, now stash everything else."

But that still involves two bucket moves. I think what would be much
more natural is to simply say "I don't want these changes right now;
move them into the stash bucket." And Dscho mentioned "git stash -i",
which does that (and theoretically, it could even be based on the "git
add -i" code).

What you propose below accomplishes the same thing, but seems mentally
reversed to me (and involves two bucket moves):

> One thing I think would make sense is to stash away _everything_ at this
> point.  That would take you to the state before you started working.  Then
> if we can selectively _unstash_ the parts that should logically be
> committed first to bring them to your work tree, then you can inspect that
> change against HEAD, test it, and when you are happy with it, you would
> make your first commit in the final sequence.

Here we say "OK, I don't want any of these changes; stash them" and then
selectively bring them back. And maybe that _is_ what you want
sometimes, or maybe how certain people think. But personally, given two
sets of changes, what makes sense to me is to directly say "these are
the changes I _don't_ want". And at any point after ditching some
changes, you can re-run your tests.

So I fundamentally think that all of these contortions are because
moving things to the stash bucket is not as featureful as moving them to
the index bucket. And there's no reason for it, since we can use the
_same_ tools to do both.

Here's a somewhat hackish implementation of "git stash -i" that just relies
on "add -i":

---
diff --git a/git-stash.sh b/git-stash.sh
index 4938ade..6685004 100755
--- a/git-stash.sh
+++ b/git-stash.sh
@@ -67,7 +67,7 @@ create_stash () {
 		GIT_INDEX_FILE="$TMP-index" &&
 		export GIT_INDEX_FILE &&
 		git read-tree -m $i_tree &&
-		git add -u &&
+		git add $ADD_OPTIONS </dev/tty >/dev/tty 2>&1 &&
 		git write-tree &&
 		rm -f "$TMP-index"
 	) ) ||
@@ -218,6 +218,11 @@ drop_stash () {
 	git rev-parse --verify "$ref_stash@{0}" > /dev/null 2>&1 || clear_stash
 }
 
+ADD_OPTIONS="-u"
+case "$1" in
+	-i|-p) ADD_OPTIONS=$1; shift
+esac
+
 # Main command set
 case "$1" in
 list)
@@ -235,7 +240,7 @@ show)
 	;;
 save)
 	shift
-	save_stash "$*" && git-reset --hard
+	save_stash "$*" && show_stash -p -R | git apply
 	;;
 apply)
 	shift
@@ -269,7 +274,7 @@ pop)
 	then
 		save_stash &&
 		echo '(To restore them type "git stash apply")' &&
-		git-reset --hard
+		show_stash -p -R | git apply
 	else
 		usage
 	fi

Re: An alternate model for preparing partial commits

[Robert Anderson]

On Fri, Jun 27, 2008 at 10:03 PM, Jeff King <peff@peff.net> wrote:
> So I fundamentally think that all of these contortions are because
> moving things to the stash bucket is not as featureful as moving them to
> the index bucket. And there's no reason for it, since we can use the
> _same_ tools to do both.
>
> Here's a somewhat hackish implementation of "git stash -i" that just relies
> on "add -i":

Are all features for moving changes to stash bi-directional in your
implementation?  Can we move a hunk out of stash, just as easily as we
can move one in?  I think this is an essential property of a good
implementation of this workflow.

As to which way people like to think and work, in terms of re-applying
changes from a fresh state one at a time, or just pushing off changes
they don't want, I think ensuring bi-directionality in the tools for
moving changes back and forth will ensure that all such scenarios will
be equally well supported.

It does seem to me at this point that extending stash functionality is
a reasonable way to approach supporting this type of workflow.

Thanks,
Bob

Re: An alternate model for preparing partial commits

[Jeff King]

On Sat, Jun 28, 2008 at 12:03:54AM -0700, Robert Anderson wrote:

> > Here's a somewhat hackish implementation of "git stash -i" that just relies
> > on "add -i":
> 
> Are all features for moving changes to stash bi-directional in your
> implementation?  Can we move a hunk out of stash, just as easily as we
> can move one in?  I think this is an essential property of a good
> implementation of this workflow.

No, they're not bi-directional. You get the full power of "add -i" when
moving things into the stash, but not out. It is the reverse of the
situation with the index; we have good tools for staging things, but not
for unstaging them.

But I agree that they _should_ be. Given the patch I posted already,
probably the simplest way to do both at once would be to give "add -i"
an "unstage" mode.

> It does seem to me at this point that extending stash functionality is
> a reasonable way to approach supporting this type of workflow.

Actually, I oversimplified a little bit in my "buckets" description.
Stash actually stashes two things: the current index and the current
worktree. So in that sense, it is not just another bucket as I
described. For the purposes of the workflow we're discussing, I think
that is how we want it to function. But the implementation will be a bit
trickier than it might otherwise be because of this. I just didn't want
to have to introduce another, slightly different type of stash.

-Peff

Re: An alternate model for preparing partial commits

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Actually, I oversimplified a little bit in my "buckets" description.
> Stash actually stashes two things: the current index and the current
> worktree. So in that sense, it is not just another bucket as I
> described. For the purposes of the workflow we're discussing, I think
> that is how we want it to function. But the implementation will be a bit
> trickier than it might otherwise be because of this. I just didn't want
> to have to introduce another, slightly different type of stash.

You do *not* have to use stash that has different index and work tree
components and then your buckets description makes sense.

The index is to hold good changes verified to be fine to make commits, the
work tree is to test and verify outstanding changes, and the stash is to
hold the remainder (i.e. further changes that does not belong to what you
are currently looking at in your work tree).

When your workflow is to "verify and immediately commit", then the need
for buckets to your particular workflow can degenerate to a one that does
not need the index.  That is essentially Robert's workflow (but it does
not mean it is the only valid one).

What I do these days is this:

 * Fork a new topic from the commit I pushed out the last time to the
   public ("git checkout -b jc/topic ko/master").

 * Think, hack, commit, think, hack, commit, lather, rinse, repeat.

 * Make sure everything is worthy for the final state.  There can be (and
   need to be to use the current set of tools) some uncommitted changes.
   Make a stash, so that the work tree component records the final tree,
   and mentally name it the "commit goal".

 * I have never grew comfortable operating "edit" insn in "rebase -i", so
   the workflow from this point does not use it.  Instead, I detech the
   HEAD to the root of the series ("git checkout ko/master^0") at this
   point.  Now, none of my change is in the work tree.

 * Repeat the following:

   * Recreate the work tree state for the next round to be built on HEAD
     and make a commit, after verifying what I have in the commit is good.

     Examples of the tools at my disposal are:

     * "cherry-pick jc/topic~$N" to get the necessary changes from my
       earlier "snapshots", which can possibly be followed by a "git
       commit --amend".  This "going forward" is easiest especially in the
       early part of the sequence.

       "format-patch --stdout jc/topic~$N..jc/topic~$M | git am" is a
       slight variant of the above when I already had a good logical
       sequence (someday we probably will have "cherry-pick A..B").

     * "read-tree -m -u stash" to read the final state of the tree,
       selectively _remove_ the parts I do not want in this round, and
       make a commit ("git add -i && git commit && git reset --hard").
       This "going backward" is easier near the end of the sequence than
       other method that goes forward.

   * If I find some issues to be fixed in the state that was stashed
     (which I earlier thought was perfect) during the above:

    * "read-tree -m -u stash" to read the (previous) final state, fix it
      up, "stash drop" and "stash save" to update our "commit goal".

   The above is repeated until "git diff HEAD stash" does not have
   anything I need in the final series.

   The lower-level "read-tree -m -u" probably can be replaced with "stash
   apply" in real life, but I tend to try to ask for the final tree
   explicitly, because there is no reason to perform three-way merge dance
   "stash apply" does for these steps.

 * Final re-review:

   * "git diff jc/topic HEAD" to see if I did not miss anything (and review
     the "oops, the earlier commit goal was faulty" fixes).

   * "git log --reverse -p ko/master.." to see the final shape of the
     series.

 * "git branch -f jc/topic" to finish it off.

The commits from the session are merely convenient snapshot points that I
can use during the clean-up phase for series of cherry-picks to prepare
bulk of each of the logical change in the final series.  If somebody
subscribes to a dogma not to make commits of unproven changes, that is
fine, and such a person may not have any commits during "think, hack,
lather, rinse, repeat" phase, and that is fine.  But fortunately I don't.

Re: An alternate model for preparing partial commits

[Johannes Schindelin]

Hi.

On Sat, 28 Jun 2008, Jeff King wrote:

> Here's a somewhat hackish implementation of "git stash -i" that just 
> relies on "add -i":

I like it.

> +ADD_OPTIONS="-u"
> +case "$1" in
> +	-i|-p) ADD_OPTIONS=$1; shift
> +esac

I'll squash in "|-e" here.

Note that some time ago, I was working on "git stash apply -i", which I 
never came around to finish.  Yesterday, I was briefly considering working 
on it, but I have other things to tend to now.

Ciao,
Dscho

Re: An alternate model for preparing partial commits

[Jeff King]

On Sat, Jun 28, 2008 at 03:51:12PM +0100, Johannes Schindelin wrote:

> > Here's a somewhat hackish implementation of "git stash -i" that just 
> > relies on "add -i":
> 
> I like it.

Thinking about this some more, it seems to me to lack one really
important feature that "git add -i" has: you must stash all in one go.

That is, I may do some of the adds as "git add <file1> <file2>" and then
pick out the rest of the changes with "git add -p". And traditionally,
stashing has been about dumping all changes, so everything happened at
once.

But I think what I would really like here is to say "now I don't want to
stage for a commit; I want to stage into some bucket, so that I can
clear my workspace for making the commit". And then proceed to use "add"
or "add -i" in the usual way, except that they go into my bucket. And at
the end, I switch back to staging for a commit, make the commit, and
then start picking things out of my bucket.

And that workflow is not too hard to imagine by just pointing
GIT_INDEX_FILE to the bucket.

But I am still thinking on this, so I'll let it percolate and then maybe
try to implement something once I have a better sense of exactly what
workflow I want. I just thought I would throw it out there for others to
ponder.

-Peff

Re: An alternate model for preparing partial commits

[David Jeske]

Robert, I'm new to git, but I understand where you are going.

Why limit it only to working tree changes? For me, the stash machinery is of no
help here, because I commit super-often. What I end up with is 30 commits in a
topic branch, but where not every point passes 100% of tests. I want to go back
through and order them properly, and decide which points are sensible as
upstream committs. (especially as I read more about bisect).

git has all the concepts I want except one. However, it makes the process
pretty manual. Here is an idea about automating it. I'll talk about that one
new concept at the bottom.

I think of this as reorder/merge/split...

reorder: Picture that a list of commits on this branch opens in an editor. You
are free to rearrange the lines in any order you want, but you have to keep all
the lines. When you are done reordering the lines, the tool creates a new topic
branch and applies the changes (probably with cherrypick) to the new topic
branch. If there are no conflicts, you're done.

merge: Picture now that in your editor you can create groupings of those
individual commits that should make up separate topic-branches. The operation
can still be performed automatically, and at the end, it can compose those
topic branches into a single branch just like your original. At this point, you
can "isolate" any one of those topic branches and test/push that topic branch.

split: Picture now that you can list the same commit in more than one of the
topic-branches. This is a little more tricky, and there is no way to do it
automatically.. It drops you into an editor and asks you to select the lines of
the diff for the first topic. The remaining lines are put in the next topic.
This can continue for multiple topics.

This seems like something that could be assembled pretty easily on top of the
current git mechanisms, except for one thing.

If you use merge, the history will be a mess. If you use rebase, anyone else
who pulled your topic branch will be in a world of hurt.

I've been thinking about a solution for this I think of as "supercede".

Once you have completed the above reorder/merge/split, your new topic branch
should be EXACTLY the same as your old topic branch. (if it's not, it needs to
be trued up to be so). At that point, it is safe to ask for that new line of
commits to supercede the old line. Other people who have pulled in your older
ordered topic branch would then be able to pull/rebase/etc, and the merge
machinery would be able to back out their set of changes, and supercede them
with your new ordering.

This mechanism is intended to combine the benefits of rebase-clean-history and
the benefits of the dag-links for merging. I find it convenient to think of it
as stack push/pop for portions of the dag. Because of the supercede - the
history knows it can avoid showing you all the superceded dag nodes, however,
because those nodes are there, it can still use them to compute merges.

If this behaves the way I think, this has another powerful effect. You can pull
in a set of draft changes; you can build off them; you can periodically rebase
them, and if those draft changes end up in the mainline, because the
merge-history is still there, git can 'do the right thing' and remove those
changes from your topic branch. In fact, because of the SHA1 strong naming, it
doesn't even matter where you got them from. You could apply a patch from the
mailing list and as long as everyone applies that patch as only a single
commit, when the string of supercedes shows up on the main branch git will just
'do-the right thing' to remove them from your topic branch when you rebase (or
skip them during a merge down to your topic branch).

Re: An alternate model for preparing partial commits

[David Jeske]

Robert, I'm new to git, but I understand where you are going.

Why limit it only to working tree changes? For me, the stash machinery is of no
help here, because I commit super-often. What I end up with is 30 commits in a
topic branch, but where not every point passes 100% of tests. I want to go back
through and order them properly, and decide which points are sensible as
upstream committs. (especially as I read more about bisect).

git has all the concepts I want except one. However, it makes the process
pretty manual. Here is an idea about automating it. I'll talk about that one
new concept at the bottom.

I think of this as reorder/merge/split...

reorder: Picture that a list of commits on this branch opens in an editor. You
are free to rearrange the lines in any order you want, but you have to keep all
the lines. When you are done reordering the lines, the tool creates a new topic
branch and applies the changes (probably with cherrypick) to the new topic
branch. If there are no conflicts, you're done.

merge: Picture now that in your editor you can create groupings of those
individual commits that should make up separate topic-branches. The operation
can still be performed automatically, and at the end, it can compose those
topic branches into a single branch just like your original. At this point, you
can "isolate" any one of those topic branches and test/push that topic branch.

split: Picture now that you can list the same commit in more than one of the
topic-branches. This is a little more tricky, and there is no way to do it
automatically.. It drops you into an editor and asks you to select the lines of
the diff for the first topic. The remaining lines are put in the next topic.
This can continue for multiple topics.

This seems like something that could be assembled pretty easily on top of the
current git mechanisms, except for one thing.

If you use merge, the history will be a mess. If you use rebase, anyone else
who pulled your topic branch will be in a world of hurt.

I've been thinking about a solution for this I think of as "supercede".

Once you have completed the above reorder/merge/split, your new topic branch
should be EXACTLY the same as your old topic branch. (if it's not, it needs to
be trued up to be so). At that point, it is safe to ask for that new line of
commits to supercede the old line. Other people who have pulled in your older
ordered topic branch would then be able to pull/rebase/etc, and the merge
machinery would be able to back out their set of changes, and supercede them
with your new ordering.

This mechanism is intended to combine the benefits of rebase-clean-history and
the benefits of the dag-links for merging. I find it convenient to think of it
as stack push/pop for portions of the dag. Because of the supercede - the
history knows it can avoid showing you all the superceded dag nodes, however,
because those nodes are there, it can still use them to compute merges.

If this behaves the way I think, this has another powerful effect. You can pull
in a set of draft changes; you can build off them; you can periodically rebase
them, and if those draft changes end up in the mainline, because the
merge-history is still there, git can 'do the right thing' and remove those
changes from your topic branch. In fact, because of the SHA1 strong naming, it
doesn't even matter where you got them from. You could apply a patch from the
mailing list and as long as everyone applies that patch as only a single
commit, when the string of supercedes shows up on the main branch git will just
'do-the right thing' to remove them from your topic branch when you rebase (or
skip them during a merge down to your topic branch).

Re: An alternate model for preparing partial commits

[Jakub Narebski]

David Jeske wrote:

> reorder: Picture that a list of commits on this branch opens in an editor. You
> are free to rearrange the lines in any order you want, but you have to keep all
> the lines. When you are done reordering the lines, the tool creates a new topic
> branch and applies the changes (probably with cherrypick) to the new topic
> branch. If there are no conflicts, you're done.
> 
> merge: Picture now that in your editor you can create groupings of those
> individual commits that should make up separate topic-branches. The operation
> can still be performed automatically, and at the end, it can compose those
> topic branches into a single branch just like your original. At this point, you
> can "isolate" any one of those topic branches and test/push that topic branch.

git rebase --interactive?
Any patch management interface (StGIT, Guilt)?
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: An alternate model for preparing partial commits

[David Jeske]

-- Jakub Narebski wrote:
> git rebase --interactive?
> Any patch management interface (StGIT, Guilt)?

Yes, as I said, that set of operations can be performed with git today.

What git can't do, is let me "supercede" the old DAG-subset, so people I shared
them with can get my new changes without hurting their world. Currently git
seems to rely on the idea that "if you accept changes into your tree that will
be later rebased, it's up to you to figure it out". I don't see why that is the
case.

Re: An alternate model for preparing partial commits

[David Jeske]

-- Jakub Narebski wrote:
> git rebase --interactive?
> Any patch management interface (StGIT, Guilt)?

Yes, as I said, that set of operations can be performed with git today.

What git can't do, is let me "supercede" the old DAG-subset, so people I shared
them with can get my new changes without hurting their world. Currently git
seems to rely on the idea that "if you accept changes into your tree that will
be later rebased, it's up to you to figure it out". I don't see why that is the
case.

Re: An alternate model for preparing partial commits

[Robert Anderson]

On Fri, Jun 27, 2008 at 1:51 PM, David Jeske <jeske@willowmail.com> wrote:
> -- Jakub Narebski wrote:
>> git rebase --interactive?
>> Any patch management interface (StGIT, Guilt)?
>
> Yes, as I said, that set of operations can be performed with git today.
>
> What git can't do, is let me "supercede" the old DAG-subset, so people I shared
> them with can get my new changes without hurting their world. Currently git
> seems to rely on the idea that "if you accept changes into your tree that will
> be later rebased, it's up to you to figure it out". I don't see why that is the
> case.


Possibly a succinct way of moving this conversation forward is to say that:

What is desired is a workflow where partial commits can be tested,
when it is desirable not to change history.

There are good reasons for desiring a workflow that does not routinely
change history as part of the usual workflow.  Maybe there are clones
of your repo.  Maybe as part of your workflow discipline you do not
want HEAD states that cannot be pushed to public, because you don't
want to manually keep track of when it is ok and when it is not ok to
push HEAD to public, since git cannot tell you this.

Thanks,
Bob

Re: An alternate model for preparing partial commits

[Junio C Hamano]

"Robert Anderson" <rwa000@gmail.com> writes:

> There are good reasons for desiring a workflow that does not routinely
> change history as part of the usual workflow.  Maybe there are clones
> of your repo.  Maybe as part of your workflow discipline you do not
> want HEAD states that cannot be pushed to public, because you don't
> want to manually keep track of when it is ok and when it is not ok to
> push HEAD to public, since git cannot tell you this.

Surely you can arrange that.  You keep track of what you pushed out, and
you refrain from rebasing beyond that point.

And fast-forward check in the push to the public will notice if you break
that "workflow discipline" by accident.

Re: An alternate model for preparing partial commits

[Robert Anderson]

On Fri, Jun 27, 2008 at 4:14 PM, Junio C Hamano <gitster@pobox.com> wrote:
> "Robert Anderson" <rwa000@gmail.com> writes:
>
>> There are good reasons for desiring a workflow that does not routinely
>> change history as part of the usual workflow.  Maybe there are clones
>> of your repo.  Maybe as part of your workflow discipline you do not
>> want HEAD states that cannot be pushed to public, because you don't
>> want to manually keep track of when it is ok and when it is not ok to
>> push HEAD to public, since git cannot tell you this.
>
> Surely you can arrange that.  You keep track of what you pushed out, and
> you refrain from rebasing beyond that point.

First, the constraint of achieving this workflow when there are clones
of the repo in question removes any need for additional rationales.

That being said...

In the existing model which is being suggested as a way to get the
desired workflow, there are two distinct classes of commits:  commits
that are "for real", and commits that are "temporary", that are being
used as some kind of workspace for orthogonalizing a set of changes,
which will eventually be replaced by "for real" commits.  Yet git has
no metadata to distinguish these two types of commits.  When only "for
real" commits exist, I can push HEAD.  When "temporary" commits exist,
I cannot, or insidious problems will ensue.  This metadata concerning
"for real" or "temporary" commits is only maintained manually in the
developer's head.  But, this is the sort of thing that computers are
for, and should not be the burden of the developer to maintain this
information in his mental cache, especially when the price of making
an error can be quite high.  It would be very easy to forget that you
intended to do more work on your last N commits, and push HEAD because
somebody asks you to make sure you've pushed your latest work.  Your
tools should prevent this kind of scenario.  Git cannot, because it
doesn't know what you intend when you "commit" because two different
ideas are being conflated with "commit" in this model.

In my proposed model, there is a clear distinction, kept track of by
git, not the developer, between changes which are in the "temporary
workspace" which is still being worked on, and changes which are "for
real", that can be shared, either through a push to a public repo or
if there are clones of my repo.  They become "for real" when they are
committed.  That's the purpose of a "commit" in this workflow.  To
bless a change as ready for viewing by others, even if you do not want
to make it available yet (by a push to a public repo).  I can walk
away from a working tree for six months, come back, and there can be
no confusion about which changes were "temporary" and possibly in need
of revision, and which changes are blessed as ready for public
consumption.  If I come back to a branch on which there are several
commits which have not been pushed yet, how do I know which are
"temporary" and which are "for real" commits?  I cannot.  It is
impossible.  The information is not there.

But, all of this is moot when you consider the simple case of a repo
which has been cloned, on which you'd like to make partial commits,
and test the committed state before doing so.

Thanks,
Bob

Re: An alternate model for preparing partial commits

[Dmitry Potapov]

On Fri, Jun 27, 2008 at 05:08:57PM -0700, Robert Anderson wrote:
> 
> In the existing model which is being suggested as a way to get the
> desired workflow, there are two distinct classes of commits:  commits
> that are "for real", and commits that are "temporary", that are being
> used as some kind of workspace for orthogonalizing a set of changes,
> which will eventually be replaced by "for real" commits.

Not really. Good commits do not get replaced by anything. They are
just pushed to the public repo after being tested. Those commits
that have not passed the test should be amended and tested again.

> Yet git has
> no metadata to distinguish these two types of commits.  When only "for
> real" commits exist, I can push HEAD.  When "temporary" commits exist,
> I cannot, or insidious problems will ensue.  This metadata concerning
> "for real" or "temporary" commits is only maintained manually in the
> developer's head.

No, you can use a tag for that, which marks the tip of tested commits;
or you can make your test procedure to push commits automatically after
successful testing. There is no reason for Git to have any metadata for
that.

Dmitry

Re: An alternate model for preparing partial commits

[Robert Anderson]

On Fri, Jun 27, 2008 at 7:57 PM, Dmitry Potapov <dpotapov@gmail.com> wrote:
> On Fri, Jun 27, 2008 at 05:08:57PM -0700, Robert Anderson wrote:
>>
>> In the existing model which is being suggested as a way to get the
>> desired workflow, there are two distinct classes of commits:  commits
>> that are "for real", and commits that are "temporary", that are being
>> used as some kind of workspace for orthogonalizing a set of changes,
>> which will eventually be replaced by "for real" commits.
>
> Not really. Good commits do not get replaced by anything. They are
> just pushed to the public repo after being tested. Those commits
> that have not passed the test should be amended and tested again.
>
>> Yet git has
>> no metadata to distinguish these two types of commits.  When only "for
>> real" commits exist, I can push HEAD.  When "temporary" commits exist,
>> I cannot, or insidious problems will ensue.  This metadata concerning
>> "for real" or "temporary" commits is only maintained manually in the
>> developer's head.
>
> No, you can use a tag for that, which marks the tip of tested commits;
> or you can make your test procedure to push commits automatically after
> successful testing. There is no reason for Git to have any metadata for
> that.
>
> Dmitry

There is a disconnect here.  Rather than argue somewhat abstractly,  I
will try to construct some concrete examples which illustrate the
essential problems with the current model and some proposed solutions
to improve it.

Bob

Re: An alternate model for preparing partial commits

[Stephen Sinclair]

On Fri, Jun 27, 2008 at 8:08 PM, Robert Anderson <rwa000@gmail.com> wrote:
> If I come back to a branch on which there are several
> commits which have not been pushed yet, how do I know which are
> "temporary" and which are "for real" commits?  I cannot.  It is
> impossible.  The information is not there.

I use the comments for this.  I frequently have comments that say...
"NOT FINISHED, AMEND this".  I make sure they don't get into my public
repo.

You could even automate it with a hook.  You could program your public
repo to refuse a push that contains commits with some keyword like
"WIP", and get used to putting WIP into any new commit until you're
sure about it.  I think you can even program a hook (commit-msg?) to
generate the initial comment for any commit, so you could make it the
default.


> But, all of this is moot when you consider the simple case of a repo
> which has been cloned, on which you'd like to make partial commits,
> and test the committed state before doing so.

You can work in a branch which is not intended to be cloned, and only
merge/rebase to master when you're ready.  You can work in a separate
repo and only push to public when you're ready.  There are several
solutions here for what you want to do.

The answer is simple: you should not be making partial commits to a
repo that has been cloned.  You should instead be working somewhere
else and then pushing to it.  So this whole sentence is just a moot
point itself.


Steve

Re: An alternate model for preparing partial commits

[Robert Anderson]

On Sat, Jun 28, 2008 at 7:34 AM, Stephen Sinclair <radarsat1@gmail.com> wrote:
> The answer is simple: you should not be making partial commits to a
> repo that has been cloned.  You should instead be working somewhere
> else and then pushing to it.  So this whole sentence is just a moot
> point itself.

Ah, now you've hit the crux.  Thank you for the "svn style" response
here.  I "should not" because git has a deficiency.  Absolutely no
other reason.

Thanks,
Bob

Re: An alternate model for preparing partial commits

[Robert Anderson]

On Sat, Jun 28, 2008 at 7:34 AM, Stephen Sinclair <radarsat1@gmail.com> wrote:
> The answer is simple: you should not be making partial commits to a
> repo that has been cloned.  You should instead be working somewhere
> else and then pushing to it.  So this whole sentence is just a moot
> point itself.

Let me amplify my objection to this.

Who has 100% foresight that what they are doing is going to end up in
a state where they'd like to make partial commits?  To take a quote
from a blog post, 'Git means never having to say, "you should have"'.
And mostly it doesn't, and that's big improvement over other systems.
But, that is what you are saying here.  I "should have" realized that
I should have pulled and fiddled with my changes there, and then
pushed.

Well, Dmitri and others will now say, why not just always pull and
work somewhere else?  And the reason is that because this creates
extra, unnecessary steps the vast majority of the time when I do
create a commit that I like and want to keep as-is in the first try.
Why should I have to pull, commit, hack, and push, when hack and
commit is all I need to do the vast majority of the time?  It is
redundant, unnecessary work and complexity that I should not have to
pay for when I don't need it.

Thanks,
Bob

Re: An alternate model for preparing partial commits

[Jakub Narebski]

"Robert Anderson" <rwa000@gmail.com> writes:

> On Sat, Jun 28, 2008 at 7:34 AM, Stephen Sinclair <radarsat1@gmail.com> wrote:
> > The answer is simple: you should not be making partial commits to a
> > repo that has been cloned.  You should instead be working somewhere
> > else and then pushing to it.  So this whole sentence is just a moot
> > point itself.
> 
> Let me amplify my objection to this.
> 
> Who has 100% foresight that what they are doing is going to end up in
> a state where they'd like to make partial commits?  To take a quote
> from a blog post, 'Git means never having to say, "you should have"'.
> And mostly it doesn't, and that's big improvement over other systems.
> But, that is what you are saying here.  I "should have" realized that
> I should have pulled and fiddled with my changes there, and then
> pushed.
> 
> Well, Dmitri and others will now say, why not just always pull and
> work somewhere else?  And the reason is that because this creates
> extra, unnecessary steps the vast majority of the time when I do
> create a commit that I like and want to keep as-is in the first try.
> Why should I have to pull, commit, hack, and push, when hack and
> commit is all I need to do the vast majority of the time?  It is
> redundant, unnecessary work and complexity that I should not have to
> pay for when I don't need it.

I think that in most cases the setup looks like the following: there
is private non-bare repository, ehere you can use topic branches,
and change and rewrite those changes at will, be it using plain rebase,
rebase --interactive, or some patch management interface like StGit
or Guilt.  That is where you clean up your commits till they all fill
some standards / convention, like compiling (at least), or pass the
testsuite.

Then you merge it into one of long-lived "development" branches, and
push to your public (usually bare) publishing repository, which can
be for example on repo.or.cz, or on kernel.org.


That said "git stash (save|apply) --interactive" would be good
improvement for the situation where you have realized that you have
several unrelated changes in working directory, and want to decouple
them.  One solution would be to commit, then split using interactive
rebase (or patch management tool), but with stash I guess it could be
simpler.

You should avoid such situation by stashing and changing the branch,
or refrshing current patch (to return to it later) and generating new
one if you use StGit or Guilt...

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: An alternate model for preparing partial commits

[Robert Anderson]

On Sat, Jun 28, 2008 at 10:12 AM, Jakub Narebski <jnareb@gmail.com> wrote:
> "Robert Anderson" <rwa000@gmail.com> writes:
>
>> On Sat, Jun 28, 2008 at 7:34 AM, Stephen Sinclair <radarsat1@gmail.com> wrote:
>> > The answer is simple: you should not be making partial commits to a
>> > repo that has been cloned.  You should instead be working somewhere
>> > else and then pushing to it.  So this whole sentence is just a moot
>> > point itself.
>>
>> Let me amplify my objection to this.
>>
>> Who has 100% foresight that what they are doing is going to end up in
>> a state where they'd like to make partial commits?  To take a quote
>> from a blog post, 'Git means never having to say, "you should have"'.
>> And mostly it doesn't, and that's big improvement over other systems.
>> But, that is what you are saying here.  I "should have" realized that
>> I should have pulled and fiddled with my changes there, and then
>> pushed.
>>
>> Well, Dmitri and others will now say, why not just always pull and
>> work somewhere else?  And the reason is that because this creates
>> extra, unnecessary steps the vast majority of the time when I do
>> create a commit that I like and want to keep as-is in the first try.
>> Why should I have to pull, commit, hack, and push, when hack and
>> commit is all I need to do the vast majority of the time?  It is
>> redundant, unnecessary work and complexity that I should not have to
>> pay for when I don't need it.
>
> I think that in most cases the setup looks like the following: there
> is private non-bare repository, ehere you can use topic branches,
> and change and rewrite those changes at will, be it using plain rebase,
> rebase --interactive, or some patch management interface like StGit
> or Guilt.  That is where you clean up your commits till they all fill
> some standards / convention, like compiling (at least), or pass the
> testsuite.

In my opinion it looks like that because git has poor support for
creating good partial commits the first time around.

I also think all of this rebase -i style approach is fine for an
experienced power user if they like that approach, but that splitting
the changes in a working tree is a very basic usage that should be a
part of a "Git in 20 minutes" tutorial.  StGit, rebase -i, etc.,
should not.

In my experience this is pretty much the *first* thing a developer
faces when they decide that having a good incremental change history
is a good practice they want to follow.  They usually shrug, commit
the hairball, and resolve to do better next time.  But this is an
unavoidable scenario in real world development, and a good SCM manager
should have convenient facilities for dealing with it, that don't
reach into the power user toolbox like rebase -i (and let's face it,
that is an absurdly counterintuitive command for the job at hand), or
require auxilliary tools like StGit.  The index is a start.  But, IMO,
it isn't sufficient.

Stay tuned for some proposals about improving partial commit support.

Thanks,
Bob

Re: An alternate model for preparing partial commits

[David Jeske]

-- Robert Anderson wrote:
> If I come back to a branch on which there are several
> commits which have not been pushed yet, how do I know which are
> "temporary" and which are "for real" commits?  I cannot.  It is
> impossible.  The information is not there.

This comment resonates with me Robert. Because git's concept of a workspace
is a "non-tracking branch", git doesn't automate having "temporary" and "for
real" commits intermixed in the workspace, or having more than one "named set
of temporary commits".

Imagine if we use git's mechanisms to setup a "p4 change" style workflow. In
that workflow, we can have multiple outstanding named changes in the same
working-files. This seems simple and reasonable to me, I use it quite a bit in
p4. My most common use of it is to prevent myself from accidentally publishing
a set of edited files that are supposed to remain local as I'm iterating (I
throw them in a "don't submit this yet" pending change).

If we dynamically compose a "workspace" out of 2 temporary "changes" it might
look like this:


:        /---PQ    workspace/1
:   |<- Q   /      change/feature1
:   |<---- P       change/feature2
:   |
: --A<--B          master

You're working in workspace 1, and you could pretty easily make the equivilant
of "git commit -c change/feature2". That would say, "hey, commit this change,
and then merge it down to "change/feature2 branch" as R'.

:        /---PQ---R       workspace/1
:   |<- Q   /    /        change/feature1
:   |<---- P<---R'        change/feature2
:   |
: --A<--B          master

This seems doable, and git's mechanisms would make it pretty easy (though it
feels dangerous to me).

You can then merge either change onto the master when it's done. Your little
wrapper could give you a "workspace rebase" which could destroy the workspace,
rebase all the features, and recompose the workspace. If you want to rebase one
of the changes onto the master, again, your wrapper destroys the workspace,
rebases and pushes the change, rebases the rest of the changes, and recreates
your workspace.

All of that "destroying the workspace" is required because rebase is a change
operation which does not contain merge-history. In the setup above, your
"change/*" branches are effectively "published to your workspace", so if you
rebase either of them, you're technically rebasing something that's published,
and "workspace" will be in a world of hurt.  Therefore, one solution is to
destroy the workspace, making the rebase a valid operation, then recompose it.
Personally, I'd like to see rebase made to have merge history, using some kind
of "soft-dag-link" that would cause it to work for anyone who had those
changes, while not causing the history to show or hold onto those parents.

Re: An alternate model for preparing partial commits

[David Jeske]

-- Robert Anderson wrote:
> If I come back to a branch on which there are several
> commits which have not been pushed yet, how do I know which are
> "temporary" and which are "for real" commits?  I cannot.  It is
> impossible.  The information is not there.

This comment resonates with me Robert. Because git's concept of a workspace
is a "non-tracking branch", git doesn't automate having "temporary" and "for
real" commits intermixed in the workspace, or having more than one "named set
of temporary commits".

Imagine if we use git's mechanisms to setup a "p4 change" style workflow. In
that workflow, we can have multiple outstanding named changes in the same
working-files. This seems simple and reasonable to me, I use it quite a bit in
p4. My most common use of it is to prevent myself from accidentally publishing
a set of edited files that are supposed to remain local as I'm iterating (I
throw them in a "don't submit this yet" pending change).

If we dynamically compose a "workspace" out of 2 temporary "changes" it might
look like this:


:        /---PQ    workspace/1
:   |<- Q   /      change/feature1
:   |<---- P       change/feature2
:   |
: --A<--B          master

You're working in workspace 1, and you could pretty easily make the equivilant
of "git commit -c change/feature2". That would say, "hey, commit this change,
and then merge it down to "change/feature2 branch" as R'.

:        /---PQ---R       workspace/1
:   |<- Q   /    /        change/feature1
:   |<---- P<---R'        change/feature2
:   |
: --A<--B          master

This seems doable, and git's mechanisms would make it pretty easy (though it
feels dangerous to me).

You can then merge either change onto the master when it's done. Your little
wrapper could give you a "workspace rebase" which could destroy the workspace,
rebase all the features, and recompose the workspace. If you want to rebase one
of the changes onto the master, again, your wrapper destroys the workspace,
rebases and pushes the change, rebases the rest of the changes, and recreates
your workspace.

All of that "destroying the workspace" is required because rebase is a change
operation which does not contain merge-history. In the setup above, your
"change/*" branches are effectively "published to your workspace", so if you
rebase either of them, you're technically rebasing something that's published,
and "workspace" will be in a world of hurt.  Therefore, one solution is to
destroy the workspace, making the rebase a valid operation, then recompose it.
Personally, I'd like to see rebase made to have merge history, using some kind
of "soft-dag-link" that would cause it to work for anyone who had those
changes, while not causing the history to show or hold onto those parents.

Re: An alternate model for preparing partial commits

[Stephen Sinclair]

On Sat, Jun 28, 2008 at 12:00 PM, Robert Anderson <rwa000@gmail.com> wrote:
> On Sat, Jun 28, 2008 at 7:34 AM, Stephen Sinclair <radarsat1@gmail.com> wrote:
>> The answer is simple: you should not be making partial commits to a
>> repo that has been cloned.  You should instead be working somewhere
>> else and then pushing to it.  So this whole sentence is just a moot
>> point itself.
>
> Ah, now you've hit the crux.  Thank you for the "svn style" response
> here.  I "should not" because git has a deficiency.  Absolutely no
> other reason.

No, you said that a certain operation (testing partial commits) was
impossible.  I told you how I approach the problem with git and tried
to show that it was entirely possible.  That is all.


On Sat, Jun 28, 2008 at 12:30 PM, Robert Anderson <rwa000@gmail.com> wrote:
> Why should I have to pull, commit, hack, and push, when hack and
> commit is all I need to do the vast majority of the time?

But what you have described here, the difference between "commit/push"
and just "commit", is _exactly_ what differentiates distributed and
centralized SCM systems.

To go back to the basic problem that you suggested, that partial
commits can easily go untested, in the "hack and commit" model this is
impossible to work around.  In the "hack, commit, go back and fix,
commit, push" model, which git makes possible, it _is_ possible to do
the testing that you desire.


Steve

Re: An alternate model for preparing partial commits

[David Jeske]

-- Robert Anderson wrote:
> Possibly a succinct way of moving this conversation forward is to say that:
>
> What is desired is a workflow where partial commits can be tested,
> when it is desirable not to change history.

Why don't we just make a model where history changes are handled safetly? Then
we don't have to be afraid to publish DAGs we may want to change later.

Re: An alternate model for preparing partial commits

[David Jeske]

-- Robert Anderson wrote:
> Possibly a succinct way of moving this conversation forward is to say that:
>
> What is desired is a workflow where partial commits can be tested,
> when it is desirable not to change history.

Why don't we just make a model where history changes are handled safetly? Then
we don't have to be afraid to publish DAGs we may want to change later.

Re: An alternate model for preparing partial commits

[Theodore Tso]

On Fri, Jun 27, 2008 at 08:29:15PM -0000, David Jeske wrote:
> git has all the concepts I want except one. However, it makes the process
> pretty manual. Here is an idea about automating it. I'll talk about that one
> new concept at the bottom.
> 
> I think of this as reorder/merge/split...
> 
> reorder: Picture that a list of commits on this branch opens in an
> editor. You are free to rearrange the lines in any order you want,
> but you have to keep all the lines. When you are done reordering the
> lines, the tool creates a new topic branch and applies the changes
> (probably with cherrypick) to the new topic branch. If there are no
> conflicts, you're done.....

Read the man page for for "git rebase --interactive".  I think you
will find that it handles reording and merging (its called squashing)
already.

Splitting patches is harder to do and probably is one of the things
you have to do manually.

						- Ted

Re: An alternate model for preparing partial commits

[Miklos Vajna]

[-- Attachment #1: Type: text/plain, Size: 288 bytes --]

On Fri, Jun 27, 2008 at 09:17:24PM -0400, Theodore Tso <tytso@mit.edu> wrote:
> Splitting patches is harder to do and probably is one of the things
> you have to do manually.

Or not that hard:

http://git.kernel.org/?p=git/git.git;a=blob;f=Documentation/git-rebase.txt;hb=HEAD#l363

:-)

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: An alternate model for preparing partial commits

[Theodore Tso]

On Fri, Jun 27, 2008 at 08:51:41PM -0000, David Jeske wrote:
> -- Jakub Narebski wrote:
> > git rebase --interactive?
> > Any patch management interface (StGIT, Guilt)?
> 
> Yes, as I said, that set of operations can be performed with git today.
> 
> What git can't do, is let me "supercede" the old DAG-subset, so
> people I shared them with can get my new changes without hurting
> their world. Currently git seems to rely on the idea that "if you
> accept changes into your tree that will be later rebased, it's up to
> you to figure it out". I don't see why that is the case

Because it's fundamentally hard and fraught with danger --- especially
since with git-rebase --interactive you actually can *edit* an
intervening patch, so commits that are based off of a rebased branch
have absolutely no guarantee of correctly applying against the rebased
branch.

If they do apply cleanly, you can use git rebase --onto, but the
possibility of merge conflicts are endless.

The general workflow here is that you simply don't share your git
state until it's been refactored and you're positive that it you've
gotten it right.  Or if you do share, it you do so in a branch where
you tell people --- don't base work off of this!  (i.e., a "pu" branch
like git has.)

As I said earlier, premature sharing is highly over-rated.  And having
people develop against an unstable branch is just extra work for them
anyway.  So commit frequently, and if you want to backup your local
hard drive, push regularly to your own private git repository located
off-machine.  Just don't ask people base any branches off your private
branch until you're ready to publish it in a form after which you
don't do rebase operations.  This works quite well for many projects
and many developers.

						- Ted

Re: An alternate model for preparing partial commits

[Stephen R. van den Berg]

David Jeske wrote:
>- I could supercede 2-commits with 1, effectively making the bad-commit
>disappear in the linear history. Users who already have the history, however,
>would be unaffected, because the start/end endpoints are the same.

Superseding does not work in git trees and is a disruptive operation to
*all* commits following the point being superseded.
Also, superseding preserves the startpoint, but *always* alters the
endpoint.

I suggest you take a look at how a git-repository is stored and learn
about how the SHA1-hashes interact with the stored repository.
-- 
Sincerely,
           Stephen R. van den Berg.

Heisenberg might have been here.