Re: [PATCH 3/3] stash: require a clean index to apply

Re: [PATCH 3/3] stash: require a clean index to apply

[Jonathan Kamens]

I'm writing about the patch that Jeff King submitted on April 22, in 
<20150422193101.GC27945@peff.net>, in particular, 
https://github.com/git/git/commit/ed178ef13a26136d86ff4e33bb7b1afb5033f908 
. It appears that this patch was included in git 2.4.2, and it breaks my 
workflow.

In particular, I have a pre-commit hook whith does the following:

1. Stash unstaged changes ("git stash -k").
2. Run flake8 over all staged changes.
3. If flake8 complains, then error out of the commit.
4. Otherwise, apply the stash and exit.

This way I am prevented from committing staged changes that don't pass 
flake8. I can't imagine that this is a terribly uncommon workflow.

This worked fine until the aforementioned comment, after which my hook 
complains, "Cannot apply stash: Your index contains uncommitted changes."

The reason I have to do things this way is as follows. Suppose I did the 
following:

1. Stage changes that have a flake8 violation.
2. Fix the flake8 violation in the unstaged version of the staged file.
3. Commit the previously staged changes.

If my commit hook runs over the unstaged version of the file, then it 
won't detect the flake8 violation, and as a result the violation will be 
committed.

If anyone has a suggestion for how I can achieve the desired goal within 
the constraints of the 2.4.2 version of git-stash.sh, I'd love to hear 
it. Otherwise, I'd like to ask for this patch to be reconsidered.

Thank you,

Jonathan Kamens

Re: [PATCH 3/3] stash: require a clean index to apply

[Jeff King]

On Thu, Jun 04, 2015 at 08:43:00PM -0400, Jonathan Kamens wrote:

> I'm writing about the patch that Jeff King submitted on April 22, in
> <20150422193101.GC27945@peff.net>, in particular,
> https://github.com/git/git/commit/ed178ef13a26136d86ff4e33bb7b1afb5033f908 .
> It appears that this patch was included in git 2.4.2, and it breaks my
> workflow.
> 
> In particular, I have a pre-commit hook whith does the following:
> 
> 1. Stash unstaged changes ("git stash -k").
> 2. Run flake8 over all staged changes.
> 3. If flake8 complains, then error out of the commit.
> 4. Otherwise, apply the stash and exit.

Hrm. The new protection in v2.4.2 is meant to prevent you from losing
your index state during step 4 when we run into a conflict. But here you
know something that git doesn't: that we just created the stash based on
this same state, so it should apply cleanly.

So besides the obvious fix of reverting the patch, we could perhaps do
something along the lines of:

  1. Add a --force option to tell git to do it anyway.

  2. Only have the protection kick in when we would in fact have a
     conflict. This is probably hard to implement, though, as we don't
     know until we do the merge (so it would probably involve teaching
     merge a mode where it bails immediately on conflicts rather than
     writing out the conflicted entries to the index).

However, I am puzzled by something in your workflow: does it work when
you have staged and working tree changes to the same hunk? For example:

  [differing content for HEAD, index, and working tree]
  $ git init
  $ echo base >file && git add file && git commit -m base
  $ echo index >file && git add file
  $ echo working >file

  [make our stash]
  $ git stash -k
  Saved working directory and index state WIP on master: dc7f0dd base
  HEAD is now at dc7f0dd base

  [new version]
  $ git.v2.4.2 stash apply
  Cannot apply stash: Your index contains uncommitted changes.

  [old version]
  $ git.v2.4.1 stash apply
  Auto-merging file
  CONFLICT (content): Merge conflict in file
  $ git diff
  diff --cc file
  index 9015a7a,d26b33d..0000000
  --- a/file
  +++ b/file
  @@@ -1,1 -1,1 +1,5 @@@
  ++<<<<<<< Updated upstream
   +index
  ++=======
  + working
  ++>>>>>>> Stashed changes

So v2.4.1 shows a conflict, and loses the index state you had. Is there
something more to your hook that I'm missing (stash options, or
something else) that covers this case?

> The reason I have to do things this way is as follows. Suppose I did the
> following:
> 
> 1. Stage changes that have a flake8 violation.
> 2. Fix the flake8 violation in the unstaged version of the staged file.
> 3. Commit the previously staged changes.
> 
> If my commit hook runs over the unstaged version of the file, then it won't
> detect the flake8 violation, and as a result the violation will be
> committed.

Yeah, the fundamental pattern makes sense. You want to test what is
going into the commit, not what happens to be in the working tree.

One way to do that would be to checkout the proposed index to a
temporary directory and operate on that. But of course that's
inefficient if most of the files are unchanged.

Are you running flake8 across the whole tree, or just on the files with
proposed changes? Does it need to see the whole tree? If you can get
away with feeding it single files, then it should be very efficient to
loop over the output of "git diff-index HEAD" and feed the proposed
updated blobs to it. If you can get away with feeding single lines, then
feeding the actual diffs to it would be even better, but I assume that
is not enough (I do not use flake8 myself).

-Peff

Re: [PATCH 3/3] stash: require a clean index to apply

[Jeff King]

On Sun, Jun 07, 2015 at 08:40:01AM -0400, Jeff King wrote:

> Are you running flake8 across the whole tree, or just on the files with
> proposed changes? Does it need to see the whole tree? If you can get
> away with feeding it single files, then it should be very efficient to
> loop over the output of "git diff-index HEAD" and feed the proposed
> updated blobs to it. If you can get away with feeding single lines, then
> feeding the actual diffs to it would be even better, but I assume that
> is not enough (I do not use flake8 myself).

Like I said, I do not use it, but peeking at the flake8 source code, it
has an "--install-hook" option to set up a pre-commit hook. It looks
like the hook runs "git diff-index --cached --name-only HEAD" to get
the list of modified files, filters that only for "*.py", and then
copies the results into a temporary directory to operate on.

-Peff

Re: [PATCH 3/3] stash: require a clean index to apply

[bär]

On Sun, Jun 7, 2015 at 9:40 AM, Jeff King <peff@peff.net> wrote:
> Hrm. The new protection in v2.4.2 is meant to prevent you from losing
> your index state during step 4 when we run into a conflict. But here you
> know something that git doesn't: that we just created the stash based on
> this same state, so it should apply cleanly.


It is strange that `git stash --keep-index && git stash pop` while
having something in the index, fails with a "Cannot apply stash: Your
index contains uncommitted changes." error, even if we have a
`--force` param it find it awkward that one needs to force
applying/pop'ing even though the stash was created from the same
snapshot where the stash is being merged with.

I understand the original issue, but at least it was expected, when
you stash save/pop/apply, you should know what you are doing anyways.

-- 
Ber Clausen

Re: [PATCH 3/3] stash: require a clean index to apply

[Jeff King]

On Wed, Jun 10, 2015 at 03:19:41PM -0300, bär wrote:

> On Sun, Jun 7, 2015 at 9:40 AM, Jeff King <peff@peff.net> wrote:
> > Hrm. The new protection in v2.4.2 is meant to prevent you from losing
> > your index state during step 4 when we run into a conflict. But here you
> > know something that git doesn't: that we just created the stash based on
> > this same state, so it should apply cleanly.
> 
> 
> It is strange that `git stash --keep-index && git stash pop` while
> having something in the index, fails with a "Cannot apply stash: Your
> index contains uncommitted changes." error, even if we have a
> `--force` param it find it awkward that one needs to force
> applying/pop'ing even though the stash was created from the same
> snapshot where the stash is being merged with.
> 
> I understand the original issue, but at least it was expected, when
> you stash save/pop/apply, you should know what you are doing anyways.

Yeah, I would expect "stash && pop" to work in general. But I find it
puzzling that it does not always with "-k" (this is with v2.3.x):

  $ git init -q
  $ echo head >file && git add file && git commit -qm head
  $ echo index >file && git add file
  $ echo tree >file
  $ git stash --keep-index && git stash pop
  Saved working directory and index state WIP on master: 74f6d33 head
  HEAD is now at 74f6d33 head
  Auto-merging file
  CONFLICT (content): Merge conflict in file

So I am trying to figure out what the use case here is. Clearly the
above is a toy case, but why is "stash -k" followed by a quick pop
useful in general? Certainly I use "stash" (without "-k") and a quick
pop all the time, and I think that is what stash was designed for.

The best use case I can think of is Jonathan's original: to see only the
staged content in the working tree, and then restore the original state.
But stash does not currently work very well for that, as shown above.

-Peff

Re: [PATCH 3/3] stash: require a clean index to apply

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> So I am trying to figure out what the use case here is. Clearly the
> above is a toy case, but why is "stash -k" followed by a quick pop
> useful in general? Certainly I use "stash" (without "-k") and a quick
> pop all the time, and I think that is what stash was designed for.
>
> The best use case I can think of is Jonathan's original: to see only the
> staged content in the working tree, and then restore the original state.
> But stash does not currently work very well for that, as shown above.

The canonical use case for "stash -k" is to see only the content to
be committed (for testing), commit it after testing and then pop on
top of the committed result, which is the same as what you saw in
the working tree and the index when you did "stash -k".  I do not
think "stash -k && stash pop" was in the design parameter when "-k"
was added (as you demonstrated, it would not fundamentally work
reliably depending on the differences between HEAD-Index-Worktree).

Re: [PATCH 3/3] stash: require a clean index to apply

[Jeff King]

On Wed, Jun 10, 2015 at 12:16:25PM -0700, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > So I am trying to figure out what the use case here is. Clearly the
> > above is a toy case, but why is "stash -k" followed by a quick pop
> > useful in general? Certainly I use "stash" (without "-k") and a quick
> > pop all the time, and I think that is what stash was designed for.
> >
> > The best use case I can think of is Jonathan's original: to see only the
> > staged content in the working tree, and then restore the original state.
> > But stash does not currently work very well for that, as shown above.
> 
> The canonical use case for "stash -k" is to see only the content to
> be committed (for testing), commit it after testing and then pop on
> top of the committed result, which is the same as what you saw in
> the working tree and the index when you did "stash -k".  I do not
> think "stash -k && stash pop" was in the design parameter when "-k"
> was added (as you demonstrated, it would not fundamentally work
> reliably depending on the differences between HEAD-Index-Worktree).

It seems like applying a stash made with "-k" is fundamentally
misdesigned in the current code. We would want to apply to the working
tree the difference between the index and the working tree, but instead
we try to apply the difference between the HEAD and the working tree.
Which is nonsensical for this use case (i.e., to apply the diff between
$stash and $stash^2, not $stash^1).

I don't think there is any way to tell that "-k" was used, though. But
even if the user knew that, I do not think there is any option to tell
"stash apply" to do it this way.

I dunno. With respect to the original patch, I am OK if we just want to
revert it. This area of stash seems a bit under-designed IMHO, but if
people were happy enough with it before, I do not think the safety
benefit from ed178ef is that great (it is not saving you from destroying
working tree content, only the index state; the individual content blobs
are still available from git-fsck).

-Peff

Re: [PATCH 3/3] stash: require a clean index to apply

[bär]

On Wed, Jun 10, 2015 at 4:27 PM, Jeff King <peff@peff.net> wrote:
> I dunno. With respect to the original patch, I am OK if we just want to
> revert it. This area of stash seems a bit under-designed IMHO, but if
> people were happy enough with it before, I do not think the safety
> benefit from ed178ef is that great (it is not saving you from destroying
> working tree content, only the index state; the individual content blobs
> are still available from git-fsck).

I feel the same way, in fact I'm +1 to revert it until we figure out a
better way to deal with this properly.

-- 
Ber Clausen

Re: [PATCH 3/3] stash: require a clean index to apply

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> It seems like applying a stash made with "-k" is fundamentally
> misdesigned in the current code. We would want to apply to the working
> tree the difference between the index and the working tree, but instead
> we try to apply the difference between the HEAD and the working tree.
> Which is nonsensical for this use case (i.e., to apply the diff between
> $stash and $stash^2, not $stash^1).
>
> I don't think there is any way to tell that "-k" was used, though. But
> even if the user knew that, I do not think there is any option to tell
> "stash apply" to do it this way.
>
> I dunno. With respect to the original patch, I am OK if we just want to
> revert it. This area of stash seems a bit under-designed IMHO, but if
> people were happy enough with it before, I do not think the safety
> benefit from ed178ef is that great (it is not saving you from destroying
> working tree content, only the index state; the individual content blobs
> are still available from git-fsck).

Yeah, I agree.   Somebody care to do the log message?

This is a tangent, but I'd actually not just agree with "'stash -k'
is a bit under-designed" but also would say something stronger than
that, IMHO ;-)

The very original idea of "git stash" to ssave away working tree
changes with respect to HEAD without bothering the index was at
least internally consistent.  You save away and then may later pop
the change on top of a different commit after dealing with an
emergency, at which time there may be conflicts that you would need
to resolve, and because it is only between HEAD and working tree,
you can safely use the index to resolve the conflicts just like in
any other situation when you help Git to resolve them.  In that
context, the flaw ed178ef1 (stash: require a clean index to apply,
2015-04-22) tried to correct made a lot of sense.

What broke the entire system was the throwing the "save the index,
too" into the mix, which was done without careful thinking.  While
it made the command more useful (when it works), it made the command
internally inconsistent---the command has to punt restoring index
state when it can't, for example, losing the state it tried to save.
I think that is where the "under-designed" started, I would think;
of course, "-k" needed to build on top of "save also the index",
but that is not the primary culprit.

[PATCH] Revert "stash: require a clean index to apply"

[Jeff King]

On Mon, Jun 15, 2015 at 10:42:18AM -0700, Junio C Hamano wrote:

> > I dunno. With respect to the original patch, I am OK if we just want to
> > revert it. This area of stash seems a bit under-designed IMHO, but if
> > people were happy enough with it before, I do not think the safety
> > benefit from ed178ef is that great (it is not saving you from destroying
> > working tree content, only the index state; the individual content blobs
> > are still available from git-fsck).
> 
> Yeah, I agree.   Somebody care to do the log message?

Patch is below. It's a straight revert. The other option would be to
allow it with "--force", and teach people to use that. I'm not sure it's
worth the effort.

> This is a tangent, but I'd actually not just agree with "'stash -k'
> is a bit under-designed" but also would say something stronger than
> that, IMHO ;-)

Yeah, I agree with everything you said here. :)

-- >8 --
Subject: Revert "stash: require a clean index to apply"

This reverts commit ed178ef13a26136d86ff4e33bb7b1afb5033f908.

That commit was an attempt to improve the safety of applying
a stash, because the application process may create
conflicted index entries, after which it is hard to restore
the original index state.

Unfortunately, this hurts some common workflows around "git
stash -k", like:

    git add -p       ;# (1) stage set of proposed changes
    git stash -k     ;# (2) get rid of everything else
    make test        ;# (3) make sure proposal is reasonable
    git stash apply  ;# (4) restore original working tree

If you "git commit" between steps (3) and (4), then this
just works. However, if these steps are part of a pre-commit
hook, you don't have that opportunity (you have to restore
the original state regardless of whether the tests passed or
failed).

It's possible that we could provide better tools for this
sort of workflow. In particular, even before ed178ef, it
could fail with a conflict if there were conflicting hunks
in the working tree and index (since the "stash -k" puts the
index version into the working tree, and we then attempt to
apply the differences between HEAD and the old working tree
on top of that). But the fact remains that people have been
using it happily for a while, and the safety provided by
ed178ef is simply not that great. Let's revert it for now.
In the long run, people can work on improving stash for this
sort of workflow, but the safety tradeoff is not worth it in
the meantime.

Signed-off-by: Jeff King <peff@peff.net>
---
This is directly on jk/stash-require-clean-index, but it should merge
fine up to master.

 git-stash.sh     | 2 --
 t/t3903-stash.sh | 7 -------
 2 files changed, 9 deletions(-)

diff --git a/git-stash.sh b/git-stash.sh
index cc28368..d4cf818 100755
--- a/git-stash.sh
+++ b/git-stash.sh
@@ -442,8 +442,6 @@ apply_stash () {
 	assert_stash_like "$@"
 
 	git update-index -q --refresh || die "$(gettext "unable to refresh index")"
-	git diff-index --cached --quiet --ignore-submodules HEAD -- ||
-		die "$(gettext "Cannot apply stash: Your index contains uncommitted changes.")"
 
 	# current index state
 	c_tree=$(git write-tree) ||
diff --git a/t/t3903-stash.sh b/t/t3903-stash.sh
index 0746eee..f179c93 100755
--- a/t/t3903-stash.sh
+++ b/t/t3903-stash.sh
@@ -45,13 +45,6 @@ test_expect_success 'applying bogus stash does nothing' '
 	test_cmp expect file
 '
 
-test_expect_success 'apply requires a clean index' '
-	test_when_finished "git reset --hard" &&
-	echo changed >other-file &&
-	git add other-file &&
-	test_must_fail git stash apply
-'
-
 test_expect_success 'apply does not need clean working directory' '
 	echo 4 >other-file &&
 	git stash apply &&
-- 
2.4.3.699.g84b4da7

Re: [PATCH] Revert "stash: require a clean index to apply"

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Subject: Revert "stash: require a clean index to apply"
>
> This reverts commit ed178ef13a26136d86ff4e33bb7b1afb5033f908.

Thanks.

Re: [PATCH] Revert "stash: require a clean index to apply"

[Jonathan Kamens]

Is this revert going to be applied and released?

Sent from my Android device



-----Original Message-----
From: Junio C Hamano <gitster@pobox.com>
To: Jeff King <peff@peff.net>
Cc: "bär" <crashcookie@gmail.com>, Jonathan Kamens <jkamens@quantopian.com>, Git List <git@vger.kernel.org>
Sent: Mon, 15 Jun 2015 4:11 PM
Subject: Re: [PATCH] Revert "stash: require a clean index to apply"

Jeff King <peff@peff.net> writes:

> Subject: Revert "stash: require a clean index to apply"
>
> This reverts commit ed178ef13a26136d86ff4e33bb7b1afb5033f908.

Thanks.

Re: [PATCH] Revert "stash: require a clean index to apply"

[Jeff King]

On Thu, Jun 25, 2015 at 05:49:11PM -0400, Jonathan Kamens wrote:

> Is this revert going to be applied and released?

It is on "master", and part of v2.5.0-rc0 (it is not part of v2.4.x, because
the original problem was not there, either).

-Peff

Re: [PATCH] Revert "stash: require a clean index to apply"

[Jonathan Kamens]

Um.

I encountered this issue in git 2.4.3 on Fedora 22.

These lines appear in /usr/libexec/git-core/git-stash on my Fedora 22 
system:

         git diff-index --cached --quiet --ignore-submodules HEAD -- ||
                 die "$(gettext "Cannot apply stash: Your index contains 
uncommitted changes.")"

They also appear in 
https://github.com/git/git/blob/69f9a6e54a46c4a75dff680047a465d04cca20ca/git-stash.sh#L445 
, which is the commit tagged v2.4.3.

In fact, it appears they were released in v2.4.2, at least according to 
https://github.com/git/git/compare/v2.4.1...v2.4.2 .

So it appears to me that this patch was, in fact, released in v2.4.x and 
therefore needs to be reverted in v2.4.x.

   jik

On 06/25/2015 08:27 PM, Jeff King wrote:
> On Thu, Jun 25, 2015 at 05:49:11PM -0400, Jonathan Kamens wrote:
>
>> Is this revert going to be applied and released?
> It is on "master", and part of v2.5.0-rc0 (it is not part of v2.4.x, because
> the original problem was not there, either).
>
> -Peff

Re: [PATCH] Revert "stash: require a clean index to apply"

[Jeff King]

On Thu, Jun 25, 2015 at 09:12:55PM -0400, Jonathan Kamens wrote:

> I encountered this issue in git 2.4.3 on Fedora 22.

Ah, sorry, you're right. I must have fed the wrong sha1 to "git tag
--contains" earlier.

I agree it can probably go onto the v2.4.x maintenance track. It is
already in v2.5.0-rc0.

-Peff

Re: [PATCH] Revert "stash: require a clean index to apply"

[Junio C Hamano]

On Thu, Jun 25, 2015 at 9:03 PM, Jeff King <peff@peff.net> wrote:
> On Thu, Jun 25, 2015 at 09:12:55PM -0400, Jonathan Kamens wrote:
>
>> I encountered this issue in git 2.4.3 on Fedora 22.
>
> Ah, sorry, you're right. I must have fed the wrong sha1 to "git tag
> --contains" earlier.
>
> I agree it can probably go onto the v2.4.x maintenance track. It is
> already in v2.5.0-rc0.

Yeah, thanks for clarifying while I was away. As with any other changes,
a revert also follows the "master first and then maintenance tracks later"
pattern. Perhaps in 2.4.6 which I expect we would do during the 2.5-rc
period.