[PATCH] new test fails "add -p" for adds on the top line

[PATCH] new test fails "add -p" for adds on the top line

[Matt Graham]

add -p doesn't work for some diffs.  diffs adding a new line at the top of
the file with other adds later in the file are one way to trigger the problem.

during add -p, split the diff and then answer y for all segments.  the file
won't have been added to the index.

Signed-off-by: Matthew Graham <mdg149@gmail.com>
---
 t/t3701-add-interactive.sh |   32 ++++++++++++++++++++++++++++++++
 1 files changed, 32 insertions(+), 0 deletions(-)

diff --git a/t/t3701-add-interactive.sh b/t/t3701-add-interactive.sh
index dfc6560..45da6c8 100755
--- a/t/t3701-add-interactive.sh
+++ b/t/t3701-add-interactive.sh
@@ -163,6 +163,38 @@ test_expect_success FILEMODE 'stage mode but not hunk' '
 	git diff          file | grep "+content"
 '

+# Write the patch file with a new line at the top and bottom
+cat >patch <<EOF
+index 180b47c..b6f2c08 100644
+--- a/file
++++ b/file
+@@ -1,2 +1,4 @@
++firstline
+ baseline
+ content
++lastline
+EOF
+# Expected output, similar to the patch but w/ diff at the top
+cat >expected <<EOF
+diff --git a/file b/file
+index b6f2c08..61b9053 100755
+--- a/file
++++ b/file
+@@ -1,2 +1,4 @@
++firstline
+ baseline
+ content
++lastline
+EOF
+# Test splitting the first patch, then adding both
+test_expect_failure 'add first line works' '
+	git commit -am "clear local changes" &&
+	git apply patch &&
+	(echo s; echo y; echo y) | git add -p file &&
+	git diff --cached > diff &&
+	test_cmp expected diff
+'
+
 # end of tests disabled when filemode is not usable

 test_done
-- 
1.6.3.9.g6345

Re: [PATCH] new test fails "add -p" for adds on the top line

[Nanako Shiraishi]

Quoting Matt Graham <mdg149@gmail.com>:

> add -p doesn't work for some diffs.  diffs adding a new line at the top of
> the file with other adds later in the file are one way to trigger the problem.
>
> during add -p, split the diff and then answer y for all segments.  the file
> won't have been added to the index.
>
> Signed-off-by: Matthew Graham <mdg149@gmail.com>

I tried "git-add -p" from different versions and I found out that versions before the commit 0beee4c6dec15292415e3d56075c16a76a22af54 doesn't have this problem.

commit 0beee4c6dec15292415e3d56075c16a76a22af54
Author: Thomas Rast <trast@student.ethz.ch>
Date:   Wed Jul 2 23:59:44 2008 +0200

    git-add--interactive: remove hunk coalescing
    
    Current git-apply has no trouble at all applying chunks that have
    overlapping context, as produced by the splitting feature. So we can
    drop the manual coalescing.
    
    Signed-off-by: Thomas Rast <trast@student.ethz.ch>
    Signed-off-by: Junio C Hamano <gitster@pobox.com>

-- 
Nanako Shiraishi
http://ivory.ap.teacup.com/nanako3/

Re: [PATCH] new test fails "add -p" for adds on the top line

[Thomas Rast]

[-- Attachment #1: Type: text/plain, Size: 2133 bytes --]

Nanako Shiraishi wrote:
> Quoting Matt Graham <mdg149@gmail.com>:
> 
> > add -p doesn't work for some diffs.  diffs adding a new line at the top of
> > the file with other adds later in the file are one way to trigger the problem.
> >
> > during add -p, split the diff and then answer y for all segments.  the file
> > won't have been added to the index.
> >
> > Signed-off-by: Matthew Graham <mdg149@gmail.com>
> 
> I tried "git-add -p" from different versions and I found out that versions before the commit 0beee4c6dec15292415e3d56075c16a76a22af54 doesn't have this problem.
> 
> commit 0beee4c6dec15292415e3d56075c16a76a22af54
> Author: Thomas Rast <trast@student.ethz.ch>
> Date:   Wed Jul 2 23:59:44 2008 +0200
> 
>     git-add--interactive: remove hunk coalescing
>     
>     Current git-apply has no trouble at all applying chunks that have
>     overlapping context, as produced by the splitting feature. So we can
>     drop the manual coalescing.
>     
>     Signed-off-by: Thomas Rast <trast@student.ethz.ch>
>     Signed-off-by: Junio C Hamano <gitster@pobox.com>

The above commit still reverts cleanly, but AFAICS merge_hunk blindly
trusts the hunk headers, an assumption that is no longer valid due to
the 'edit' feature.  So either we need to recount the hunk headers
prior to merging (which was rejected back in the 'edit' feature
discussion due to code complexity) or find some other solution.

Passing either --unidiff-zero or -C1 with the failing patch fixes the
problem, but oddly (to me at least) -C2 does not.  The generated error
looks like

  $ git apply --check -v -C2 < patch
  Checking patch file...
  error: while searching for:
  baseline
  content

  error: patch failed: file:1
  error: file: patch does not apply

The corresponding call (builtin-apply.c:2093) is

			error("while searching for:\n%.*s",
			      (int)(old - oldlines), oldlines);

so it does not seem to insert the extra newline.  Is it actually
looking for a blank line in the context?  If so, wouldn't that be a
git-apply bug?

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [PATCH] new test fails "add -p" for adds on the top line

[Junio C Hamano]

Thomas Rast <trast@student.ethz.ch> writes:

> Nanako Shiraishi wrote:
>> Quoting Matt Graham <mdg149@gmail.com>:
>> 
>> > add -p doesn't work for some diffs.  diffs adding a new line at the top of
>> > the file with other adds later in the file are one way to trigger the problem.
>> >
>> > during add -p, split the diff and then answer y for all segments.  the file
>> > won't have been added to the index.
>> >
>> > Signed-off-by: Matthew Graham <mdg149@gmail.com>
>> 
>> I tried "git-add -p" from different versions and I found out that versions before the commit 0beee4c6dec15292415e3d56075c16a76a22af54 doesn't have this problem.
>> 
>> commit 0beee4c6dec15292415e3d56075c16a76a22af54
>> Author: Thomas Rast <trast@student.ethz.ch>
>> Date:   Wed Jul 2 23:59:44 2008 +0200
>> 
>>     git-add--interactive: remove hunk coalescing
>>     
>>     Current git-apply has no trouble at all applying chunks that have
>>     overlapping context, as produced by the splitting feature. So we can
>>     drop the manual coalescing.
>>     
>>     Signed-off-by: Thomas Rast <trast@student.ethz.ch>
>>     Signed-off-by: Junio C Hamano <gitster@pobox.com>
>
> The above commit still reverts cleanly, but AFAICS merge_hunk blindly
> trusts the hunk headers, an assumption that is no longer valid due to
> the 'edit' feature.

Heh, here is my "I told you so" moment ;-).

We could also remove that "edit" thing; I do not use nor trust it
(fundamentally you cannot trust it).

But how about doing it this way?

-- >8 --
Subject: Revert "git-add--interactive: remove hunk coalescing"

This reverts commit 0beee4c6dec15292415e3d56075c16a76a22af54 but with a
bit of twist, as we have added "edit hunk manually" hack and we cannot
rely on the original line numbers of the hunks that were manually edited.
---
 git-add--interactive.perl  |   96 +++++++++++++++++++++++++++++++++++++++++++-
 t/t3701-add-interactive.sh |    2 +-
 2 files changed, 96 insertions(+), 2 deletions(-)

diff --git a/git-add--interactive.perl b/git-add--interactive.perl
index f6e536e..a06172c 100755
--- a/git-add--interactive.perl
+++ b/git-add--interactive.perl
@@ -767,6 +767,96 @@ sub split_hunk {
 	return @split;
 }
 
+sub find_last_o_ctx {
+	my ($it) = @_;
+	my $text = $it->{TEXT};
+	my ($o_ofs, $o_cnt) = parse_hunk_header($text->[0]);
+	my $i = @{$text};
+	my $last_o_ctx = $o_ofs + $o_cnt;
+	while (0 < --$i) {
+		my $line = $text->[$i];
+		if ($line =~ /^ /) {
+			$last_o_ctx--;
+			next;
+		}
+		last;
+	}
+	return $last_o_ctx;
+}
+
+sub merge_hunk {
+	my ($prev, $this) = @_;
+	my ($o0_ofs, $o0_cnt, $n0_ofs, $n0_cnt) =
+	    parse_hunk_header($prev->{TEXT}[0]);
+	my ($o1_ofs, $o1_cnt, $n1_ofs, $n1_cnt) =
+	    parse_hunk_header($this->{TEXT}[0]);
+
+	my (@line, $i, $ofs, $o_cnt, $n_cnt);
+	$ofs = $o0_ofs;
+	$o_cnt = $n_cnt = 0;
+	for ($i = 1; $i < @{$prev->{TEXT}}; $i++) {
+		my $line = $prev->{TEXT}[$i];
+		if ($line =~ /^\+/) {
+			$n_cnt++;
+			push @line, $line;
+			next;
+		}
+
+		last if ($o1_ofs <= $ofs);
+
+		$o_cnt++;
+		$ofs++;
+		if ($line =~ /^ /) {
+			$n_cnt++;
+		}
+		push @line, $line;
+	}
+
+	for ($i = 1; $i < @{$this->{TEXT}}; $i++) {
+		my $line = $this->{TEXT}[$i];
+		if ($line =~ /^\+/) {
+			$n_cnt++;
+			push @line, $line;
+			next;
+		}
+		$ofs++;
+		$o_cnt++;
+		if ($line =~ /^ /) {
+			$n_cnt++;
+		}
+		push @line, $line;
+	}
+	my $head = ("@@ -$o0_ofs" .
+		    (($o_cnt != 1) ? ",$o_cnt" : '') .
+		    " +$n0_ofs" .
+		    (($n_cnt != 1) ? ",$n_cnt" : '') .
+		    " @@\n");
+	@{$prev->{TEXT}} = ($head, @line);
+}
+
+sub coalesce_overlapping_hunks {
+	my (@in) = @_;
+	my @out = ();
+
+	my ($last_o_ctx, $last_was_dirty);
+
+	for (grep { $_->{USE} } @in) {
+		my $text = $_->{TEXT};
+		my ($o_ofs) = parse_hunk_header($text->[0]);
+		if (defined $last_o_ctx &&
+		    $o_ofs <= $last_o_ctx &&
+		    !$_->{DIRTY} &&
+		    !$last_was_dirty) {
+			merge_hunk($out[-1], $_);
+		}
+		else {
+			push @out, $_;
+		}
+		$last_o_ctx = find_last_o_ctx($out[-1]);
+		$last_was_dirty = $_->{DIRTY};
+	}
+	return @out;
+}
 
 sub color_diff {
 	return map {
@@ -878,7 +968,8 @@ sub edit_hunk_loop {
 		my $newhunk = {
 			TEXT => $text,
 			TYPE => $hunk->[$ix]->{TYPE},
-			USE => 1
+			USE => 1,
+			DIRTY => 1,
 		};
 		if (diff_applies($head,
 				 @{$hunk}[0..$ix-1],
@@ -1210,6 +1301,8 @@ sub patch_update_file {
 		}
 	}
 
+	@hunk = coalesce_overlapping_hunks(@hunk);
+
 	my $n_lofs = 0;
 	my @result = ();
 	for (@hunk) {
@@ -1224,6 +1317,7 @@ sub patch_update_file {
 		open $fh, '| git apply --cached --recount';
 		for (@{$head->{TEXT}}, @result) {
 			print $fh $_;
+			print STDERR $_;
 		}
 		if (!close $fh) {
 			for (@{$head->{TEXT}}, @result) {
diff --git a/t/t3701-add-interactive.sh b/t/t3701-add-interactive.sh
index 45da6c8..c5220a1 100755
--- a/t/t3701-add-interactive.sh
+++ b/t/t3701-add-interactive.sh
@@ -187,7 +187,7 @@ index b6f2c08..61b9053 100755
 +lastline
 EOF
 # Test splitting the first patch, then adding both
-test_expect_failure 'add first line works' '
+test_expect_success 'add first line works' '
 	git commit -am "clear local changes" &&
 	git apply patch &&
 	(echo s; echo y; echo y) | git add -p file &&
-- 
1.6.3.1.9.g95405b

Re: [PATCH] new test fails "add -p" for adds on the top line

[Sverre Rabbelier]

Heya,

On Sat, May 16, 2009 at 19:48, Junio C Hamano <gitster@pobox.com> wrote:
> We could also remove that "edit" thing; I do not use nor trust it
> (fundamentally you cannot trust it).

But it is very useful to split up patches! Of course, I always verify
the end result (usually by diffing against HEAD@{...}), but it is
definitely a very useful tool that I would hate to see removed :(.

-- 
Cheers,

Sverre Rabbelier

Re: [PATCH] new test fails "add -p" for adds on the top line

[Junio C Hamano]

Sverre Rabbelier <srabbelier@gmail.com> writes:

> Heya,
>
> On Sat, May 16, 2009 at 19:48, Junio C Hamano <gitster@pobox.com> wrote:
>> We could also remove that "edit" thing; I do not use nor trust it
>> (fundamentally you cannot trust it).
>
> But it is very useful to split up patches! Of course, I always verify
> the end result (usually by diffing against HEAD@{...}), but it is
> definitely a very useful tool that I would hate to see removed :(.

Sorry, forgot a smiley ;-).

Re: [PATCH] new test fails "add -p" for adds on the top line

[Sverre Rabbelier]

Heya,

On Sat, May 16, 2009 at 21:13, Junio C Hamano <gitster@pobox.com> wrote:
> Sorry, forgot a smiley ;-).

Phew, what would be without those :).

-- 
Cheers,

Sverre Rabbelier

Re: [PATCH] new test fails "add -p" for adds on the top line

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Thomas Rast <trast@student.ethz.ch> writes:
> ...
>> The above commit still reverts cleanly, but AFAICS merge_hunk blindly
>> trusts the hunk headers, an assumption that is no longer valid due to
>> the 'edit' feature.
>
> Heh, here is my "I told you so" moment ;-).

It never blindly trusted before the edit 'feature'; it counted carefully
and it could do so because it had all the necessary information.

I told you that 'edit' could remember the line offset and line numbers
before giving the buffer to the end user, and then recount and adjust the
count after getting the edited results back, to update the offset and
count with the same carefulness.  You (and I think there was somebody else
who was helping) didn't listen.

Fundamentally, after you remove some hunks (and worse yet, you modify
some) from the patch and feed that to "git apply --recount", it can never
do as thorough a job as you could do inside "add -p" itself.  The latter
has more information (the omitted hunks, and the hunks before/after the
user edited) necessary to reconstruct the line numbers and hunk size.  To
keep the whole process more robust and trustworthy, you must do the
necessary computation while you still have all the information about the
hunks you are not feeding to the downstream.

That was what the "I told you so" was about in my message.

It is not too late to teach the 'edit hack' to do so.  That would allow us
to remove the "$_->{DIRTY}" bit my "how about this" patch adds, and I'll
stop calling it the 'edit hack' and start calling it the 'edit feature'
when that happens ;-).

But at least the "how about this" patch should restore the original
behaviour as long as the user does not use the 'edit hack' for now.