quiltimport mode detection oddity

quiltimport mode detection oddity

[Andrew Morton]

Hi all.

hp2:/usr/src/mm> git --version
git version 2.43.0


I'm getting an odd warning from quiltimport:

hp2:/usr/src/mm> ls -l tools/testing/radix-tree/generated/autoconf.h
-rw-rw-r-- 1 akpm akpm 54 Aug  1 15:43 tools/testing/radix-tree/generated/autoconf.h

hp2:/usr/src/mm> git quiltimport --series series
tools-separate-out-shared-radix-tree-components.patch
warning: tools/testing/radix-tree/generated/autoconf.h has type 100644, expected 100664




That patch has

diff --git a/tools/testing/radix-tree/generated/autoconf.h a/tools/testing/radix-tree/generated/autoconf.h
deleted file mode 100664
--- a/tools/testing/radix-tree/generated/autoconf.h
+++ /dev/null
@@ -1,2 +0,0 @@
-#include "bit-length.h"
-#define CONFIG_XARRAY_MULTI 1




after quiltimport:

hp2:/usr/src/mm> ls -l tools/testing/radix-tree/generated/autoconf.h
ls: cannot access 'tools/testing/radix-tree/generated/autoconf.h': No such file or directory



I can't figure what I've done to make quiltimport (git-apply?) think that the file
had 100644 permissions.  Maybe the lack of an index line tripped it up.


(btw, "has type" should be "has permissions" in that message, no?)


Thanks.

Re: quiltimport mode detection oddity

[Junio C Hamano]

Andrew Morton <akpm@linux-foundation.org> writes:

> hp2:/usr/src/mm> git quiltimport --series series
> tools-separate-out-shared-radix-tree-components.patch
> warning: tools/testing/radix-tree/generated/autoconf.h has type 100644, expected 100664
>
>
>
>
> That patch has
>
> diff --git a/tools/testing/radix-tree/generated/autoconf.h a/tools/testing/radix-tree/generated/autoconf.h
> deleted file mode 100664
> --- a/tools/testing/radix-tree/generated/autoconf.h
> +++ /dev/null
> @@ -1,2 +0,0 @@
> -#include "bit-length.h"
> -#define CONFIG_XARRAY_MULTI 1

So, the patch removes autoconf.h file from that directory.  The
"extended header" part between "diff --git" and "--- a/..." has
"deleted file mode 100664" and that is where the warning comes.

I do not quite recall at which point "git quiltimport" calls "git
apply", but the "has type 100644, expected 100664" does ring a bell.

> after quiltimport:
>
> hp2:/usr/src/mm> ls -l tools/testing/radix-tree/generated/autoconf.h
> ls: cannot access 'tools/testing/radix-tree/generated/autoconf.h': No such file or directory

That is to be expected, if that patch was successfully applied, no?
After all, the patch you quoted above seems to be a removal of
autoconf.h from that path.

> I can't figure what I've done to make quiltimport (git-apply?) think that the file
> had 100644 permissions.  Maybe the lack of an index line tripped it up.

You said "That patch has", and I take it to mean that the input
material before "git quiltimport" touched it already had the
extended header that records the removal of a file whose mode is
100664?  

And lack of the index line is probably a red herring.  EVen if there
were an index line, it would just have recorded the two object names
(the blob object name of the original contents before removal,
followed by a double-dot "..", followed by all-0 to signal removal).
We do not read mode bits out of that line.

> (btw, "has type" should be "has permissions" in that message, no?)

If leading prefix 100 did not exist, yes, permissions would be more
appropriate, but if the prefix changed from 100644 to say 120000,
that would be a type change from plain blob to a symlink.  So "type"
is not quite wrong, either.

Re: quiltimport mode detection oddity

[Andrew Morton]

On Thu, 01 Aug 2024 17:33:48 -0700 Junio C Hamano <gitster@pobox.com> wrote:

> > hp2:/usr/src/mm> git quiltimport --series series
> > tools-separate-out-shared-radix-tree-components.patch
> > warning: tools/testing/radix-tree/generated/autoconf.h has type 100644, expected 100664
> >
> >
> >
> >
> > That patch has
> >
> > diff --git a/tools/testing/radix-tree/generated/autoconf.h a/tools/testing/radix-tree/generated/autoconf.h
> > deleted file mode 100664
> > --- a/tools/testing/radix-tree/generated/autoconf.h
> > +++ /dev/null
> > @@ -1,2 +0,0 @@
> > -#include "bit-length.h"
> > -#define CONFIG_XARRAY_MULTI 1
> 
> So, the patch removes autoconf.h file from that directory.  The
> "extended header" part between "diff --git" and "--- a/..." has
> "deleted file mode 100664" and that is where the warning comes.

yup yup.  The patch says "remove this file which has mode 100664".

The file has mode 100664.

quiltimport says it had mode 100644.  Incorrectly, I suggest.

Re: quiltimport mode detection oddity

[Jeff King]

On Thu, Aug 01, 2024 at 06:07:06PM -0700, Andrew Morton wrote:

> > So, the patch removes autoconf.h file from that directory.  The
> > "extended header" part between "diff --git" and "--- a/..." has
> > "deleted file mode 100664" and that is where the warning comes.
> 
> yup yup.  The patch says "remove this file which has mode 100664".
> 
> The file has mode 100664.
> 
> quiltimport says it had mode 100644.  Incorrectly, I suggest.

It's definitely a weird case. Git does not record full modes, but just
cares about the execute bit. So it normalizes modes for regular files to
100644 or 100755. You can see that with a simple example:

  git init
  echo foo >file
  chmod 664 file
  git add file
  git commit -m 'add file'

  git ls-files -s

  cat >patch <<\EOF
  diff --git a/file b/file
  deleted file mode 100664
  --- a/file
  +++ /dev/null
  @@ -1 +0,0 @@
  -foo
  EOF
  ls -l file
  git apply patch

Even though the filesystem has 100664, the index records 100644 (which
you can see from the "ls-files" output). And then when we apply the
patch, we get the "file has type 100644, expected 100664" message.
AFAICT, it has been that way forever (I tried as far back as git 1.6.6).
So this is nothing new, and I don't think Git would ever produce a patch
that said "file mode 100664" itself (I'm assuming in your case the patch
is coming from quilt).

Given that, I think it is reasonable for git to also normalize the mode
of the patches it reads, so that we are consistently working in the
world of simplified modes. I.e., this:

diff --git a/apply.c b/apply.c
index 142e3d913c..3d50fade78 100644
--- a/apply.c
+++ b/apply.c
@@ -995,6 +995,7 @@ static int parse_mode_line(const char *line, int linenr, unsigned int *mode)
 	*mode = strtoul(line, &end, 8);
 	if (end == line || !isspace(*end))
 		return error(_("invalid mode on line %d: %s"), linenr, line);
+	*mode = canon_mode(*mode);
 	return 0;
 }
 

which makes the warning go away in the example above. But I'm not sure
if there could be other fallout. E.g., is there a mode for git-apply to
just touch the working tree and not the index, where we'd perhaps want
to retain the original to compare against the filesystem mode? I don't
think so.

Alternatively (or maybe in addition), I wonder if quilt should similarly
canonicalize the mode. git-apply is certainly meant to work with patches
generated elsewhere, but normal patches don't have modes in them at all.
The "deleted file mode" line is git-ism, so here we have something which
is implementing the git line in a (slightly) incompatible way.

-Peff

Re: quiltimport mode detection oddity

[Andrew Morton]

On Thu, 1 Aug 2024 23:51:21 -0400 Jeff King <peff@peff.net> wrote:

> On Thu, Aug 01, 2024 at 06:07:06PM -0700, Andrew Morton wrote:
> 
> > > So, the patch removes autoconf.h file from that directory.  The
> > > "extended header" part between "diff --git" and "--- a/..." has
> > > "deleted file mode 100664" and that is where the warning comes.
> > 
> > yup yup.  The patch says "remove this file which has mode 100664".
> > 
> > The file has mode 100664.
> > 
> > quiltimport says it had mode 100644.  Incorrectly, I suggest.
> 
> It's definitely a weird case. Git does not record full modes, but just
> cares about the execute bit. So it normalizes modes for regular files to
> 100644 or 100755. You can see that with a simple example:
> 
>   git init
>   echo foo >file
>   chmod 664 file
>   git add file
>   git commit -m 'add file'
> 
>   git ls-files -s
> 
>   cat >patch <<\EOF
>   diff --git a/file b/file
>   deleted file mode 100664
>   --- a/file
>   +++ /dev/null
>   @@ -1 +0,0 @@
>   -foo
>   EOF
>   ls -l file
>   git apply patch
> 
> Even though the filesystem has 100664, the index records 100644 (which
> you can see from the "ls-files" output). And then when we apply the
> patch, we get the "file has type 100644, expected 100664" message.

OK.

> AFAICT, it has been that way forever (I tried as far back as git 1.6.6).
> So this is nothing new, and I don't think Git would ever produce a patch
> that said "file mode 100664" itself (I'm assuming in your case the patch
> is coming from quilt).

yup.

> Alternatively (or maybe in addition), I wonder if quilt should similarly
> canonicalize the mode.

I'll hard code 100644 ;)

> generated elsewhere, but normal patches don't have modes in them at all.
> The "deleted file mode" line is git-ism, so here we have something which
> is implementing the git line in a (slightly) incompatible way.

yup, thanks.

Re: quiltimport mode detection oddity

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Given that, I think it is reasonable for git to also normalize the mode
> of the patches it reads, so that we are consistently working in the
> world of simplified modes. I.e., this:
>
> diff --git a/apply.c b/apply.c
> index 142e3d913c..3d50fade78 100644
> --- a/apply.c
> +++ b/apply.c
> @@ -995,6 +995,7 @@ static int parse_mode_line(const char *line, int linenr, unsigned int *mode)
>  	*mode = strtoul(line, &end, 8);
>  	if (end == line || !isspace(*end))
>  		return error(_("invalid mode on line %d: %s"), linenr, line);
> +	*mode = canon_mode(*mode);
>  	return 0;
>  }
>  
>
> which makes the warning go away in the example above. But I'm not sure
> if there could be other fallout. E.g., is there a mode for git-apply to
> just touch the working tree and not the index, where we'd perhaps want
> to retain the original to compare against the filesystem mode? I don't
> think so.

Makes sense.

The above is consistent with what we do for the permission bits;
only the execute bit matters, and the patch recording 100664 should
mean the same thing to us as permission bits 100644---we should warn
if the on-disk file is executable while applying such a patch, and
we should not warn otherwise.

> Alternatively (or maybe in addition), I wonder if quilt should similarly
> canonicalize the mode. git-apply is certainly meant to work with patches
> generated elsewhere, but normal patches don't have modes in them at all.
> The "deleted file mode" line is git-ism, so here we have something which
> is implementing the git line in a (slightly) incompatible way.

It's an orthogonal fix and probably worth doing.

If a third-party tool adds git-ism mode lines, we should be lenient
when we see a wrong mode, as long as the leniency does not affect
our normal mode of operation negatively.  It is OK if they record a
non-executable regular file with 100666.  Using 664 (no type bits)
or 100755, however, crosses the line and they must stop producing
such a bogus mode line, if they do not want to see a warning.

[PATCH] apply: canonicalize modes read from patches

[Jeff King]

On Fri, Aug 02, 2024 at 07:57:54AM -0700, Junio C Hamano wrote:

> Makes sense.
> 
> The above is consistent with what we do for the permission bits;
> only the execute bit matters, and the patch recording 100664 should
> mean the same thing to us as permission bits 100644---we should warn
> if the on-disk file is executable while applying such a patch, and
> we should not warn otherwise.

OK, here it is with tests and a commit message. I dug around to make
sure there were no cases where the unusual mode would cause other
behavior changes, but there aren't any. We are careful to use the
canonical mode whenever we create a file.

So the tests here may be overkill (since except for the warning message,
they'd pass already), but I thought it worth demonstrating the complete
set of expected behavior. Likewise the commit message is long because I
laid out all of the things I poked at.

I didn't add tests confirming that we complain when the executable bit
is not as expected. Earlier tests in t4129 already cover that.

-- >8 --
Subject: apply: canonicalize modes read from patches

Git stores only canonical modes for blobs. So for a regular file, we
care about only "100644" or "100755" (depending only on the executable
bit), but never modes where the group or other permissions are more
exotic. So never "100664", "100700", etc. When a file in the working
tree has such a mode, we quietly turn it into one of the two canonical
modes, and that's what is stored both in the index and in tree objects.

However, we don't canonicalize modes we read from incoming patches in
git-apply. These may appear in a few lines:

  - "old mode" / "new mode" lines for mode changes

  - "new file mode" lines for newly created files

  - "deleted file mode" for removing files

For "new mode" and for "new file mode", this is harmless. The patch is
asking the result to have a certain mode, but:

  - when we add an index entry (for --index or --cached), it is
    canonicalized as we create the entry, via create_ce_mode().

  - for a working tree file, try_create_file() passes either 0777 or
    0666 to open(), so what you get depends only on your umask, not any
    other bits (aside from the executable bit) in the original mode.

However, for "old mode" and "deleted file mode", there is a minor
annoyance. We compare the patch's expected preimage mode with the
current state. But that current state is always going to be a canonical
mode itself:

  - updating an index entry via --cached will have the canonical mode in
    the index

  - for updating a working tree file, check_preimage() runs the mode
    through ce_mode_from_stat(), which does the usual canonicalization

So if the patch feeds a non-canonical mode, it's impossible for it to
match, and we will always complain with something like:

  file has type 100644, expected 100664

Since this is just a warning, the operation proceeds, but it's
confusing and annoying.

These cases should be pretty rare in practice. Git would never produce a
patch with non-canonical modes itself (since it doesn't store them).
And while we do accept patches from other programs, all of those lines
were invented by Git. So you'd need a program trying to be Git
compatible, but not handling canonicalization the same way. Reportedly
"quilt" is such a program.

We should canonicalize the modes as we read them so that the user never
sees the useless warning.

A few notes on the tests:

  - I've covered instances of all lines for completeness, even though
    the "new mode" / "new file mode" ones behave OK currently.

  - the tests apply patches to both the index and working tree, and
    check the result of both. Again, we know that all of these paths
    canonicalize anyway, but it's giving us extra coverage (although we
    are even less likely to have such a bug now since we canonicalize up
    front).

  - the test patches are missing "index" lines, which is also something
    Git would never produce. But they don't matter for the test, they do
    match the case from quilt we saw in the wild, and they avoid some
    sha1/sha256 complexity.

Reported-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Jeff King <peff@peff.net>
---
 apply.c                   |  1 +
 t/t4129-apply-samemode.sh | 62 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+)

diff --git a/apply.c b/apply.c
index 0f2f5dabe3..6e1060a952 100644
--- a/apply.c
+++ b/apply.c
@@ -995,6 +995,7 @@ static int parse_mode_line(const char *line, int linenr, unsigned int *mode)
 	*mode = strtoul(line, &end, 8);
 	if (end == line || !isspace(*end))
 		return error(_("invalid mode on line %d: %s"), linenr, line);
+	*mode = canon_mode(*mode);
 	return 0;
 }
 
diff --git a/t/t4129-apply-samemode.sh b/t/t4129-apply-samemode.sh
index 4eb8444029..d9a1084b5e 100755
--- a/t/t4129-apply-samemode.sh
+++ b/t/t4129-apply-samemode.sh
@@ -130,4 +130,66 @@ test_expect_success 'git apply respects core.fileMode' '
 	test_grep ! "has type 100644, expected 100755" err
 '
 
+test_expect_success POSIXPERM 'patch mode for new file is canonicalized' '
+	cat >patch <<-\EOF &&
+	diff --git a/non-canon b/non-canon
+	new file mode 100660
+	--- /dev/null
+	+++ b/non-canon
+	+content
+	EOF
+	test_when_finished "git reset --hard" &&
+	(
+		umask 0 &&
+		git apply --index patch 2>err
+	) &&
+	test_must_be_empty err &&
+	git ls-files -s -- non-canon >staged &&
+	test_grep "^100644" staged &&
+	ls -l non-canon >worktree &&
+	test_grep "^-rw-rw-rw" worktree
+'
+
+test_expect_success POSIXPERM 'patch mode for deleted file is canonicalized' '
+	test_when_finished "git reset --hard" &&
+	echo content >non-canon &&
+	git add non-canon &&
+	chmod 666 non-canon &&
+
+	cat >patch <<-\EOF &&
+	diff --git a/non-canon b/non-canon
+	deleted file mode 100660
+	--- a/non-canon
+	+++ /dev/null
+	@@ -1 +0,0 @@
+	-content
+	EOF
+	git apply --index patch 2>err &&
+	test_must_be_empty err &&
+	git ls-files -- non-canon >staged &&
+	test_must_be_empty staged &&
+	test_path_is_missing non-canon
+'
+
+test_expect_success POSIXPERM 'patch mode for mode change is canonicalized' '
+	test_when_finished "git reset --hard" &&
+	echo content >non-canon &&
+	git add non-canon &&
+
+	cat >patch <<-\EOF &&
+	diff --git a/non-canon b/non-canon
+	old mode 100660
+	new mode 100770
+	EOF
+	(
+		umask 0 &&
+		git apply --index patch 2>err
+	) &&
+	test_must_be_empty err &&
+	git ls-files -s -- non-canon >staged &&
+	test_grep "^100755" staged &&
+	ls -l non-canon >worktree &&
+	test_grep "^-rwxrwxrwx" worktree
+'
+
 test_done
-- 
2.46.0.257.g39958a5326

Re: [PATCH] apply: canonicalize modes read from patches

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> +test_expect_success POSIXPERM 'patch mode for deleted file is canonicalized' '

This test seems to fail under "--stress" and I need to borrow a
brain better clued than mine.  It appears to be fooled by mtime that
is not updated immediately and failing match_stat check, but since
the index file is written on the other side of the second resolution
boundary, racy-git double-checking code does not trigger, or
something like that.

Here is how it fails:

expecting success of 4129.13 'patch mode for deleted file is canonicalized':
        test_when_finished "git reset --hard" &&
        echo content >non-canon &&
        git add non-canon &&
        chmod 666 non-canon &&

        cat >patch <<-\EOF &&
        diff --git a/non-canon b/non-canon
        deleted file mode 100660
        --- a/non-canon
        +++ /dev/null
        @@ -1 +0,0 @@
        -content
        EOF
        git apply --index patch 2>err &&
        test_must_be_empty err &&
        git ls-files -- non-canon >staged &&
        test_must_be_empty staged &&
        test_path_is_missing non-canon

++ test_when_finished 'git reset --hard'
++ test 0 = 0
++ test_cleanup='{ git reset --hard
                } && (exit "$eval_ret"); eval_ret=$?; :'
++ echo content
++ git add non-canon
++ chmod 666 non-canon
++ cat
++ git apply --index patch
error: last command exited with $?=1
not ok 13 - patch mode for deleted file is canonicalized
#
#               test_when_finished "git reset --hard" &&
#               echo content >non-canon &&
#               git add non-canon &&
#               chmod 666 non-canon &&
#
#               cat >patch <<-\EOF &&
#               diff --git a/non-canon b/non-canon
#               deleted file mode 100660
#               --- a/non-canon
#               +++ /dev/null
#               @@ -1 +0,0 @@
#               -content
#               EOF
#               git apply --index patch 2>err &&
#               test_must_be_empty err &&
#               git ls-files -- non-canon >staged &&
#               test_must_be_empty staged &&
#               test_path_is_missing non-canon
#
1..13
$ echo $?
1
$ git -C trash\ directory.t4129-apply-samemode.stress-failed/.git ls-files --debug non-canon
non-canon
  ctime: 1723701364:980719772
  mtime: 1723701364:980719772
  dev: 65024    ino: 1980747
  uid: 110493   gid: 89939
  size: 8       flags: 0
: git t/master; stat trash\ directory.t4129-apply-samemode.stress-failed/non-canon
  File: trash directory.t4129-apply-samemode.stress-failed/non-canon
  Size: 8               Blocks: 8          IO Block: 4096   regular file
Device: 254,0   Inode: 1980747     Links: 1
Access: (0666/-rw-rw-rw-)  Uid: (110493/     jch)   Gid: (89939/primarygroup)
Access: 2024-08-15 06:54:43.808293635 -0700
Modify: 2024-08-14 22:56:04.980719772 -0700
Change: 2024-08-14 22:56:05.020719706 -0700
 Birth: 2024-08-14 22:56:04.980719772 -0700
$ stat trash\ directory.t4129-apply-samemode.stress-failed/.git/index
  File: trash directory.t4129-apply-samemode.stress-failed/.git/index
  Size: 432             Blocks: 8          IO Block: 4096   regular file
Device: 254,0   Inode: 1980724     Links: 1
Access: (0600/-rw-------)  Uid: (110493/     jch)   Gid: (89939/primarygroup)
Access: 2024-08-14 22:56:05.044719667 -0700
Modify: 2024-08-14 22:56:05.008719726 -0700
Change: 2024-08-14 22:56:05.016719713 -0700
 Birth: 2024-08-14 22:56:04.996719746 -0700

[PATCH] t4129: fix racy index when calling chmod after git-add

[Jeff King]

On Thu, Aug 15, 2024 at 07:52:47AM -0700, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > +test_expect_success POSIXPERM 'patch mode for deleted file is canonicalized' '
> 
> This test seems to fail under "--stress" and I need to borrow a
> brain better clued than mine.  It appears to be fooled by mtime that
> is not updated immediately and failing match_stat check, but since
> the index file is written on the other side of the second resolution
> boundary, racy-git double-checking code does not trigger, or
> something like that.

Ah, thanks. I saw this fail once in CI, and then later succeed. But for
some reason I wrote it off as CI flakiness rather than trying --stress
locally. It's easy to reproduce the issue.

I didn't puzzle out the exact condition that causes the race, but I
think the fix is pretty clearly this:

-- >8 --
Subject: [PATCH] t4129: fix racy index when calling chmod after git-add

This patch fixes a racy test failure in t4129.

The deletion test added by e95d515141 (apply: canonicalize modes read
from patches, 2024-08-05) wants to make sure that git-apply does not
complain about a non-canonical mode in the patch, even if that mode does
not match the working tree file. So it does this:

	echo content >non-canon &&
	git add non-canon &&
	chmod 666 non-canon &&

This is wrong, because running chmod will update the ctime on the file,
making it stat-dirty and causing git-apply to refuse to apply the patch.
But this only happens sometimes, since it depends on the timestamps
crossing a second boundary (but it triggers pretty quickly when run with
--stress).

We can fix this by doing the chmod before updating the index. The order
isn't important here, as the mode will be canonicalized to 100644 in the
index anyway (in fact, the chmod is not even that important in the first
place, since git-apply will only look at the index; I only added it as
an extra confirmation that git-apply would not be confused by it).

Signed-off-by: Jeff King <peff@peff.net>
---
 t/t4129-apply-samemode.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/t4129-apply-samemode.sh b/t/t4129-apply-samemode.sh
index d9a1084b5e..87ffd2b8e1 100755
--- a/t/t4129-apply-samemode.sh
+++ b/t/t4129-apply-samemode.sh
@@ -153,8 +153,8 @@ test_expect_success POSIXPERM 'patch mode for new file is canonicalized' '
 test_expect_success POSIXPERM 'patch mode for deleted file is canonicalized' '
 	test_when_finished "git reset --hard" &&
 	echo content >non-canon &&
-	git add non-canon &&
 	chmod 666 non-canon &&
+	git add non-canon &&
 
 	cat >patch <<-\EOF &&
 	diff --git a/non-canon b/non-canon
-- 
2.46.0.476.g32f0e7348d

Re: [PATCH] t4129: fix racy index when calling chmod after git-add

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Ah, thanks. I saw this fail once in CI, and then later succeed. But for
> some reason I wrote it off as CI flakiness rather than trying --stress
> locally. It's easy to reproduce the issue.

Thanks.  Will queue.