* RE: [Breakage] t0410 - subtests report unable to remove non-existent file.
2020-03-10 11:00 ` SZEDER Gábor
@ 2020-03-10 14:23 ` Randall S. Becker
2020-03-11 17:51 ` Junio C Hamano
2020-03-12 0:03 ` brian m. carlson
2 siblings, 0 replies; 6+ messages in thread
From: Randall S. Becker @ 2020-03-10 14:23 UTC (permalink / raw)
To: 'SZEDER Gábor'; +Cc: git
On March 10, 2020 7:00 AM, SZEDER Gábor wrote:
> On Fri, Feb 28, 2020 at 05:32:57PM -0500, Randall S. Becker wrote:
> > Starting at t0410, subtest 5 (missing ref object, but promised, passes
> > fsck), on the NonStop L-series platform, we are seeing errors like the
> > following:
> >
> > not ok 5 - missing ref object, but promised, passes fsck #
> > # rm -rf repo &&
> > # test_create_repo repo &&
> > # test_commit -C repo my_commit &&
> > #
> > # A=$(git -C repo commit-tree -m a HEAD^{tree}) &&
> > #
> > # # Reference $A only from ref
> > # git -C repo branch my_branch "$A" &&
> > # promise_and_delete "$A" &&
> > #
> > # git -C repo config core.repositoryformatversion 1 &&
> > # git -C repo config extensions.partialclone "arbitrary
> > string" &&
> > # git -C repo fsck
> > #
> >
> > With verbose output as follows:
>
> Try to run tests with '-x' tracing enabled for additional info about what's
> going on, and thus potentially additional clues about what might go wrong.
>
> > Initialized empty Git repository in /home/ituglib/randall/git/t/trash
> > directory.t0410-partial-clone/repo/.git/
> > [master (root-commit) 9df77b9] my_commit
> > Author: A U Thor <author@example.com>
> > 1 file changed, 1 insertion(+)
> > create mode 100644 my_commit.t
> > Enumerating objects: 1, done.
> > Counting objects: 100% (1/1), done.
> > Writing objects: 100% (1/1), done.
> > Total 1 (delta 0), reused 0 (delta 0)
> > a391e3e0447189aa0050c8f206462a1b0530a34a
> > rm: cannot remove
> > 'repo/.git/objects/a3/91e3e0447189aa0050c8f206462a1b0530a34a': No
> such
> > file or directory
>
> So this failing 'rm' happens inside the 'promise_and_delete' helper function,
> which does the following, simplified a bit for the purpose of this discussion:
>
> promise_and_delete () {
> HASH=$(git -C repo rev-parse "$1") &&
> <...>
> git -C repo rev-parse my_annotated_tag | pack_as_from_promisor &&
> <...>
> delete_object repo "$HASH"
> }
>
> The failing 'rm' is in the 'delete_object_repo' helper function.
> The 'pack_as_from_promisor' does the following:
>
> pack_as_from_promisor () {
> HASH=$(git -C repo pack-objects .git/objects/pack/pack) &&
> >repo/.git/objects/pack/pack-$HASH.promisor &&
> echo $HASH
> }
>
> Notice that both 'promise_and_delete' and 'pack_as_from_promisor' set the
> $HASH variable. This is usually not an issue, because
> 'pack_as_from_promisor' is invoked in a pipe, and thus most shells execute it
> in a subshell environment.
>
> However, apparently 'ksh' doesn't run this helper function in a subshell
> environment, and the value set in 'pack_as_from_promisor'
> overwrites the value set in its caller, thus 'promise_and_delete' ends up
> trying to delete a non-existing object (it's the SHA1 of a packfile).
>
> See the trimmed/annotated '-x' output with 'ksh' from t0410.5:
>
> + promise_and_delete 383670739c2f993999f3c10911ac5cb5c6591523
> + git -C repo rev-parse 383670739c2f993999f3c10911ac5cb5c6591523
> # Setting $HASH in 'promise_and_delete':
> + HASH=383670739c2f993999f3c10911ac5cb5c6591523
> + git -C repo tag -a -m message my_annotated_tag
> 383670739c2f993999f3c10911ac5cb5c6591523
> + pack_as_from_promisor
> + git -C repo rev-parse my_annotated_tag
> + git -C repo pack-objects .git/objects/pack/pack
> Enumerating objects: 1, done.
> Counting objects: 100% (1/1), done.
> Writing objects: 100% (1/1), done.
> Total 1 (delta 0), reused 0 (delta 0)
> # Setting $HASH in 'pack_as_from_promisor', and overwriting its value # in
> the caller:
> + HASH=a391e3e0447189aa0050c8f206462a1b0530a34a
> + 1> repo/.git/objects/pack/pack-
> a391e3e0447189aa0050c8f206462a1b0530a34a.promisor
> + echo a391e3e0447189aa0050c8f206462a1b0530a34a
> a391e3e0447189aa0050c8f206462a1b0530a34a
> + git -C repo tag -d my_annotated_tag
> + 1> /dev/null
> # Using the new value in the caller:
> + delete_object repo a391e3e0447189aa0050c8f206462a1b0530a34a
> + sed -e 's|^..|&/|'
> + echo a391e3e0447189aa0050c8f206462a1b0530a34a
> + rm repo/.git/objects/a3/91e3e0447189aa0050c8f206462a1b0530a34a
> rm: cannot remove
> 'repo/.git/objects/a3/91e3e0447189aa0050c8f206462a1b0530a34a': No
> such file or directory
>
> Note, however, that 'ksh' is not utterly wrong in doing so, because POSIX
> does allow this behavior: POSIX 2.12 Shell Execution Environment, the last
> paragraph of the section:
>
> "each command of a multi-command pipeline is in a subshell
> environment; as an extension, however, any or all commands in a
> pipeline may be executed in the current environment."
>
>
> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.ht
> ml#tag_18_12
>
> So apparently 'ksh' implements this extension.
>
> The trivial fix would be to mark $HASH as 'local' in both helper functions, but
> this would not help 'ksh', of course, as it doesn't support 'local'. However,
> since we use more and more 'local's in our testsuite, 'ksh' might be
> considered a lost cause anyway.
>
> Or we could rename these HASH variables to something more specific to
> prevent this name collision, e.g. PACK_HASH in 'pack_as_from_promisor'.
>
> Note that there are tests in t0410 that set and use the $HASH variable
> outside of these helper function, and, worse, there is a test that uses the
> $HASH variable set in the previous test. Luckily, none of those tests use
> 'promise_and_delete' or 'pack_as_from_promisor'.
I think we are going to consider ksh a lost cause. We have to move to bash because the ksh implementation does not support a large enough environment to run all of the tests, so are now running with:
make SHELL=/usr/coreutils/bin/bash
t0410 passes using bash so we'll stick with that rather than trying to patch ksh that won't run properly inside make as of 2.25.0 (we hit the environment size limit).
Thanks for the response.
Regards,
Randall
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [Breakage] t0410 - subtests report unable to remove non-existent file.
2020-03-10 11:00 ` SZEDER Gábor
2020-03-10 14:23 ` Randall S. Becker
@ 2020-03-11 17:51 ` Junio C Hamano
2020-03-12 0:03 ` brian m. carlson
2 siblings, 0 replies; 6+ messages in thread
From: Junio C Hamano @ 2020-03-11 17:51 UTC (permalink / raw)
To: SZEDER Gábor; +Cc: Randall S. Becker, git
SZEDER Gábor <szeder.dev@gmail.com> writes:
> promise_and_delete () {
> HASH=$(git -C repo rev-parse "$1") &&
> <...>
> git -C repo rev-parse my_annotated_tag | pack_as_from_promisor &&
> <...>
> delete_object repo "$HASH"
> }
>
> The failing 'rm' is in the 'delete_object_repo' helper function.
> The 'pack_as_from_promisor' does the following:
>
> pack_as_from_promisor () {
> HASH=$(git -C repo pack-objects .git/objects/pack/pack) &&
> >repo/.git/objects/pack/pack-$HASH.promisor &&
> echo $HASH
> }
>
> Notice that both 'promise_and_delete' and 'pack_as_from_promisor' set
> the $HASH variable. This is usually not an issue, because
> 'pack_as_from_promisor' is invoked in a pipe, and thus most shells
> execute it in a subshell environment.
>
> However, apparently 'ksh' doesn't run this helper function in a
> subshell environment, and the value set in 'pack_as_from_promisor'
> overwrites the value set in its caller, thus 'promise_and_delete' ends
> up trying to delete a non-existing object (it's the SHA1 of a
> packfile).
> ...
> Note, however, that 'ksh' is not utterly wrong in doing so, because
> POSIX does allow this behavior: POSIX 2.12 Shell Execution Environment,
> the last paragraph of the section:
>
> "each command of a multi-command pipeline is in a subshell
> environment; as an extension, however, any or all commands in a
> pipeline may be executed in the current environment."
>
> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_12
>
> So apparently 'ksh' implements this extension.
>
> The trivial fix would be to mark $HASH as 'local' in both helper
> functions,
Or
git -C repo rev-parse my_annotated_tag |
(pack_as_from_promisor) &&
that would not require local?
> but this would not help 'ksh', of course, as it doesn't
> support 'local'. However, since we use more and more 'local's in our
> testsuite, 'ksh' might be considered a lost cause anyway.
That's somewhat sad, especially since "local" is outside POSIX, but
probably re-whipping our test suite into shape to be usable with
POSIX shells is too much effort for little benefit X-<.
> Or we could rename these HASH variables to something more specific to
> prevent this name collision, e.g. PACK_HASH in 'pack_as_from_promisor'.
>
> Note that there are tests in t0410 that set and use the $HASH variable
> outside of these helper function, and, worse, there is a test that uses
> the $HASH variable set in the previous test. Luckily, none of those
> tests use 'promise_and_delete' or 'pack_as_from_promisor'.
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [Breakage] t0410 - subtests report unable to remove non-existent file.
2020-03-10 11:00 ` SZEDER Gábor
2020-03-10 14:23 ` Randall S. Becker
2020-03-11 17:51 ` Junio C Hamano
@ 2020-03-12 0:03 ` brian m. carlson
2 siblings, 0 replies; 6+ messages in thread
From: brian m. carlson @ 2020-03-12 0:03 UTC (permalink / raw)
To: SZEDER Gábor; +Cc: Randall S. Becker, git
[-- Attachment #1: Type: text/plain, Size: 3600 bytes --]
On 2020-03-10 at 11:00:08, SZEDER Gábor wrote:
> Note, however, that 'ksh' is not utterly wrong in doing so, because
> POSIX does allow this behavior: POSIX 2.12 Shell Execution Environment,
> the last paragraph of the section:
>
> "each command of a multi-command pipeline is in a subshell
> environment; as an extension, however, any or all commands in a
> pipeline may be executed in the current environment."
>
> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_12
>
> So apparently 'ksh' implements this extension.
In my experience, trying to fix this is not productive in our test
suite. zsh does the same thing and many of our tests fail with it for
that reason. In the zsh case, I sent a patch to make it not do that
when emulating sh. It got lost on their list, but I'll resend it.
There are many, many other pieces of software that are broken in such a
case, such that trying to use zsh (or any other shell that does this) as
/bin/sh (or a POSIX sh) will result in widespread breakage. My goal in
fixing zsh is that it's a candidate for /bin/sh in macOS Catalina, and
I'd like to avoid users seeing breakage if they run in that
configuration.
My reading of the allowance for this as an extension is that zsh and ksh
can do this when invoked as zsh or ksh, but they cannot when invoked as
sh, because in that mode they are supposed to strictly conform to POSIX.
POSIX states the following:
Non-standard extensions, when used, may change the behavior of
utilities, functions, or facilities defined by POSIX.1-2017. The
conformance document shall define an environment in which an
application can be run with the behavior specified by POSIX.1-2017.
> The trivial fix would be to mark $HASH as 'local' in both helper
> functions, but this would not help 'ksh', of course, as it doesn't
> support 'local'. However, since we use more and more 'local's in our
> testsuite, 'ksh' might be considered a lost cause anyway.
If this is the AT&T ksh, then yeah, it's a lost cause. It's possible to
emulate local with typeset, but that only works in ksh-style functions,
not POSIX-style functions. It also has the issue we were discussing
above.
pdksh and mksh both do the right thing with local and the last item in a
pipeline, and both are suitable for a POSIX (or Debian) /bin/sh. I
believe OpenBSD's ksh is also fine, since it is also used as /bin/sh
there and it does support local.
I should point out that Debian has a reasonably restricted set of
requirements[0] on /bin/sh that allow a wide variety of shells to be
used, and which we may want to consider:
* /bin/sh must implement the POSIX.1-2017 shell command language.
* echo -n, if implemented as shell built-in, must not generate a
newline.
* test must support -o and -a.
* local to create a scoped variable must be supported, including
multiple variables and included assignments.
* kill -N where N is an integer must be supported with certain signal
numbers.
* trap must work with certain numeric signals.
The latter two are already required by the XSI extension to POSIX. I
personally don't see the utility in the second or third, and the first
is supported in almost every modern shell anyway (possibly with the
exception of command -v).
Of course, ksh is not included in the set of supported shells for the
two reasons outlined above.
[0] https://www.debian.org/doc/debian-policy/ch-files.html#scripts
--
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread