fatal from submodule status --recursive when used with grep -q

fatal from submodule status --recursive when used with grep -q

[Matt Liberty]

Hi

I have a repo with submodule, like:

% git submodule status --recursive
+834b7a0fb6de03c8e584256af2ecb7889b568343 tools/OpenROAD (v2.0-15587-g834b7a0fb)
 aafee90f8a21bf7867cef2e159929440cf45b2e5 tools/OpenROAD/src/sta
(v2.2.0-1106-gaafee90)
 ef5389d31526003c2ebd7e6d6d6fe3848a20f0a2
tools/OpenROAD/third-party/abc (remotes/origin/HEAD)
 80ba43d26264738c93900129dc0aab7fab36c53f tools/yosys (yosys-0.44)
 28d955ca97a1c4be3aed4062aec0241a734fac5d tools/yosys/abc (yosys-0.44)

If I try to grep the output I get a fatal error:

% git submodule status --recursive | grep -q "^+"
fatal: failed to recurse into submodule 'tools/OpenROAD'

I didn't expect any output but did want the return status (0).  I'm
guessing git is unhappy that grep -q exits on the first occurrence of
the pattern.  I don't feel fatal is appropriate here.

I can work around this by writing to a tmpfile but would like to see
this addressed.

Thanks!
Matt

Re: fatal from submodule status --recursive when used with grep -q

[Phillip Wood]

Hi Matt

On 16/09/2024 16:08, Matt Liberty wrote:
> 
> If I try to grep the output I get a fatal error:
> 
> % git submodule status --recursive | grep -q "^+"
> fatal: failed to recurse into submodule 'tools/OpenROAD'
> 
> I didn't expect any output but did want the return status (0).  I'm
> guessing git is unhappy that grep -q exits on the first occurrence of
> the pattern.  I don't feel fatal is appropriate here.

I assume git is dying with SIGPIPE. As the only purpose of "git 
submodule status" is to write the status information to stdout that 
sounds reasonable. If you want to collect the exit status you need to 
consume the whole output so that the command runs to completion.

Best Wishes

Phillip

> I can work around this by writing to a tmpfile but would like to see
> this addressed.
> 
> Thanks!
> Matt
> 

Re: fatal from submodule status --recursive when used with grep -q

[Matt Liberty]

I'm not trying to capture the exit status of git but of grep.  Compare
it to "yes|grep -q y" which doesn't generate any error even though yes
generates endless output.

Thanks

Matt


On Wed, Sep 18, 2024 at 3:05 AM Phillip Wood <phillip.wood123@gmail.com> wrote:
>
> Hi Matt
>
> On 16/09/2024 16:08, Matt Liberty wrote:
> >
> > If I try to grep the output I get a fatal error:
> >
> > % git submodule status --recursive | grep -q "^+"
> > fatal: failed to recurse into submodule 'tools/OpenROAD'
> >
> > I didn't expect any output but did want the return status (0).  I'm
> > guessing git is unhappy that grep -q exits on the first occurrence of
> > the pattern.  I don't feel fatal is appropriate here.
>
> I assume git is dying with SIGPIPE. As the only purpose of "git
> submodule status" is to write the status information to stdout that
> sounds reasonable. If you want to collect the exit status you need to
> consume the whole output so that the command runs to completion.
>
> Best Wishes
>
> Phillip
>
> > I can work around this by writing to a tmpfile but would like to see
> > this addressed.
> >
> > Thanks!
> > Matt
> >

Re: fatal from submodule status --recursive when used with grep -q

[Junio C Hamano]

Phillip Wood <phillip.wood123@gmail.com> writes:

> On 16/09/2024 16:08, Matt Liberty wrote:
>> If I try to grep the output I get a fatal error:
>> % git submodule status --recursive | grep -q "^+"
>> fatal: failed to recurse into submodule 'tools/OpenROAD'
>> I didn't expect any output but did want the return status (0).  I'm
>> guessing git is unhappy that grep -q exits on the first occurrence of
>> the pattern.  I don't feel fatal is appropriate here.
>
> I assume git is dying with SIGPIPE. As the only purpose of "git
> submodule status" is to write the status information to stdout that
> sounds reasonable. If you want to collect the exit status you need to
> consume the whole output so that the command runs to completion.

Yes, but isn't the main point of that complaint that "failed to
recurse into" is not a good way to say "because you closed the
reading end of the pipe, we are aborting the operation and not
decending into 'tools/OpenROAD' submodule", is it?

Re: fatal from submodule status --recursive when used with grep -q

[Matt Liberty]

I'm saying most unix tools do not error when the pipe is closed and I
gave 'yes' as a simple example but there are many others.  grep found
what I was looking for and I don't care that git didn't recurse
further (in fact I'm happy that it didn't waste time doing so).  There
is no problem except that git insists on complaining about the closed
pipe.

Matt


On Wed, Sep 18, 2024 at 6:16 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Phillip Wood <phillip.wood123@gmail.com> writes:
>
> > On 16/09/2024 16:08, Matt Liberty wrote:
> >> If I try to grep the output I get a fatal error:
> >> % git submodule status --recursive | grep -q "^+"
> >> fatal: failed to recurse into submodule 'tools/OpenROAD'
> >> I didn't expect any output but did want the return status (0).  I'm
> >> guessing git is unhappy that grep -q exits on the first occurrence of
> >> the pattern.  I don't feel fatal is appropriate here.
> >
> > I assume git is dying with SIGPIPE. As the only purpose of "git
> > submodule status" is to write the status information to stdout that
> > sounds reasonable. If you want to collect the exit status you need to
> > consume the whole output so that the command runs to completion.
>
> Yes, but isn't the main point of that complaint that "failed to
> recurse into" is not a good way to say "because you closed the
> reading end of the pipe, we are aborting the operation and not
> decending into 'tools/OpenROAD' submodule", is it?
>

Re: fatal from submodule status --recursive when used with grep -q

[phillip.wood123]

On 18/09/2024 15:28, Matt Liberty wrote:
> I'm not trying to capture the exit status of git but of grep.  Compare 
> it to "yes|grep -q y" which doesn't generate any error even though yes 
> generates endless output.

Sorry, I'd misunderstood your original message. So the problem is that 
you want "git submodule status" to be quiet (i.e. not print an error 
message) when it dies of SIGPIPE?

Best Wishes

Phillip

> Matt
> 
> On Wed, Sep 18, 2024 at 3:05 AM Phillip Wood <phillip.wood123@gmail.com 
> <mailto:phillip.wood123@gmail.com>> wrote:
> 
>     Hi Matt
> 
>     On 16/09/2024 16:08, Matt Liberty wrote:
>      >
>      > If I try to grep the output I get a fatal error:
>      >
>      > % git submodule status --recursive | grep -q "^+"
>      > fatal: failed to recurse into submodule 'tools/OpenROAD'
>      >
>      > I didn't expect any output but did want the return status (0).  I'm
>      > guessing git is unhappy that grep -q exits on the first occurrence of
>      > the pattern.  I don't feel fatal is appropriate here.
> 
>     I assume git is dying with SIGPIPE. As the only purpose of "git
>     submodule status" is to write the status information to stdout that
>     sounds reasonable. If you want to collect the exit status you need to
>     consume the whole output so that the command runs to completion.
> 
>     Best Wishes
> 
>     PhillipSo
> 
>      > I can work around this by writing to a tmpfile but would like to see
>      > this addressed.
>      >
>      > Thanks!
>      > Matt
>      >
> 

Re: fatal from submodule status --recursive when used with grep -q

[Matt Liberty]

Yes and also to have a 0 exit code.  No error occurred from my perspective.

Matt


On Thu, Sep 19, 2024 at 7:22 AM <phillip.wood123@gmail.com> wrote:
>
> On 18/09/2024 15:28, Matt Liberty wrote:
> > I'm not trying to capture the exit status of git but of grep.  Compare
> > it to "yes|grep -q y" which doesn't generate any error even though yes
> > generates endless output.
>
> Sorry, I'd misunderstood your original message. So the problem is that
> you want "git submodule status" to be quiet (i.e. not print an error
> message) when it dies of SIGPIPE?
>
> Best Wishes
>
> Phillip
>
> > Matt
> >
> > On Wed, Sep 18, 2024 at 3:05 AM Phillip Wood <phillip.wood123@gmail.com
> > <mailto:phillip.wood123@gmail.com>> wrote:
> >
> >     Hi Matt
> >
> >     On 16/09/2024 16:08, Matt Liberty wrote:
> >      >
> >      > If I try to grep the output I get a fatal error:
> >      >
> >      > % git submodule status --recursive | grep -q "^+"
> >      > fatal: failed to recurse into submodule 'tools/OpenROAD'
> >      >
> >      > I didn't expect any output but did want the return status (0).  I'm
> >      > guessing git is unhappy that grep -q exits on the first occurrence of
> >      > the pattern.  I don't feel fatal is appropriate here.
> >
> >     I assume git is dying with SIGPIPE. As the only purpose of "git
> >     submodule status" is to write the status information to stdout that
> >     sounds reasonable. If you want to collect the exit status you need to
> >     consume the whole output so that the command runs to completion.
> >
> >     Best Wishes
> >
> >     PhillipSo
> >
> >      > I can work around this by writing to a tmpfile but would like to see
> >      > this addressed.
> >      >
> >      > Thanks!
> >      > Matt
> >      >
> >

Re: fatal from submodule status --recursive when used with grep -q

[Junio C Hamano]

Matt Liberty <mliberty@precisioninno.com> writes:

[administrivia: you are not supposed to top-post on this list]

> On Thu, Sep 19, 2024 at 7:22 AM <phillip.wood123@gmail.com> wrote:
>>
>> On 18/09/2024 15:28, Matt Liberty wrote:
>> > I'm not trying to capture the exit status of git but of grep.  Compare
>> > it to "yes|grep -q y" which doesn't generate any error even though yes
>> > generates endless output.
>>
>> Sorry, I'd misunderstood your original message. So the problem is that
>> you want "git submodule status" to be quiet (i.e. not print an error
>> message) when it dies of SIGPIPE?
>
> Yes and also to have a 0 exit code.  No error occurred from my perspective.

I am sympathetic to the line of reasoning, but I wouldn't go that
far.

Running "yes | less" and killing it with 'q' saying "ok I've seen
enough", and running "yes" and killing it with '^C' saying "ok I've
seen enough" are pretty much the same from the end-user perspective.
You told the command to go away, and the command complied.  But I
haven't seen anybody complaining if they see a "Killed" message or
"echo $?" immediately after it says 130 for the latter.

Re: fatal from submodule status --recursive when used with grep -q

[Eric Sunshine]

On Thu, Sep 19, 2024 at 3:10 PM Junio C Hamano <gitster@pobox.com> wrote:
> Matt Liberty <mliberty@precisioninno.com> writes:
> > Yes and also to have a 0 exit code.  No error occurred from my perspective.
>
> I am sympathetic to the line of reasoning, but I wouldn't go that
> far.
>
> Running "yes | less" and killing it with 'q' saying "ok I've seen
> enough", and running "yes" and killing it with '^C' saying "ok I've
> seen enough" are pretty much the same from the end-user perspective.
> You told the command to go away, and the command complied.  But I
> haven't seen anybody complaining if they see a "Killed" message or
> "echo $?" immediately after it says 130 for the latter.

Regarding the exit code, it's not even clear how that factors into
this discussion considering that, in the presented example,
git-submodule is upstream of a pipe, thus its exit code is lost
anyhow.

Re: fatal from submodule status --recursive when used with grep -q

[Matt Liberty]

The exit code matters if you do "set -euo pipefail".

Matt


On Thu, Sep 19, 2024 at 12:40 PM Eric Sunshine <sunshine@sunshineco.com> wrote:
>
> On Thu, Sep 19, 2024 at 3:10 PM Junio C Hamano <gitster@pobox.com> wrote:
> > Matt Liberty <mliberty@precisioninno.com> writes:
> > > Yes and also to have a 0 exit code.  No error occurred from my perspective.
> >
> > I am sympathetic to the line of reasoning, but I wouldn't go that
> > far.
> >
> > Running "yes | less" and killing it with 'q' saying "ok I've seen
> > enough", and running "yes" and killing it with '^C' saying "ok I've
> > seen enough" are pretty much the same from the end-user perspective.
> > You told the command to go away, and the command complied.  But I
> > haven't seen anybody complaining if they see a "Killed" message or
> > "echo $?" immediately after it says 130 for the latter.
>
> Regarding the exit code, it's not even clear how that factors into
> this discussion considering that, in the presented example,
> git-submodule is upstream of a pipe, thus its exit code is lost
> anyhow.

Re: fatal from submodule status --recursive when used with grep -q

[phillip.wood123]

Hi Matt

[please reply in-line to avoid breaking the flow of the conversation]

On 19/09/2024 21:10, Matt Liberty wrote:
> On Thu, Sep 19, 2024 at 12:40 PM Eric Sunshine <sunshine@sunshineco.com> wrote:
>>
>> Regarding the exit code, it's not even clear how that factors into
>> this discussion considering that, in the presented example,
>> git-submodule is upstream of a pipe, thus its exit code is lost
>> anyhow.
 >
 > The exit code matters if you do "set -euo pipefail".

When you use "grep -q" you know that if it is successful the upstream 
commands in the pipeline are likely to receive SIGPIPE. Therefore if you 
want to use "-o pipefail" to check that they do not fail unexpectedly 
you need to remap the exit code associated with SIGPIPE to zero. 
Something like [1]

	hide_sigpipe() {
		status=$?
		test $status -eq 141 && exit 0
		exit $status
	}
	(yes; hide_sigpipe) | grep -q y


I've posted a patch[2] that fixes the error message you were seeing.

Best Wishes

Phillip

[1] 
https://unix.stackexchange.com/questions/582844/how-to-suppress-sigpipe-in-bash
[2] 
https://lore.kernel.org/git/pull.1799.git.1726837642511.gitgitgadget@gmail.com