[PATCH 0/3] GPG running out of pipes fixes

[PATCH 0/3] GPG running out of pipes fixes

[Stephen Boyd]

While running --show-signatures on the linux kernel I noticed that after
a while git failed with an error message indicating it had run out of 
file descriptors. The first patch fixes this problem, and the next
two are randmom bits since I was in the area.

Stephen Boyd (3):
  gpg: Close stderr once finished with it in verify_signed_buffer()
  run-command: Be more informative about what failed
  gpg: Allow translation of more error messages

 gpg-interface.c | 8 +++++---
 run-command.c   | 8 ++++++--
 2 files changed, 11 insertions(+), 5 deletions(-)

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
hosted by The Linux Foundation

[PATCH 1/3] gpg: Close stderr once finished with it in verify_signed_buffer()

[Stephen Boyd]

Failing to close the stderr pipe in verify_signed_buffer() causes
git to run out of file descriptors if there are many calls to
verify_signed_buffer(). An easy way to trigger this is to run

 git log --show-signature --merges | grep "key"

on the linux kernel git repo. Eventually it will fail with

 error: cannot create pipe for gpg: Too many open files
 error: could not run gpg.

Close the stderr pipe so that this can't happen.

Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>
---
 gpg-interface.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/gpg-interface.c b/gpg-interface.c
index 0863c61..2c0bed3 100644
--- a/gpg-interface.c
+++ b/gpg-interface.c
@@ -133,6 +133,8 @@ int verify_signed_buffer(const char *payload, size_t payload_size,
 	if (gpg_output)
 		strbuf_read(gpg_output, gpg.err, 0);
 	ret = finish_command(&gpg);
+	if (gpg_output)
+		close(gpg.err);
 
 	unlink_or_warn(path);
 
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
hosted by The Linux Foundation

Re: [PATCH 1/3] gpg: Close stderr once finished with it in verify_signed_buffer()

[Jeff King]

On Wed, Jan 30, 2013 at 06:01:04PM -0800, Stephen Boyd wrote:

> Failing to close the stderr pipe in verify_signed_buffer() causes
> git to run out of file descriptors if there are many calls to
> verify_signed_buffer(). An easy way to trigger this is to run
> 
>  git log --show-signature --merges | grep "key"
> 
> on the linux kernel git repo. Eventually it will fail with
> 
>  error: cannot create pipe for gpg: Too many open files
>  error: could not run gpg.
> 
> Close the stderr pipe so that this can't happen.

I was able to easily reproduce the bug and verify your fix here.

> diff --git a/gpg-interface.c b/gpg-interface.c
> index 0863c61..2c0bed3 100644
> --- a/gpg-interface.c
> +++ b/gpg-interface.c
> @@ -133,6 +133,8 @@ int verify_signed_buffer(const char *payload, size_t payload_size,
>  	if (gpg_output)
>  		strbuf_read(gpg_output, gpg.err, 0);
>  	ret = finish_command(&gpg);
> +	if (gpg_output)
> +		close(gpg.err);

The strbuf_read above will read to EOF, so it should be equivalent (and
IMHO slightly more readable) to do:

diff --git a/gpg-interface.c b/gpg-interface.c
index 0863c61..5f142f6 100644
--- a/gpg-interface.c
+++ b/gpg-interface.c
@@ -130,8 +130,10 @@ int verify_signed_buffer(const char *payload, size_t payload_size,
 	write_in_full(gpg.in, payload, payload_size);
 	close(gpg.in);
 
-	if (gpg_output)
+	if (gpg_output) {
 		strbuf_read(gpg_output, gpg.err, 0);
+		close(gpg.err);
+	}
 	ret = finish_command(&gpg);
 
 	unlink_or_warn(path);

But that is a minor nit; either way, the patch looks good to me.

-Peff

Re: [PATCH 1/3] gpg: Close stderr once finished with it in verify_signed_buffer()

[Stephen Boyd]

On 01/30/13 21:50, Jeff King wrote:
>
> The strbuf_read above will read to EOF, so it should be equivalent (and
> IMHO slightly more readable) to do:
>
> diff --git a/gpg-interface.c b/gpg-interface.c
> index 0863c61..5f142f6 100644
> --- a/gpg-interface.c
> +++ b/gpg-interface.c
> @@ -130,8 +130,10 @@ int verify_signed_buffer(const char *payload, size_t payload_size,
>  	write_in_full(gpg.in, payload, payload_size);
>  	close(gpg.in);
>  
> -	if (gpg_output)
> +	if (gpg_output) {
>  		strbuf_read(gpg_output, gpg.err, 0);
> +		close(gpg.err);
> +	}
>  	ret = finish_command(&gpg);
>  
>  	unlink_or_warn(path);
>
> But that is a minor nit; either way, the patch looks good to me.

Looks better. I'll resend with this.

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

[PATCHv2 1/3] gpg: Close stderr once finished with it in verify_signed_buffer()

[Stephen Boyd]

Failing to close the stderr pipe in verify_signed_buffer() causes
git to run out of file descriptors if there are many calls to
verify_signed_buffer(). An easy way to trigger this is to run

 git log --show-signature --merges | grep "key"

on the linux kernel git repo. Eventually it will fail with

 error: cannot create pipe for gpg: Too many open files
 error: could not run gpg.

Close the stderr pipe so that this can't happen.

Suggested-by: Jeff King <peff@peff.net>
Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>
---
 gpg-interface.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/gpg-interface.c b/gpg-interface.c
index 0863c61..5f142f6 100644
--- a/gpg-interface.c
+++ b/gpg-interface.c
@@ -130,8 +130,10 @@ int verify_signed_buffer(const char *payload, size_t payload_size,
 	write_in_full(gpg.in, payload, payload_size);
 	close(gpg.in);
 
-	if (gpg_output)
+	if (gpg_output) {
 		strbuf_read(gpg_output, gpg.err, 0);
+		close(gpg.err);
+	}
 	ret = finish_command(&gpg);
 
 	unlink_or_warn(path);
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
hosted by The Linux Foundation

Re: [PATCHv2 1/3] gpg: Close stderr once finished with it in verify_signed_buffer()

[Jeff King]

On Thu, Jan 31, 2013 at 10:18:40AM -0800, Stephen Boyd wrote:

> Failing to close the stderr pipe in verify_signed_buffer() causes
> git to run out of file descriptors if there are many calls to
> verify_signed_buffer(). An easy way to trigger this is to run
> 
>  git log --show-signature --merges | grep "key"
> 
> on the linux kernel git repo. Eventually it will fail with
> 
>  error: cannot create pipe for gpg: Too many open files
>  error: could not run gpg.
> 
> Close the stderr pipe so that this can't happen.
> 
> Suggested-by: Jeff King <peff@peff.net>
> Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>

Thanks, looks good to me (obviously :) ). The rest of the series looks
fine, too, with the caveat I mentioned on 2/3. Thanks for fixing this.

-Peff

[PATCH 2/3] run-command: Be more informative about what failed

[Stephen Boyd]

While debugging an error with verify_signed_buffer() the error
messages from run-command weren't very useful:

 error: cannot create pipe for gpg: Too many open files
 error: could not run gpg.

because they didn't indicate *which* pipe couldn't be created.

Print which pipe failed to be created in the error message so we
can more easily debug similar problems in the future.

For example, the above error now prints:

 error: cannot create stderr pipe for gpg: Too many open files
 error: could not run gpg.

Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>
---
 run-command.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/run-command.c b/run-command.c
index 12d4ddb..016dd05 100644
--- a/run-command.c
+++ b/run-command.c
@@ -274,6 +274,7 @@ int start_command(struct child_process *cmd)
 	int need_in, need_out, need_err;
 	int fdin[2], fdout[2], fderr[2];
 	int failed_errno = failed_errno;
+	char *str;
 
 	/*
 	 * In case of errors we must keep the promise to close FDs
@@ -286,6 +287,7 @@ int start_command(struct child_process *cmd)
 			failed_errno = errno;
 			if (cmd->out > 0)
 				close(cmd->out);
+			str = "stdin";
 			goto fail_pipe;
 		}
 		cmd->in = fdin[1];
@@ -301,6 +303,7 @@ int start_command(struct child_process *cmd)
 				close_pair(fdin);
 			else if (cmd->in)
 				close(cmd->in);
+			str = "stdout";
 			goto fail_pipe;
 		}
 		cmd->out = fdout[0];
@@ -318,9 +321,10 @@ int start_command(struct child_process *cmd)
 				close_pair(fdout);
 			else if (cmd->out)
 				close(cmd->out);
+			str = "stderr";
 fail_pipe:
-			error("cannot create pipe for %s: %s",
-				cmd->argv[0], strerror(failed_errno));
+			error("cannot create %s pipe for %s: %s",
+				str, cmd->argv[0], strerror(failed_errno));
 			errno = failed_errno;
 			return -1;
 		}
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
hosted by The Linux Foundation

Re: [PATCH 2/3] run-command: Be more informative about what failed

[Junio C Hamano]

Stephen Boyd <sboyd@codeaurora.org> writes:

> While debugging an error with verify_signed_buffer() the error
> messages from run-command weren't very useful:
>
>  error: cannot create pipe for gpg: Too many open files
>  error: could not run gpg.
>
> because they didn't indicate *which* pipe couldn't be created.

For the message emitted here with your update (or without for that
matter) to be useful, it has to hold that there is a single leaker,
that leaker fails in this codepath, and that there is nobody else
involved.  Otherwise, you may be able to tell that one caller could
not create its stdin, but the reason it couldn't may be because
somebody else consumed all the available file descriptors.

I am not opposed to this change per-se, but I am not sure that
saying "stdin" etc. makes the message more useful for the purpose of
debugging.

> For example, the above error now prints:
>
>  error: cannot create stderr pipe for gpg: Too many open files
>  error: could not run gpg.

I'd prefer to see these names spelled out (e.g. "standard error")
in any case.

Thanks.

> Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>
> ---

>  run-command.c | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/run-command.c b/run-command.c
> index 12d4ddb..016dd05 100644
> --- a/run-command.c
> +++ b/run-command.c
> @@ -274,6 +274,7 @@ int start_command(struct child_process *cmd)
>  	int need_in, need_out, need_err;
>  	int fdin[2], fdout[2], fderr[2];
>  	int failed_errno = failed_errno;
> +	char *str;
>  
>  	/*
>  	 * In case of errors we must keep the promise to close FDs
> @@ -286,6 +287,7 @@ int start_command(struct child_process *cmd)
>  			failed_errno = errno;
>  			if (cmd->out > 0)
>  				close(cmd->out);
> +			str = "stdin";
>  			goto fail_pipe;
>  		}
>  		cmd->in = fdin[1];
> @@ -301,6 +303,7 @@ int start_command(struct child_process *cmd)
>  				close_pair(fdin);
>  			else if (cmd->in)
>  				close(cmd->in);
> +			str = "stdout";
>  			goto fail_pipe;
>  		}
>  		cmd->out = fdout[0];
> @@ -318,9 +321,10 @@ int start_command(struct child_process *cmd)
>  				close_pair(fdout);
>  			else if (cmd->out)
>  				close(cmd->out);
> +			str = "stderr";
>  fail_pipe:
> -			error("cannot create pipe for %s: %s",
> -				cmd->argv[0], strerror(failed_errno));
> +			error("cannot create %s pipe for %s: %s",
> +				str, cmd->argv[0], strerror(failed_errno));
>  			errno = failed_errno;
>  			return -1;
>  		}

Re: [PATCH 2/3] run-command: Be more informative about what failed

[Stephen Boyd]

On 01/31/13 08:24, Junio C Hamano wrote:
> Stephen Boyd <sboyd@codeaurora.org> writes:
>
>> While debugging an error with verify_signed_buffer() the error
>> messages from run-command weren't very useful:
>>
>>  error: cannot create pipe for gpg: Too many open files
>>  error: could not run gpg.
>>
>> because they didn't indicate *which* pipe couldn't be created.
> For the message emitted here with your update (or without for that
> matter) to be useful, it has to hold that there is a single leaker,
> that leaker fails in this codepath, and that there is nobody else
> involved.  Otherwise, you may be able to tell that one caller could
> not create its stdin, but the reason it couldn't may be because
> somebody else consumed all the available file descriptors.
>
> I am not opposed to this change per-se, but I am not sure that
> saying "stdin" etc. makes the message more useful for the purpose of
> debugging.

It helped me avoid firing up gdb, but if you don't see much use feel
free to ignore this patch.

>
>> For example, the above error now prints:
>>
>>  error: cannot create stderr pipe for gpg: Too many open files
>>  error: could not run gpg.
> I'd prefer to see these names spelled out (e.g. "standard error")
> in any case.

Sure, I can do that.

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

Re: [PATCH 2/3] run-command: Be more informative about what failed

[Jeff King]

On Thu, Jan 31, 2013 at 08:24:21AM -0800, Junio C Hamano wrote:

> Stephen Boyd <sboyd@codeaurora.org> writes:
> 
> > While debugging an error with verify_signed_buffer() the error
> > messages from run-command weren't very useful:
> >
> >  error: cannot create pipe for gpg: Too many open files
> >  error: could not run gpg.
> >
> > because they didn't indicate *which* pipe couldn't be created.
> 
> For the message emitted here with your update (or without for that
> matter) to be useful, it has to hold that there is a single leaker,
> that leaker fails in this codepath, and that there is nobody else
> involved.  Otherwise, you may be able to tell that one caller could
> not create its stdin, but the reason it couldn't may be because
> somebody else consumed all the available file descriptors.
> 
> I am not opposed to this change per-se, but I am not sure that
> saying "stdin" etc. makes the message more useful for the purpose of
> debugging.

Yeah, I had the same feeling. All that failed is pipe(), which does not
have anything to do with what we are going to use the pipe for. So it
gives some context, perhaps, but does not necessarily tell us anything
useful.

But it is not much code, and sometimes it is surprising what information
can be helpful when debugging, so like you, I am not opposed, just
doubtful.

-Peff

Re: [PATCH 2/3] run-command: Be more informative about what failed

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> But it is not much code, and sometimes it is surprising what information
> can be helpful when debugging, so like you, I am not opposed, just
> doubtful.

Yes, exactly my feeling.

Perhaps I should just amend the 'stdin' and friends away without
asking Stephen to reroll.  In the other two I did not see any
issues.  I've queued all three of them including this one but as
separate topics.

Thanks.

[PATCH 3/3] gpg: Allow translation of more error messages

[Stephen Boyd]

Mark these strings for translation so that error messages are
printed in the user's language of choice.

Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>
---
 gpg-interface.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/gpg-interface.c b/gpg-interface.c
index 2c0bed3..474c037 100644
--- a/gpg-interface.c
+++ b/gpg-interface.c
@@ -109,10 +109,10 @@ int verify_signed_buffer(const char *payload, size_t payload_size,
 	args_gpg[0] = gpg_program;
 	fd = git_mkstemp(path, PATH_MAX, ".git_vtag_tmpXXXXXX");
 	if (fd < 0)
-		return error("could not create temporary file '%s': %s",
+		return error(_("could not create temporary file '%s': %s"),
 			     path, strerror(errno));
 	if (write_in_full(fd, signature, signature_size) < 0)
-		return error("failed writing detached signature to '%s': %s",
+		return error(_("failed writing detached signature to '%s': %s"),
 			     path, strerror(errno));
 	close(fd);
 
@@ -124,7 +124,7 @@ int verify_signed_buffer(const char *payload, size_t payload_size,
 	args_gpg[2] = path;
 	if (start_command(&gpg)) {
 		unlink(path);
-		return error("could not run gpg.");
+		return error(_("could not run gpg."));
 	}
 
 	write_in_full(gpg.in, payload, payload_size);
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
hosted by The Linux Foundation

Re: [PATCH 3/3] gpg: Allow translation of more error messages

[Jonathan Nieder]

Stephen Boyd wrote:

> Mark these strings for translation so that error messages are
> printed in the user's language of choice.

Yeah.  Other error messages in this file are already translated, so
it's oddly inconsistent.

Assuming this passes tests with GETTEXT_POISON=YesPlease,
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>