(resend) [PATCH] Don't ignore write failure from git-diff, git-log, etc.

(resend) [PATCH] Don't ignore write failure from git-diff, git-log, etc.

[Jim Meyering]

Hi Jun,

Here's a copy of my patch, from here:
http://thread.gmane.org/gmane.comp.version-control.git/48469/focus=48636

Jim

-----------------------------
From 42e3a6f676e9ae4e9640bc2ff36b7ab0b061a60e Mon Sep 17 00:00:00 2001
From: Jim Meyering <jim@meyering.net>
Date: Sat, 26 May 2007 13:43:07 +0200
Subject: [PATCH] Don't ignore write failure from git-diff, git-log, etc.

Currently, when git-diff writes to a full device or gets an I/O error,
it fails to detect the write error:

    $ git-diff |wc -c
    3984
    $ git-diff > /dev/full && echo ignored write failure
    ignored write failure

git-log does the same thing:

    $ git-log -n1 > /dev/full && echo ignored write failure
    ignored write failure

Each and every git command should report such a failure.
Some already do, but with the patch below, they all do, and we
won't have to rely on code in each command's implementation to
perform the right incantation.

    $ ./git-log -n1 > /dev/full
    fatal: write failure on standard output: No space left on device
    [Exit 128]
    $ ./git-diff > /dev/full
    fatal: write failure on standard output: No space left on device
    [Exit 128]

You can demonstrate this with git's own --version output, too:
(but git --help detects the failure without this patch)

    $ ./git --version > /dev/full
    fatal: write failure on standard output: No space left on device
    [Exit 128]

Note that the fcntl test (for whether the fileno may be closed) is
required in order to avoid EBADF upon closing an already-closed stdout,
as would happen for each git command that already closes stdout; I think
update-index was the one I noticed in the failure of t5400, before I
added that test.

Also, to be consistent with e.g., write_or_die, do not
diagnose EPIPE write failures.

Signed-off-by: Jim Meyering <jim@meyering.net>
---
 git.c |   19 ++++++++++++++++++-
 1 files changed, 18 insertions(+), 1 deletions(-)

diff --git a/git.c b/git.c
index 29b55a1..8258885 100644
--- a/git.c
+++ b/git.c
@@ -308,6 +308,7 @@ static void handle_internal_command(int argc, const char **argv, char **envp)
 	for (i = 0; i < ARRAY_SIZE(commands); i++) {
 		struct cmd_struct *p = commands+i;
 		const char *prefix;
+		int status;
 		if (strcmp(p->cmd, cmd))
 			continue;

@@ -321,7 +322,23 @@ static void handle_internal_command(int argc, const char **argv, char **envp)
 			die("%s must be run in a work tree", cmd);
 		trace_argv_printf(argv, argc, "trace: built-in: git");

-		exit(p->fn(argc, argv, prefix));
+		status = p->fn(argc, argv, prefix);
+
+		/* Close stdout if necessary, and diagnose any failure
+		   other than EPIPE.  */
+		if (fcntl(fileno (stdout), F_GETFD) >= 0) {
+			errno = 0;
+			if ((ferror(stdout) || fclose(stdout))
+			    && errno != EPIPE) {
+				if (errno == 0)
+					die("write failure on standard output");
+				else
+					die("write failure on standard output"
+					    ": %s", strerror(errno));
+			}
+		}
+
+		exit(status);
 	}
 }

--
1.5.2.73.g18bece

Re: (resend) [PATCH] Don't ignore write failure from git-diff, git-log, etc.

[Junio C Hamano]

Jim Meyering <jim@meyering.net> writes:

> From: Jim Meyering <jim@meyering.net>
> Date: Sat, 26 May 2007 13:43:07 +0200
> Subject: [PATCH] Don't ignore write failure from git-diff, git-log, etc.
>
> Currently, when git-diff writes to a full device or gets an I/O error,
> it fails to detect the write error:
> ...
> Also, to be consistent with e.g., write_or_die, do not
> diagnose EPIPE write failures.

I still do not like the fact that this patch makes an error from
the final stdout flushing override the return value from p->fn()
even when the function already diagnosed an error, but otherwise
I think it is a good change, as it allows us to catch one error
case that we currently don't, without introducing an annoying
EPIPE diagnosis.

Naks, or vetoes?

Re: (resend) [PATCH] Don't ignore write failure from git-diff, git-log, etc.

[Linus Torvalds]

On Sun, 24 Jun 2007, Junio C Hamano wrote:
> 
> I still do not like the fact that this patch makes an error from
> the final stdout flushing override the return value from p->fn()
> even when the function already diagnosed an error

Yeah.

I also don't think it's very _pretty_ code, and it violates my personal 
coding standards by adding way too deep indentation for the new error 
cases. It was already three indents deep (reasonably fine, but that 
NOT_BARE test wass already pretty ugly), but now it becomes five 
indentation levels deep at its deepest, which is just a sign that things 
should be split up.

I'd also like to know why it does that fcntl() is done, and I also wonder 
about that "ferror()" call: it is entirely possible that ferror() is set 
due to EPIPE, and in that case, it will *not* set errno to EPIPE at all, 
so it will *still* complain about what I consider an invalid situation.

I dunno. I think the ENOSPC worry is a very real and valid one, but I 
would really tend prefer something different.

How about this following series of two patches instead, which I'll send as 
replies to this email..

		Linus

[PATCH 1/2] Clean up internal command handling

[Linus Torvalds]

This should change no code at all, it just moves the definition of "struct 
cmd_struct" out, and then splits out the running of the right command into 
the "run_command()" function.

It also removes the long-unused 'envp' pointer passing.

This is just preparation for adding some more error checking.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
---

 git.c |   52 ++++++++++++++++++++++++++++++----------------------
 1 files changed, 30 insertions(+), 22 deletions(-)

diff --git a/git.c b/git.c
index 29b55a1..6c728e4 100644
--- a/git.c
+++ b/git.c
@@ -216,14 +216,34 @@ const char git_version_string[] = GIT_VERSION;
  */
 #define NOT_BARE 	(1<<2)
 
-static void handle_internal_command(int argc, const char **argv, char **envp)
+struct cmd_struct {
+	const char *cmd;
+	int (*fn)(int, const char **, const char *);
+	int option;
+};
+
+static NORETURN void run_command(struct cmd_struct *p, int argc, const char **argv)
+{
+	const char *prefix;
+
+	prefix = NULL;
+	if (p->option & RUN_SETUP)
+		prefix = setup_git_directory();
+	if (p->option & USE_PAGER)
+		setup_pager();
+	if (p->option & NOT_BARE) {
+		if (is_bare_repository() || is_inside_git_dir())
+			die("%s must be run in a work tree", p->cmd);
+	}
+	trace_argv_printf(argv, argc, "trace: built-in: git");
+
+	exit(p->fn(argc, argv, prefix));
+}
+
+static void handle_internal_command(int argc, const char **argv)
 {
 	const char *cmd = argv[0];
-	static struct cmd_struct {
-		const char *cmd;
-		int (*fn)(int, const char **, const char *);
-		int option;
-	} commands[] = {
+	static struct cmd_struct commands[] = {
 		{ "add", cmd_add, RUN_SETUP | NOT_BARE },
 		{ "annotate", cmd_annotate, RUN_SETUP | USE_PAGER },
 		{ "apply", cmd_apply },
@@ -307,25 +327,13 @@ static void handle_internal_command(int argc, const char **argv, char **envp)
 
 	for (i = 0; i < ARRAY_SIZE(commands); i++) {
 		struct cmd_struct *p = commands+i;
-		const char *prefix;
 		if (strcmp(p->cmd, cmd))
 			continue;
-
-		prefix = NULL;
-		if (p->option & RUN_SETUP)
-			prefix = setup_git_directory();
-		if (p->option & USE_PAGER)
-			setup_pager();
-		if ((p->option & NOT_BARE) &&
-				(is_bare_repository() || is_inside_git_dir()))
-			die("%s must be run in a work tree", cmd);
-		trace_argv_printf(argv, argc, "trace: built-in: git");
-
-		exit(p->fn(argc, argv, prefix));
+		run_command(p, argc, argv);
 	}
 }
 
-int main(int argc, const char **argv, char **envp)
+int main(int argc, const char **argv)
 {
 	const char *cmd = argv[0] ? argv[0] : "git-help";
 	char *slash = strrchr(cmd, '/');
@@ -358,7 +366,7 @@ int main(int argc, const char **argv, char **envp)
 	if (!prefixcmp(cmd, "git-")) {
 		cmd += 4;
 		argv[0] = cmd;
-		handle_internal_command(argc, argv, envp);
+		handle_internal_command(argc, argv);
 		die("cannot handle %s internally", cmd);
 	}
 
@@ -390,7 +398,7 @@ int main(int argc, const char **argv, char **envp)
 
 	while (1) {
 		/* See if it's an internal command */
-		handle_internal_command(argc, argv, envp);
+		handle_internal_command(argc, argv);
 
 		/* .. then try the external ones */
 		execv_git_cmd(argv);

[PATCH 2/2] Check for IO errors after running a command

[Linus Torvalds]

This is trying to implement the strict IO error checks that Jim Meyering 
suggested, but explicitly limits it to just regular files. If a pipe gets 
closed on us, we shouldn't complain about it.

[ Side note: feel free to change the S_ISREG() to a !S_ISFIFO(). That 
  would allow easier testing with /dev/full, which tends to be a character 
  special device. That said, I think sockets and pipes are generally 
  interchangeable, which is why I limited it to just regular files. ]

If the subcommand already returned an error, that takes precedence (and we 
assume that the subcommand already printed out any relevant messages 
relating to it)

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
---

Hmm? I'm not saying this is the only way to do this, but I think this is 
at least likely to be obviously just an improvement, and it leaves room 
for further tweaking of the logic if Jim or others find other cases that 
should be handled.

Side note: I think I made a mistake in making the run_command() a NORETURN 
function and putting the exit() into it. It's probably better to instead 
just make it return "int", and make the caller do

	exit(run_command(...));

and that makes it much prettier to have "run_command()" just return early 
if an error happens (or doesn't happen).

For example, then we could just do

	status = p->fn(...);
	if (status)
		return status;
	/* Somebody closed stdout? */
	if (fstat(fileno(stdout), &st))
		return 0;
	/* Ignore write errors for pipes and sockets.. */
	if (S_ISFIFO(st.st_mode) || S_ISSOCK(st.st_mode))
		return 0;

which makes it easy to explain what's going on, and avoids having any deep 
indentation at all.

I dunno. This passes all the tests, but it's not like we currently test 
for ENOSPC/EIO anyway, or even can do that. If we _just_ disable the thing 
for pipes/sockets, we could add a test using /dev/full.

I did check that changing it to !S_ISFIFO() gets the right behaviour, and 
"git log | head" doesn't complain, while "git log > /dev/full" does. So 
this has gotten some very rudimentary testing, but not in the exact form 
I'm actually sending it out.

 git.c |   15 ++++++++++++++-
 1 files changed, 14 insertions(+), 1 deletions(-)

diff --git a/git.c b/git.c
index 6c728e4..db0118a 100644
--- a/git.c
+++ b/git.c
@@ -224,6 +224,8 @@ struct cmd_struct {
 
 static NORETURN void run_command(struct cmd_struct *p, int argc, const char **argv)
 {
+	int status;
+	struct stat st;
 	const char *prefix;
 
 	prefix = NULL;
@@ -237,7 +239,18 @@ static NORETURN void run_command(struct cmd_struct *p, int argc, const char **ar
 	}
 	trace_argv_printf(argv, argc, "trace: built-in: git");
 
-	exit(p->fn(argc, argv, prefix));
+	status = p->fn(argc, argv, prefix);
+	if (status)
+		exit(status);
+
+	/* Check for ENOSPC and EIO errors.. */
+	if (!fstat(fileno(stdout), &st) && S_ISREG(st.st_mode)) {
+		if (ferror(stdout))
+			die("write failure on standard output");
+		if (fflush(stdout) || fclose(stdout))
+			die("write failure on standard output: %s", strerror(errno));
+	}
+	exit(0);
 }
 
 static void handle_internal_command(int argc, const char **argv)

Re: (resend) [PATCH] Don't ignore write failure from git-diff, git-log, etc.

[Jim Meyering]

Linus Torvalds <torvalds@linux-foundation.org> wrote:
...
> I also don't think it's very _pretty_ code, and it violates my personal
> coding standards by adding way too deep indentation for the new error
> cases. It was already three indents deep (reasonably fine, but that
> NOT_BARE test wass already pretty ugly), but now it becomes five
> indentation levels deep at its deepest, which is just a sign that things
> should be split up.

I too disliked the form of my patch, and said so.

> I'd also like to know why it does that fcntl() is done,

In the quoted message, I explained that stdout was already
closed in some cases.  The fcntl test avoids the EINVAL you'd
get for closing an already-closed file descriptor.

Re: [PATCH 2/2] Check for IO errors after running a command

[Junio C Hamano]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> Side note: I think I made a mistake in making the run_command() a NORETURN 
> function and putting the exit() into it. It's probably better to instead 
> just make it return "int", and make the caller do
>
> 	exit(run_command(...));
>
> and that makes it much prettier to have "run_command()" just return early 
> if an error happens (or doesn't happen).
>
> For example, then we could just do
>
> 	status = p->fn(...);
> 	if (status)
> 		return status;
> 	/* Somebody closed stdout? */
> 	if (fstat(fileno(stdout), &st))
> 		return 0;
> 	/* Ignore write errors for pipes and sockets.. */
> 	if (S_ISFIFO(st.st_mode) || S_ISSOCK(st.st_mode))
> 		return 0;
>
> which makes it easy to explain what's going on, and avoids having any deep 
> indentation at all.

I took the liberty of munging your two patches to follow your
comments above (it was a perfect guinea-pig opportunity for
Johannes's "rebase -i").

The changes to git.c (run_command) conflicted with GIT_WORK_TREE
changes in a minor way.  Matthias, could you sanity check the
result once I push it out to 'next', please?

Re: [PATCH 2/2] Check for IO errors after running a command

[Johannes Schindelin]

Hi Junio,

On Mon, 25 Jun 2007, Junio C Hamano wrote:

> I took the liberty of munging your two patches to follow your comments 
> above (it was a perfect guinea-pig opportunity for Johannes's "rebase 
> -i").

And? How was that experience?

I am actually quite pleased how it worked out in the end; I never 
understood the syntax of rebase, and stayed away from it for that reason. 
By reworking patch-series into rebase -i, I learnt it on the way, and find 
it actually quite useful.

One thing we might consider, however: when rebasing, the current branch 
gets updated at each step. Some might consider this a bug, and prefer 
rebase to work on a detached HEAD, and only update the branch at the end, 
so that <branchname>@{1} refers to the state _before_ rebase.

Thoughts?

Ciao,
Dscho

Re: [PATCH 2/2] Check for IO errors after running a command

[Johannes Sixt]

Johannes Schindelin wrote:
> One thing we might consider, however: when rebasing, the current branch
> gets updated at each step. Some might consider this a bug, and prefer
> rebase to work on a detached HEAD, and only update the branch at the end,
> so that <branchname>@{1} refers to the state _before_ rebase.

YESSSS!

Finding the commit before the rebase in the reflog is a nightmare.

-- Hannes

Re: [PATCH 2/2] Check for IO errors after running a command

[Jim Meyering]

Junio C Hamano <gitster@pobox.com> wrote:
> Linus Torvalds <torvalds@linux-foundation.org> writes:
...
>> For example, then we could just do
>>
>> 	status = p->fn(...);
>> 	if (status)
>> 		return status;
>> 	/* Somebody closed stdout? */
>> 	if (fstat(fileno(stdout), &st))
>> 		return 0;
>> 	/* Ignore write errors for pipes and sockets.. */
>> 	if (S_ISFIFO(st.st_mode) || S_ISSOCK(st.st_mode))
>> 		return 0;
>>
>> which makes it easy to explain what's going on, and avoids having any deep
>> indentation at all.
>
> I took the liberty of munging your two patches to follow your
> comments above

That has the disadvantage of ignoring *all* pipe and socket write errors.
IMHO, git would be better served if it didn't do that, since writing to
those can fail with EIO and even a new one: EACCES (though this latter
is only for sockets).  Also possible, according to POSIX: ENOBUFS.

Of course, one can probably argue that those are all unlikely.
They may be even less likely than an actual EPIPE, but the point is
that people and tools using git plumbing should be able to rely on
it to report such write failures, no matter how unusual they are.

Re: [PATCH 2/2] Check for IO errors after running a command

[Linus Torvalds]

On Mon, 25 Jun 2007, Jim Meyering wrote:
> 
> That has the disadvantage of ignoring *all* pipe and socket write errors.

.. which is the only wayt to do it.

There's *no*way* to tell what the error was for the case of

	if (ferror(stdout))
		..

because the original errno has long long since been thrown away.

> Of course, one can probably argue that those are all unlikely.
> They may be even less likely than an actual EPIPE, but the point is
> that people and tools using git plumbing should be able to rely on
> it to report such write failures, no matter how unusual they are.

..but since what you suggest is physically impossible in a half-way 
portable manner, and since all relevant such errors are likely to have 
happened long before, I would suggest:

 - Use my patch

OR

 - Stop using stdio. 

   You *cannot* make stdio error handling sane. It's simply not possible. 
   The whole point of stdio is to simplify things, but it does so to the 
   point where portable and reliable error handling is not an option any 
   more.

   Replace all command output that you care about with some stdio 
   replacement. We do that for all the paths we *really* care about, where 
   we use raw unistd IO and do our own buffering (ie things like the 
   "write_in_full()" stuff and "write_sha1_file()" etc.

Anybody who thinks he can handle errors with stdio is simply barking up 
the wrogn tree. It can be done, but it can be done only by essentially 
making stdio be a really awkward way to do IO, and you're better off with 
the raw unistd.h interfaces.

In particular, you need to make sure you check *each* return value of 
every single stdio operation. Even then, you don't actually know if 
"errno" is set correctly if an error happens in the middle (ie most stdio 
routines are just defined to return EOF or nonzero on error, and it's not 
at all clear that errno is reliable).

If you don't, a buffer flush error may have happened at any time, and 
whatever error code it had has been thrown away, and turned into one 
single bit (the "ferror()" thing).

And yes, some libc's might have extensions that actually guarantee more. 
But even then I would be *very* surprised if they actually work and have 
been tested to any real degree (glibc does not. The stream error is a 
single bit. I checked)

So really: if you care about a particular read or write, use the 
"write_in_full()" and "read_in_full()" functions. NOTHING ELSE!

			Linus

Re: [PATCH 2/2] Check for IO errors after running a command

[Jim Meyering]

Linus Torvalds <torvalds@linux-foundation.org> wrote:
> This is trying to implement the strict IO error checks that Jim Meyering
> suggested, but explicitly limits it to just regular files. If a pipe gets
> closed on us, we shouldn't complain about it.
...
> Hmm? I'm not saying this is the only way to do this, but I think this is
> at least likely to be obviously just an improvement, and it leaves room
> for further tweaking of the logic if Jim or others find other cases that
> should be handled.
...
> +	/* Check for ENOSPC and EIO errors.. */
> +	if (!fstat(fileno(stdout), &st) && S_ISREG(st.st_mode)) {
> +		if (ferror(stdout))
> +			die("write failure on standard output");
> +		if (fflush(stdout) || fclose(stdout))
> +			die("write failure on standard output: %s", strerror(errno));
> +	}
> +	exit(0);

Here is a patch relative to "next", to restore some of the
functionality that was provided by my initially-proposed change.

>From 379aee16596d29b83c95068964c349399b9b9f47 Mon Sep 17 00:00:00 2001
From: Jim Meyering <jim@meyering.net>
Date: Mon, 25 Jun 2007 18:54:12 +0200
Subject: [PATCH] When detecting write failure, print strerror when possible.

Do not call "die" solely on the basis of ferror.
Instead, call both ferror and fclose, and *then* decide whether/how to die.

Without this patch, some commands unnecessarily omit strerror(errno)
when they fail:

    $ ./git-ls-tree HEAD > /dev/full
    fatal: write failure on standard output

With the patch, git reports the desired ENOSPC diagnostic:

    fatal: write failure on standard output: No space left on device

FWIW, this version of close_stream is similar to the one
I included in another recent patch, but, is slightly cleaner.
Also, rather than returning zero or EOF, this one simply returns
zero or nonzero.

* git.c (close_stream): New function.
(run_command): Don't die solely because of ferror.  Use close_stream.

Signed-off-by: Jim Meyering <jim@meyering.net>
---
 git.c |   26 ++++++++++++++++++++++----
 1 files changed, 22 insertions(+), 4 deletions(-)

diff --git a/git.c b/git.c
index b949cbb..b00bb1c 100644
--- a/git.c
+++ b/git.c
@@ -246,6 +246,21 @@ struct cmd_struct {
 	int option;
 };

+static int
+close_stream(FILE *stream)
+{
+	int prev_fail = ferror(stream);
+	int fclose_fail = fclose(stream);
+
+	/* If there was a previous failure, but fclose succeeded,
+	   clear errno, since ferror does not set it, and its value
+	   may be unrelated to the ferror-reported failure.  */
+	if (prev_fail && !fclose_fail)
+		errno = 0;
+
+	return prev_fail || fclose_fail;
+}
+
 static int run_command(struct cmd_struct *p, int argc, const char **argv)
 {
 	int status;
@@ -274,10 +289,13 @@ static int run_command(struct cmd_struct *p, int argc, const char **argv)
 		return 0;

 	/* Check for ENOSPC and EIO errors.. */
-	if (ferror(stdout))
-		die("write failure on standard output");
-	if (fflush(stdout) || fclose(stdout))
-		die("write failure on standard output: %s", strerror(errno));
+	if (close_stream(stdout)) {
+		if (errno == 0)
+			die("write failure on standard output");
+		else
+			die("write failure on standard output: %s",
+			    strerror(errno));
+	}

 	return 0;
 }

Re: [PATCH 2/2] Check for IO errors after running a command

[Matthias Lederhofer]

Junio C Hamano <gitster@pobox.com> wrote:
> The changes to git.c (run_command) conflicted with GIT_WORK_TREE
> changes in a minor way.  Matthias, could you sanity check the
> result once I push it out to 'next', please?

The changes look fine, the tests pass and my own short manual test
passed.