[RFC PATCH 0/1] invalid outdated temporary files

[RFC PATCH 0/1] invalid outdated temporary files

[Han Xin]

From: Han Xin <hanxin.hx@alibaba-inc.com>

When I use git on network storage, there may be some temporary files left in
git repositories due to the program quitting unexpectedly, and they also
prevent me from continuing to use the git command line until I manually
delete them.

For some obviously expired temporary files, manual deletion is an extra
expensive job, and one that I think can be avoided. So I wrote this patch.
When creating a temporary file with the same name, if the mtime of the target
file exceeds one day, unlink it first, and then try to recreate it.

Han Xin (1):
  tempfile: invalid outdated temporary files

 Documentation/config/core.txt |  6 ++++++
 cache.h                       |  1 +
 config.c                      |  5 +++++
 environment.c                 |  1 +
 t/t3210-pack-refs.sh          | 10 ++++++++++
 tempfile.c                    | 21 +++++++++++++++++++++
 6 files changed, 44 insertions(+)

-- 
2.35.1.55.gde4365fc52.agit.6.5.7

[RFC PATCH 1/1] tempfile: invalid outdated temporary files

[Han Xin]

From: Han Xin <hanxin.hx@alibaba-inc.com>

When git quits unexpectedly, some temporary files(e.g. HEAD.lock,
packed-refs.lock, packed-refs.new) may remain in the repository. These
files will prevent us from performing the corresponding operations
again, even if they were created a long time ago before, until we
manually remove them.

In order for git to automatically fix this situation, let's add a config
named "core.tempfileExpire". When an attempt is made to create a
temporary file that exists and is older than this config value, the file
will be unlinked and recreated.

Signed-off-by: Han Xin <hanxin.hx@alibaba-inc.com>
---
 Documentation/config/core.txt |  6 ++++++
 cache.h                       |  1 +
 config.c                      |  5 +++++
 environment.c                 |  1 +
 t/t3210-pack-refs.sh          | 10 ++++++++++
 tempfile.c                    | 21 +++++++++++++++++++++
 6 files changed, 44 insertions(+)

diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index c04f62a54a..9907c9e6f3 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -424,6 +424,12 @@ be delta compressed, but larger binary media files won't be.
 +
 Common unit suffixes of 'k', 'm', or 'g' are supported.
 
+core.tempfileExpire::
+	When an attempt is made to create a temporary file that exists
+	and is older than this config value, the file will be unlinked
+	and recreated. This feature is used to ignore temporary files
+	(e.g. *.lock and *.new) remaining due to abnormal exits.
+
 core.excludesFile::
 	Specifies the pathname to the file that contains patterns to
 	describe paths that are not meant to be tracked, in addition
diff --git a/cache.h b/cache.h
index 0bc0a37cec..4264fe7960 100644
--- a/cache.h
+++ b/cache.h
@@ -972,6 +972,7 @@ extern size_t packed_git_window_size;
 extern size_t packed_git_limit;
 extern size_t delta_base_cache_limit;
 extern unsigned long big_file_threshold;
+extern const char *tempfile_expire;
 extern unsigned long pack_size_limit_cfg;
 
 /*
diff --git a/config.c b/config.c
index e78397725c..36a5bc1960 100644
--- a/config.c
+++ b/config.c
@@ -1518,6 +1518,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
 		return 0;
 	}
 
+	if (!strcmp(var, "core.tempfileexpire")) {
+		git_config_get_expiry(var, &tempfile_expire);
+		return 0;
+	}
+
 	if (!strcmp(var, "core.packedgitlimit")) {
 		packed_git_limit = git_config_ulong(var, value);
 		return 0;
diff --git a/environment.c b/environment.c
index fb55bf6129..20b928e71b 100644
--- a/environment.c
+++ b/environment.c
@@ -48,6 +48,7 @@ size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
 size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
 size_t delta_base_cache_limit = 96 * 1024 * 1024;
 unsigned long big_file_threshold = 512 * 1024 * 1024;
+const char *tempfile_expire = "1.day.ago";
 int pager_use_color = 1;
 const char *editor_program;
 const char *askpass_program;
diff --git a/t/t3210-pack-refs.sh b/t/t3210-pack-refs.sh
index 577f32dc71..7d443c1cfd 100755
--- a/t/t3210-pack-refs.sh
+++ b/t/t3210-pack-refs.sh
@@ -231,6 +231,16 @@ test_expect_success 'timeout if packed-refs.lock exists' '
 	test_must_fail git pack-refs --all --prune
 '
 
+test_expect_success 'success if packed-refs.lock expires' '
+	LOCK=.git/packed-refs.lock &&
+	>"$LOCK" &&
+	test_when_finished "rm -f $LOCK" &&
+	test-tool chmtime -86000 $LOCK &&
+	test_must_fail git pack-refs --all --prune &&
+	test-tool chmtime -400 $LOCK &&
+	git pack-refs --all --prune
+'
+
 test_expect_success 'retry acquiring packed-refs.lock' '
 	LOCK=.git/packed-refs.lock &&
 	>"$LOCK" &&
diff --git a/tempfile.c b/tempfile.c
index 94aa18f3f7..f18e0121d0 100644
--- a/tempfile.c
+++ b/tempfile.c
@@ -51,6 +51,7 @@
  */
 
 #include "cache.h"
+#include "date.h"
 #include "tempfile.h"
 #include "sigchain.h"
 
@@ -137,6 +138,26 @@ struct tempfile *create_tempfile_mode(const char *path, int mode)
 	strbuf_add_absolute_path(&tempfile->filename, path);
 	tempfile->fd = open(tempfile->filename.buf,
 			    O_RDWR | O_CREAT | O_EXCL | O_CLOEXEC, mode);
+	if (tempfile->fd < 0 && errno == EEXIST) {
+		struct stat st;
+		int err = errno;
+		if (lstat(tempfile->filename.buf, &st) < 0) {
+			warning_errno(_("failed to stat %s"),
+				      tempfile->filename.buf);
+			errno = err;
+		} else if (st.st_mtime <= approxidate(tempfile_expire)) {
+			/*
+			 * If the file is older than core.tmpfileExpire, it is
+			 * probably outdated.  We'll try to remove it and try
+			 * again.
+			 */
+			warning(_("remove outdated tempfile %s"), tempfile->filename.buf);
+			unlink_or_warn(tempfile->filename.buf);
+			tempfile->fd = open(tempfile->filename.buf,
+					    O_RDWR | O_CREAT | O_EXCL | O_CLOEXEC,
+					    mode);
+		}
+	}
 	if (O_CLOEXEC && tempfile->fd < 0 && errno == EINVAL)
 		/* Try again w/o O_CLOEXEC: the kernel might not support it */
 		tempfile->fd = open(tempfile->filename.buf,
-- 
2.35.1.55.gde4365fc52.agit.6.5.7

Re: [RFC PATCH 1/1] tempfile: invalid outdated temporary files

[Junio C Hamano]

Han Xin <chiyutianyi@gmail.com> writes:

> From: Han Xin <hanxin.hx@alibaba-inc.com>
>
> When git quits unexpectedly, some temporary files(e.g. HEAD.lock,
> packed-refs.lock, packed-refs.new) may remain in the repository. These
> files will prevent us from performing the corresponding operations
> again, even if they were created a long time ago before, until we
> manually remove them.
>
> In order for git to automatically fix this situation, let's add a config

I am not sure if we want to automatically "fix" this situation in the
first place.

Unless the user is sure of the reason why these lock files are left
behind (e.g. "ah, I saw a power failure last night while I was
trying to commit"), we are looking at a potential data loss.