[PATCH] Teach fsck and prune about the new location of temporary objects

[PATCH] Teach fsck and prune about the new location of temporary objects

[Brandon Casey]

Since 5723fe7e, temporary objects are now created in their final destination
directories, rather than in .git/objects/. Teach fsck to recognize and
ignore the temporary objects it encounters, and teach prune to remove them.

Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil>
---
 builtin-fsck.c  |    4 ++++
 builtin-prune.c |   41 ++++++++++++++++++++++++-----------------
 2 files changed, 28 insertions(+), 17 deletions(-)

diff --git a/builtin-fsck.c b/builtin-fsck.c
index 7326dc3..7a4a4f1 100644
--- a/builtin-fsck.c
+++ b/builtin-fsck.c
@@ -377,6 +377,10 @@ static void fsck_dir(int i, char *path)
 			if (de->d_name[0] != '.')
 				break;
 			continue;
+		case 14:
+			if (prefixcmp(de->d_name, "tmp_obj_"))
+				break;
+			continue;
 		case 38:
 			sprintf(name, "%02x", i);
 			memcpy(name+2, de->d_name, len+1);
diff --git a/builtin-prune.c b/builtin-prune.c
index 7de4cab..947de8c 100644
--- a/builtin-prune.c
+++ b/builtin-prune.c
@@ -13,6 +13,22 @@ static const char * const prune_usage[] = {
 static int show_only;
 static unsigned long expire;
 
+static int prune_tmp_object(char *path, const char *filename)
+{
+	const char *fullpath = mkpath("%s/%s", path, filename);
+	if (expire) {
+		struct stat st;
+		if (lstat(fullpath, &st))
+			return error("Could not stat '%s'", fullpath);
+		if (st.st_mtime > expire)
+			return 0;
+	}
+	printf("Removing stale temporary file %s\n", fullpath);
+	if (!show_only)
+		unlink(fullpath);
+	return 0;
+}
+
 static int prune_object(char *path, const char *filename, const unsigned char *sha1)
 {
 	const char *fullpath = mkpath("%s/%s", path, filename);
@@ -53,6 +69,11 @@ static int prune_dir(int i, char *path)
 			if (de->d_name[0] != '.')
 				break;
 			continue;
+		case 14:
+			if (prefixcmp(de->d_name, "tmp_obj_"))
+				break;
+			prune_tmp_object(path, de->d_name);
+			continue;
 		case 38:
 			sprintf(name, "%02x", i);
 			memcpy(name+2, de->d_name, len+1);
@@ -105,23 +126,9 @@ static void remove_temporary_files(void)
 			dirname);
 		return;
 	}
-	while ((de = readdir(dir)) != NULL) {
-		if (!prefixcmp(de->d_name, "tmp_")) {
-			char name[PATH_MAX];
-			int c = snprintf(name, PATH_MAX, "%s/%s",
-					 dirname, de->d_name);
-			if (c < 0 || c >= PATH_MAX)
-				continue;
-			if (expire) {
-				struct stat st;
-				if (stat(name, &st) != 0 || st.st_mtime >= expire)
-					continue;
-			}
-			printf("Removing stale temporary file %s\n", name);
-			if (!show_only)
-				unlink(name);
-		}
-	}
+	while ((de = readdir(dir)) != NULL)
+		if (!prefixcmp(de->d_name, "tmp_"))
+			prune_tmp_object(dirname, de->d_name);
 	closedir(dir);
 }
 
-- 
1.6.0.rc0.55.g160ae.dirty

Re: [PATCH] Teach fsck and prune about the new location of temporary objects

[Johannes Schindelin]

Hi,

On Thu, 24 Jul 2008, Brandon Casey wrote:

> Since 5723fe7e, temporary objects are now created in their final destination
> directories, rather than in .git/objects/. Teach fsck to recognize and
> ignore the temporary objects it encounters,

It somehow feels wrong for fsck to ignore anything that could be used.

Ciao,
Dscho

Re: [PATCH] Teach fsck and prune about the new location of temporary objects

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> On Thu, 24 Jul 2008, Brandon Casey wrote:
>
>> Since 5723fe7e, temporary objects are now created in their final destination
>> directories, rather than in .git/objects/. Teach fsck to recognize and
>> ignore the temporary objects it encounters,
>
> It somehow feels wrong for fsck to ignore anything that could be used.

Back when we created temporaries in .git/objects/, these tmp_obj files
were not even checked nor looked at, because fsck_object_dir() checked
only those 256 fan-out directories, so this is nothing new.

And these tmp_obj files are something we _expect_ to be in the object
directory.  The user can ^C out object creation codepath anytime, and we
do not reference the final destination objects from refs nor the index to
preserve the integrity of the repository when that happens.  In other
words, they are crufts, but they are not something unusual nor get alarmed
about.

And it also is the right thing to do to remove them in prune.

[PATCH] fsck: Don't require tmp_obj_ file names are 14 bytes in length

[Shawn O. Pearce]

Not all temporary file creation routines will ensure 14 bytes are
used to generate the temporary file name.  In C Git this may be
true, but alternate implementations such as jgit are not always
able to generate a temporary file name with a specific prefix and
also ensure the file name length is 14 bytes long.

Since temporary files in a directory we are fsck'ing should be
uncommon (as they are short lived only long enough for an active
writer to finish writing the file and rename it) we shouldn't see
these show up very often.  Always using a prefixcmp() call and
ignoring the length opens up room for other implementations to use
different name generation schemes.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---

  Brandon Casey <casey@nrlssc.navy.mil> wrote:
  > Since 5723fe7e, temporary objects are now created in their final destination
  > directories, rather than in .git/objects/. Teach fsck to recognize and
  > ignore the temporary objects it encounters, and teach prune to remove them.

  jgit can't exactly follow the 14 character naming rule.
  This should be a safe way around it.

 builtin-fsck.c |    6 ++----
 1 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/builtin-fsck.c b/builtin-fsck.c
index 7a4a4f1..6eb7da8 100644
--- a/builtin-fsck.c
+++ b/builtin-fsck.c
@@ -377,10 +377,6 @@ static void fsck_dir(int i, char *path)
 			if (de->d_name[0] != '.')
 				break;
 			continue;
-		case 14:
-			if (prefixcmp(de->d_name, "tmp_obj_"))
-				break;
-			continue;
 		case 38:
 			sprintf(name, "%02x", i);
 			memcpy(name+2, de->d_name, len+1);
@@ -389,6 +385,8 @@ static void fsck_dir(int i, char *path)
 			add_sha1_list(sha1, DIRENT_SORT_HINT(de));
 			continue;
 		}
+		if (prefixcmp(de->d_name, "tmp_obj_"))
+			continue;
 		fprintf(stderr, "bad sha1 file: %s/%s\n", path, de->d_name);
 	}
 	closedir(dir);
-- 
1.6.0.rc0.182.gb96c7

-- 
Shawn.

Re: [PATCH] fsck: Don't require tmp_obj_ file names are 14 bytes in length

[Junio C Hamano]

"Shawn O. Pearce" <spearce@spearce.org> writes:

>   Brandon Casey <casey@nrlssc.navy.mil> wrote:
>   > Since 5723fe7e, temporary objects are now created in their final destination
>   > directories, rather than in .git/objects/. Teach fsck to recognize and
>   > ignore the temporary objects it encounters, and teach prune to remove them.
>
>   jgit can't exactly follow the 14 character naming rule.
>   This should be a safe way around it.

More importantly, we might want to change the temporary filename to ensure
that it is 38 bytes in length in the future (cf. $gmane/85106).