[Junio C Hamano] Re: Consolidate SHA1 object file close

[Junio C Hamano] Re: Consolidate SHA1 object file close

[Junio C Hamano]

[-- Attachment #1: Type: text/plain, Size: 75 bytes --]

Somehow the thread went off-list, so I'm diverting it back to the list...


[-- Attachment #2: Type: message/rfc822, Size: 4037 bytes --]

From: Junio C Hamano <gitster@pobox.com>
To: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Pierre Habouzit <madcoder@debian.org>
Subject: Re: Consolidate SHA1 object file close
Date: Sat, 14 Jun 2008 15:28:27 -0700
Message-ID: <7vmylnfy7o.fsf@gitster.siamese.dyndns.org>

Linus Torvalds <torvalds@linux-foundation.org> writes:

> On Sat, 14 Jun 2008, Junio C Hamano wrote:
>> 
>> I've queued this...
>> 
>> commit 6483925999cde47e5108414ac3f57760394ee2d2
>> Author: Junio C Hamano <gitster@pobox.com>
>> Date:   Fri Jun 13 23:00:51 2008 -0700
>> 
>>     sha1_file.c: dead code removal
>
> Ok, the following three emails will contain three patches on top of that 
> commit that I think are ready for inclusion. They haven't seen a ton of 
> testing, and they are obviously to very core functionality, so in that 
> sense they are a big "scary".

Other than that I had to stop for a few seconds to think at the magic "20
bytes longer", they look scary-but-correct ;-).

I've tagged 1.5.6-rc3 with this (and other fixups), but it appears the
network between me and k.org is crappy today so I'll stop hacking for the
rest of the afternoon and spend time on something else.  Will push the
results out sometime tonight.

Thanks.

Re: [Junio C Hamano] Re: Consolidate SHA1 object file close

[Linus Torvalds]

On Sat, 14 Jun 2008, Junio C Hamano wrote:
>
> Other than that I had to stop for a few seconds to think at the magic 
> "20 bytes longer", they look scary-but-correct ;-).

Actually, thinking about it, the +20 was not just fairly arbitrary and 
would have been better off with a comment, it was also not a very good 
number to begin with - it just was the old tempfile pattern rounded up to 
the next "even" number.

The _correct_ number to use is +39, with a better tempfile pattern.

That needs a few comments, though. So I added those too.

This patch is totally unimportant, but it does mean that a really 
traditional filesystem can now do the final rename in-place, because the 
temporary file is not just in the same directory as the final one, but ti 
also has the same length, so old-style filesystems can literally just edit 
the name in place and mark the buffer dirty.

And more modern filesystems will generally end up doing more complex 
things regardless of what we do, but at least this won't _hurt_ either.

There is one more funny (or sad) detail to this: the old temporary 
filename was an interesting size: 14 characters. That happens to be the 
traditional path component limit on really old UNIX filesystems, so if you 
were to try to use such a filesystem, you'd always be able to create the 
temporary file, but then you'd never actually be able to do the final link 
or rename due to the final name being too long.

So take this or leave it as you want - it's really not very important, I 
just wrote this because I was thinking about what really goes on at a very 
low level when we do that final atomic rename to create the actual 
directory entry. This makes it theoretically just a tiny bit more atomic 
on old-fashioned filesystems.

		Linus

---
 sha1_file.c |   21 ++++++++++++++++++---
 1 files changed, 18 insertions(+), 3 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 37bcc54..f9ec069 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -2110,17 +2110,31 @@ static inline int directory_size(const char *filename)
  *
  * We want to avoid cross-directory filename renames, because those
  * can have problems on various filesystems (FAT, NFS, Coda).
+ *
+ * In order to make the final rename as simple as possible to do
+ * in-place for traditional filesystems, we make the temporary
+ * filename have the same size as the final one - 38 characters
+ * (19 bytes of SHA1 info - the first byte is used for the sub-
+ * directory fan-out).
+ *
+ * The temporary filename pattern is
+ *  - 8 characters of "tmp_obj_"
+ *  - 23 characters of the object filename
+ *  - 7 characters of "_XXXXXX" for mkstemp().
  */
 static int create_tmpfile(char *buffer, size_t bufsiz, const char *filename)
 {
 	int fd, dirlen = directory_size(filename);
 
-	if (dirlen + 20 > bufsiz) {
+	if (dirlen + 39 > bufsiz) {
 		errno = ENAMETOOLONG;
 		return -1;
 	}
 	memcpy(buffer, filename, dirlen);
-	strcpy(buffer + dirlen, "tmp_obj_XXXXXX");
+	memcpy(buffer + dirlen, "tmp_obj_", 8);
+	memcpy(buffer + dirlen + 8, filename + dirlen, 23);
+	memcpy(buffer + dirlen + 31, "_XXXXXX", 8);
+	
 	fd = mkstemp(buffer);
 	if (fd < 0 && dirlen) {
 		/* Make sure the directory exists */
@@ -2129,7 +2143,8 @@ static int create_tmpfile(char *buffer, size_t bufsiz, const char *filename)
 			return -1;
 
 		/* Try again */
-		strcpy(buffer + dirlen - 1, "/tmp_obj_XXXXXX");
+		buffer[dirlen-1] = '/';
+		memcpy(buffer + dirlen + 32, "XXXXXX", 6);
 		fd = mkstemp(buffer);
 	}
 	return fd;

Re: [Junio C Hamano] Re: Consolidate SHA1 object file close

[Nicolas Pitre]

On Sun, 15 Jun 2008, Linus Torvalds wrote:

> 
> 
> On Sat, 14 Jun 2008, Junio C Hamano wrote:
> >
> > Other than that I had to stop for a few seconds to think at the magic 
> > "20 bytes longer", they look scary-but-correct ;-).

BTW, isn't the same concerns valid for temporary pack file creation too?
Or maybe we just don't care in the context of packs?

Also... this change is lacking its counterpart in 
builtin-prune.c:remove_temporary_files().


Nicolas

Re: [Junio C Hamano] Re: Consolidate SHA1 object file close

[Junio C Hamano]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> Actually, thinking about it, the +20 was not just fairly arbitrary and 
> would have been better off with a comment, it was also not a very good 
> number to begin with - it just was the old tempfile pattern rounded up to 
> the next "even" number.
>
> The _correct_ number to use is +39, with a better tempfile pattern.
>
> That needs a few comments, though. So I added those too.
>
> This patch is totally unimportant, but it does mean that a really 
> traditional filesystem can now do the final rename in-place, because the 
> temporary file is not just in the same directory as the final one, but ti 
> also has the same length, so old-style filesystems can literally just edit 
> the name in place and mark the buffer dirty.

That's interesting and somewhat amusing ;-)

> So take this or leave it as you want - it's really not very important, I 
> just wrote this because I was thinking about what really goes on at a very 
> low level when we do that final atomic rename to create the actual 
> directory entry. This makes it theoretically just a tiny bit more atomic 
> on old-fashioned filesystems.

However, this may be more important fix.  We want to make sure that
adjust_shared_perm() is called on the success codepath, especially not
when mkdir() does _not_ fail.

diff --git a/sha1_file.c b/sha1_file.c
index 500584b..e300562 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -2125,7 +2125,7 @@ static int create_tmpfile(char *buffer, size_t bufsiz, const char *filename)
 	if (fd < 0 && dirlen) {
 		/* Make sure the directory exists */
 		buffer[dirlen-1] = 0;
-		if (mkdir(buffer, 0777) && adjust_shared_perm(buffer))
+		if (mkdir(buffer, 0777) || adjust_shared_perm(buffer))
 			return -1;
 
 		/* Try again */

Re: [Junio C Hamano] Re: Consolidate SHA1 object file close

[Linus Torvalds]

On Mon, 16 Jun 2008, Junio C Hamano wrote:
> 
> However, this may be more important fix.  We want to make sure that
> adjust_shared_perm() is called on the success codepath, especially not
> when mkdir() does _not_ fail.

Oops. Yes.

The old code this came from actually did it differently: it failed if 
mkdir _succeeded_ but then adjust_shared_perm() failed. If the mkdir 
failed, it would just try again. That was what confused me - it was a 
copy-and-paste thing, but with me misreading the source of it and being 
confused.

IOW, the old code looked like

	if (!mkdir(filename, 0777) && adjust_shared_perm(filename)) {
		.. failure path ..

which is just really odd. A failing mkdir would be a "success" from this 
standpoint, leading to another "link()" being attempted. I think it may 
have wanted to return the errno from the link() or something.

Your fix is obviously superior. And equally obviously we don't seem to 
have any tests for this case. Maybe because I have a umask of 0002, so the 
mkdir() would already make it group-writable?

		Linus

Re: [Junio C Hamano] Re: Consolidate SHA1 object file close

[Pierre Habouzit]

[-- Attachment #1: Type: text/plain, Size: 550 bytes --]

On Sat, Jun 14, 2008 at 10:42:33PM +0000, Junio C Hamano wrote:
> Somehow the thread went off-list, so I'm diverting it back to the list...
> 

  FWIW, the person with that patch here do not have the issue with NFS
at all, whereas people with older git versions have had the issue. IOW,
I consider my issue with NFS fixed by this patch.

  Thanks a lot guys \o/

-- 
·O·  Pierre Habouzit
··O                                                madcoder@debian.org
OOO                                                http://www.madism.org

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]