From mboxrd@z Thu Jan 1 00:00:00 1970 From: Brodie Rao Subject: [PATCH] gc: support temporarily preserving garbage Date: Thu, 13 Nov 2014 17:16:45 -0800 Message-ID: <1415927805-53644-1-git-send-email-brodie@sf.io> Cc: Bryan Turner To: git@vger.kernel.org X-From: git-owner@vger.kernel.org Fri Nov 14 02:16:57 2014 Return-path: Envelope-to: gcvg-git-2@plane.gmane.org Received: from vger.kernel.org ([209.132.180.67]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1Xp5VQ-0003P2-Li for gcvg-git-2@plane.gmane.org; Fri, 14 Nov 2014 02:16:57 +0100 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S965081AbaKNBQx (ORCPT ); Thu, 13 Nov 2014 20:16:53 -0500 Received: from mail-pa0-f44.google.com ([209.85.220.44]:33898 "EHLO mail-pa0-f44.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S965061AbaKNBQt (ORCPT ); Thu, 13 Nov 2014 20:16:49 -0500 Received: by mail-pa0-f44.google.com with SMTP id et14so1630950pad.17 for ; Thu, 13 Nov 2014 17:16:49 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=from:to:cc:subject:date:message-id; bh=ffSWqfJdcrQsViUnC47Lmb7ETCmB9PAa18oaDsGt86I=; b=Jb7+2VRmsAzoCEW+EMtpT16nCD3nYXMiz7piqSqNdBOG/gdF4HKhJLAtMYIS/3ocrZ XPGzV55yfYphp8pjxX/ee9G22cd+rZUldNBdIC7OnG5gpoNnIld6dL9dmuaE8932W3N6 YBTPtWRlYQHMvFPSfqtRFhef0c9dEsVR0Lqjwy0x7PhmpsgASjiEuZPB4LC7AS1ePhps oO4tLoTjX6ApJ/I537FEUr55EuBf5e0Jf2TfRNiSGCmmp3murNJexwG3giaE6QcuXyTN CA3QIEhTKts6yOL8q6O1E8ZqPjZDNTLag7bmM1wPMC3XF1RIf0bOxxJFL38kwZ+0+EV/ 4IVA== X-Received: by 10.70.118.1 with SMTP id ki1mr6589507pdb.69.1415927809129; Thu, 13 Nov 2014 17:16:49 -0800 (PST) Received: from localhost (70-35-42-138.static.wiline.com. [70.35.42.138]) by mx.google.com with ESMTPSA id kk1sm25927143pbd.14.2014.11.13.17.16.48 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Nov 2014 17:16:48 -0800 (PST) X-Mailer: git-send-email 2.1.3 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org This patch adds a gc.garbageexpire setting that, when not set to "now", makes gc (and prune, prune-packed, and repack) move garbage into a temporary garbage directory instead of deleting it immediately. The garbage directory is then cleared out based on gc.garbageexpire. The motivation for this setting is to work around various NFS servers not supporting delete-on-last-close semantics between NFS clients. Without proper support for that, gc could potentially delete objects and packs that are in use by git processes on other NFS clients. If another git process has a deleted pack file mmap()ed, it could crash with a SIGBUS error on Linux. Signed-off-by: Brodie Rao --- .gitignore | 1 + Documentation/config.txt | 20 +++++++++ Documentation/git-gc.txt | 7 ++++ Documentation/git-prune-garbage.txt | 55 ++++++++++++++++++++++++ Documentation/git-prune-packed.txt | 9 ++++ Documentation/git-prune.txt | 9 ++++ Documentation/git-repack.txt | 6 +++ Documentation/git.txt | 6 +++ Makefile | 2 + builtin.h | 1 + builtin/gc.c | 20 +++++++++ builtin/prune-garbage.c | 77 ++++++++++++++++++++++++++++++++++ builtin/prune-packed.c | 3 +- builtin/prune.c | 5 ++- builtin/repack.c | 7 ++-- cache.h | 2 + command-list.txt | 1 + contrib/completion/git-completion.bash | 2 + environment.c | 12 +++++- gc.c | 60 ++++++++++++++++++++++++++ gc.h | 16 +++++++ git.c | 1 + t/t6502-gc-garbage-expire.sh | 60 ++++++++++++++++++++++++++ 23 files changed, 375 insertions(+), 7 deletions(-) create mode 100644 Documentation/git-prune-garbage.txt create mode 100644 builtin/prune-garbage.c create mode 100644 gc.c create mode 100644 gc.h create mode 100755 t/t6502-gc-garbage-expire.sh diff --git a/.gitignore b/.gitignore index a052419..a9a4e30 100644 --- a/.gitignore +++ b/.gitignore @@ -107,6 +107,7 @@ /git-parse-remote /git-patch-id /git-prune +/git-prune-garbage /git-prune-packed /git-pull /git-push diff --git a/Documentation/config.txt b/Documentation/config.txt index 9220725..0106d8f 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -1213,6 +1213,26 @@ gc.autodetach:: Make `git gc --auto` return immediately and run in background if the system supports it. Default is true. +gc.garbageexpire:: + When 'git gc' is run, objects and packs that are pruned are + immediately deleted from the file system. This setting can be + overridden to move pruned objects and packs to the garbage + directory. That leftover garbage will be deleted after the + specified grace period. The default value is "now", meaning + garbage is deleted immediately. ++ +Setting this to something other than "now" (e.g., "1.day.ago") can help +work around issues with NFS servers that don't support +delete-on-last-close semantics between NFS clients. 'git gc' will not +unlink files immediately, so a git process on another NFS client that +might be reading a garbage collected file will not crash. ++ +Note that this setting can cause the repository's size to increase as +garbage collection passes are made. Care should be taken to make sure +the grace period isn't too long. A grace period of one day might be +reasonable if you make the assumption that your git processes over NFS +won't run longer or have files open longer than one day. + gc.packrefs:: Running `git pack-refs` in a repository renders it unclonable by Git versions prior to 1.5.1.2 over dumb diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt index 273c466..f90dc0a 100644 --- a/Documentation/git-gc.txt +++ b/Documentation/git-gc.txt @@ -131,6 +131,12 @@ The optional configuration variable 'gc.pruneExpire' controls how old the unreferenced loose objects have to be before they are pruned. The default is "2 weeks ago". +The optional configurable variable 'gc.garbageexpire' controls how +pruned objects and packs are deleted. This can be overridden to move +pruned objects and packs to the garbage directory. That leftover garbage +will be deleted after the specified grace period. The default value is +"now", meaning garbage is deleted immediately. + Notes ----- @@ -156,6 +162,7 @@ linkgit:githooks[5] for more information. SEE ALSO -------- linkgit:git-prune[1] +linkgit:git-prune-garbage[1] linkgit:git-reflog[1] linkgit:git-repack[1] linkgit:git-rerere[1] diff --git a/Documentation/git-prune-garbage.txt b/Documentation/git-prune-garbage.txt new file mode 100644 index 0000000..ff130e0 --- /dev/null +++ b/Documentation/git-prune-garbage.txt @@ -0,0 +1,55 @@ +git-prune-garbage(1) +==================== + +NAME +---- +git-prune-garbage - Remove garbage objects and pack files + + +SYNOPSIS +-------- +[verse] +'git prune-garbage' [-n] [-v] [--expire ] + + +DESCRIPTION +----------- +This command deletes objects and pack files in `$GIT_GARBAGE_DIRECTORY` +that are older than the specified garbage expiration time. This is +automatically run by linkgit:git-gc[1]. + +The garbage directory is populated by linkgit:git-gc[1] and the commands +it runs (linkgit:git-prune[1], linkgit:git-prune-packed[1], and +linkgit:git-repack[1]). + +OPTIONS +------- +-n:: +--dry-run:: + Do not remove anything; just report what it would remove. + +-v:: +--verbose:: + Report all removed objects and packs. + +--expire