From: Avery Pennarun <apenwarr@gmail.com>
To: git@vger.kernel.org, Eric Wong <normalperson@yhbt.net>,
Adam Roben <aroben@apple.com>, Samuel Bronson <naesten@gmail.com>,
B.Steinbrink@gmx.de, gitster@pobox.com
Cc: Avery Pennarun <apenwarr@gmail.com>
Subject: [PATCH v2] git-svn: avoid filling up the disk with temp files.
Date: Sat, 28 Jun 2008 19:33:56 -0400 [thread overview]
Message-ID: <1214696036-8294-1-git-send-email-apenwarr@gmail.com> (raw)
In-Reply-To: <1214686673-28099-1-git-send-email-apenwarr@gmail.com>
Commit ffe256f9bac8a40ff751a9341a5869d98f72c285 ("git-svn: Speed up fetch")
introduced changes that create a temporary file for each object fetched by
svn. These files should be deleted automatically, but perl apparently
doesn't do this until the process exits (or perhaps when its garbage
collector runs).
This means that on a large fetch, especially with lots of branches, we
sometimes fill up /tmp completely, which prevents the next temp file from
being written completely. This is aggravated by the fact that a new temp
file is created for each updated file, even if that update produces a file
identical to one already in git. Thus, it can happen even if there's lots
of disk space to store the finished repository.
We weren't adequately checking for write errors, so this would result in an
invalid file getting committed, which caused git-svn to fail later with an
invalid checksum.
This patch adds a check to syswrite() so similar problems don't lead to
corruption in the future. It also unlink()'s each temp file explicitly
when we're done with it, so the disk doesn't need to fill up.
Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
---
Please use this in favour of the "Revert "git-svn: Speed up fetch" I sent
earlier. I ended up having a surprise inspiration that led to a real fix :)
git-svn.perl | 5 ++++-
1 files changed, 4 insertions(+), 1 deletions(-)
diff --git a/git-svn.perl b/git-svn.perl
index 263d66c..0011387 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -3243,7 +3243,9 @@ sub close_file {
my ($tmp_fh, $tmp_filename) = File::Temp::tempfile(UNLINK => 1);
my $result;
while ($result = sysread($fh, my $string, 1024)) {
- syswrite($tmp_fh, $string, $result);
+ my $wrote = syswrite($tmp_fh, $string, $result);
+ defined($wrote) && $wrote == $result
+ or croak("write $tmp_filename: $!\n");
}
defined $result or croak $!;
close $tmp_fh or croak $!;
@@ -3251,6 +3253,7 @@ sub close_file {
close $fh or croak $!;
$hash = $::_repository->hash_and_insert_object($tmp_filename);
+ unlink($tmp_filename);
$hash =~ /^[a-f\d]{40}$/ or die "not a sha1: $hash\n";
close $fb->{base} or croak $!;
} else {
--
1.5.4.3
next prev parent reply other threads:[~2008-06-28 23:35 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-06-28 19:48 git-svn messed up import, badly Björn Steinbrink
2008-06-28 20:57 ` [PATCH] Revert "git-svn: Speed up fetch" Avery Pennarun
2008-06-28 23:33 ` Avery Pennarun [this message]
2008-06-29 0:58 ` [PATCH v2] git-svn: avoid filling up the disk with temp files Björn Steinbrink
2008-06-29 1:21 ` [PATCH] git cat-file: Fix memory leak in batch mode Björn Steinbrink
2008-06-29 3:36 ` Junio C Hamano
2008-06-29 11:54 ` Björn Steinbrink
2008-06-29 2:24 ` [PATCH v2] git-svn: avoid filling up the disk with temp files Björn Steinbrink
2008-06-29 2:38 ` Eric Wong
2008-06-28 23:51 ` [PATCH] Revert "git-svn: Speed up fetch" Mikael Magnusson
2008-06-28 23:58 ` Avery Pennarun
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1214696036-8294-1-git-send-email-apenwarr@gmail.com \
--to=apenwarr@gmail.com \
--cc=B.Steinbrink@gmx.de \
--cc=aroben@apple.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=naesten@gmail.com \
--cc=normalperson@yhbt.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).