From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?UTF-8?q?Nguy=E1=BB=85n=20Th=C3=A1i=20Ng=E1=BB=8Dc=20Duy?= Subject: [PATCH v3 04/11] parse_object: special code path for blobs to avoid putting whole object in memory Date: Mon, 5 Mar 2012 10:43:41 +0700 Message-ID: <1330919028-6611-5-git-send-email-pclouds@gmail.com> References: <1330865996-2069-1-git-send-email-pclouds@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: Junio C Hamano , =?UTF-8?q?Nguy=E1=BB=85n=20Th=C3=A1i=20Ng=E1=BB=8Dc=20Duy?= To: git@vger.kernel.org X-From: git-owner@vger.kernel.org Mon Mar 05 04:44:59 2012 Return-path: Envelope-to: gcvg-git-2@plane.gmane.org Received: from vger.kernel.org ([209.132.180.67]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1S4Oqz-0003id-GJ for gcvg-git-2@plane.gmane.org; Mon, 05 Mar 2012 04:44:53 +0100 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755433Ab2CEDot convert rfc822-to-quoted-printable (ORCPT ); Sun, 4 Mar 2012 22:44:49 -0500 Received: from mail-pw0-f46.google.com ([209.85.160.46]:41418 "EHLO mail-pw0-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754920Ab2CEDos (ORCPT ); Sun, 4 Mar 2012 22:44:48 -0500 Received: by mail-pw0-f46.google.com with SMTP id un15so2292252pbc.19 for ; Sun, 04 Mar 2012 19:44:47 -0800 (PST) Received-SPF: pass (google.com: domain of pclouds@gmail.com designates 10.68.202.135 as permitted sender) client-ip=10.68.202.135; Authentication-Results: mr.google.com; spf=pass (google.com: domain of pclouds@gmail.com designates 10.68.202.135 as permitted sender) smtp.mail=pclouds@gmail.com; dkim=pass header.i=pclouds@gmail.com Received: from mr.google.com ([10.68.202.135]) by 10.68.202.135 with SMTP id ki7mr37539374pbc.158.1330919087882 (num_hops = 1); Sun, 04 Mar 2012 19:44:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id:x-mailer:in-reply-to:references :mime-version:content-type:content-transfer-encoding; bh=7lebsHxzz6VmqIp4GlvkUqB+tXpTkaIje4HMLOtQAi4=; b=bEFbe/BwT7/EtdPR0lOq/SxZ8l1SlZoNUu2fTH3ZpysZWqbeZ5rcZjz7c91n4aZWoP SO8vU/PCPpZINYCmAtYzVYeL85SjEWx33TeEjtrJNb1+f/Tz20i1h4vqQpJsl7uFe7G+ Bg276JsVCVzKYWlb5ccPwwA/E4HWk4udlFz+bikSyZ15eW8hjX/uR3hZ9v24oKU5SP7X yaLXhyODoUY1g5Jvg4tgvMEyPoThJPHvCekWZBcnjiXr+QCR8lSGO8hsn+FvwS1B4kX4 1czWePwYTgA1UXLK7JiQOhoRgIxOt/4DKN+7dgkgQMs3i8O0Xi3dXOQ5QuLOiHIOcuqt uJWw== Received: by 10.68.202.135 with SMTP id ki7mr32319716pbc.158.1330919087847; Sun, 04 Mar 2012 19:44:47 -0800 (PST) Received: from pclouds@gmail.com ([113.161.77.29]) by mx.google.com with ESMTPS id h2sm11629151pbv.18.2012.03.04.19.44.41 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 04 Mar 2012 19:44:44 -0800 (PST) Received: by pclouds@gmail.com (sSMTP sendmail emulation); Mon, 05 Mar 2012 10:44:19 +0700 X-Mailer: git-send-email 1.7.3.1.256.g2539c.dirty In-Reply-To: <1330865996-2069-1-git-send-email-pclouds@gmail.com> Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Archived-At: Signed-off-by: Nguy=E1=BB=85n Th=C3=A1i Ng=E1=BB=8Dc Duy --- object.c | 11 +++++++++++ sha1_file.c | 33 ++++++++++++++++++++++++++++++++- 2 files changed, 43 insertions(+), 1 deletions(-) diff --git a/object.c b/object.c index 6b06297..0498b18 100644 --- a/object.c +++ b/object.c @@ -198,6 +198,17 @@ struct object *parse_object(const unsigned char *s= ha1) if (obj && obj->parsed) return obj; =20 + if ((obj && obj->type =3D=3D OBJ_BLOB) || + (!obj && has_sha1_file(sha1) && + sha1_object_info(sha1, NULL) =3D=3D OBJ_BLOB)) { + if (check_sha1_signature(repl, NULL, 0, NULL) < 0) { + error("sha1 mismatch %s\n", sha1_to_hex(repl)); + return NULL; + } + parse_blob_buffer(lookup_blob(sha1), NULL, 0); + return lookup_object(sha1); + } + buffer =3D read_sha1_file(sha1, &type, &size); if (buffer) { if (check_sha1_signature(repl, buffer, size, typename(type)) < 0) { diff --git a/sha1_file.c b/sha1_file.c index f9f8d5e..a77ef0a 100644 --- a/sha1_file.c +++ b/sha1_file.c @@ -19,6 +19,7 @@ #include "pack-revindex.h" #include "sha1-lookup.h" #include "bulk-checkin.h" +#include "streaming.h" =20 #ifndef O_NOATIME #if defined(__linux__) && (defined(__i386__) || defined(__PPC__)) @@ -1149,7 +1150,37 @@ static const struct packed_git *has_packed_and_b= ad(const unsigned char *sha1) int check_sha1_signature(const unsigned char *sha1, void *map, unsigne= d long size, const char *type) { unsigned char real_sha1[20]; - hash_sha1_file(map, size, type, real_sha1); + enum object_type obj_type; + struct git_istream *st; + git_SHA_CTX c; + char hdr[32]; + int hdrlen; + + if (map) { + hash_sha1_file(map, size, type, real_sha1); + return hashcmp(sha1, real_sha1) ? -1 : 0; + } + + st =3D open_istream(sha1, &obj_type, &size, NULL); + if (!st) + return -1; + + /* Generate the header */ + hdrlen =3D sprintf(hdr, "%s %lu", typename(obj_type), size) + 1; + + /* Sha1.. */ + git_SHA1_Init(&c); + git_SHA1_Update(&c, hdr, hdrlen); + for (;;) { + char buf[1024 * 16]; + ssize_t readlen =3D read_istream(st, buf, sizeof(buf)); + + if (!readlen) + break; + git_SHA1_Update(&c, buf, readlen); + } + git_SHA1_Final(real_sha1, &c); + close_istream(st); return hashcmp(sha1, real_sha1) ? -1 : 0; } =20 --=20 1.7.3.1.256.g2539c.dirty