From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?UTF-8?q?Nguy=E1=BB=85n=20Th=C3=A1i=20Ng=E1=BB=8Dc=20Duy?= Subject: [PATCH 1/5] read-cache: add watchman 'WAMA' extension Date: Sun, 1 Nov 2015 14:55:42 +0100 Message-ID: <1446386146-10438-2-git-send-email-pclouds@gmail.com> References: <1446386146-10438-1-git-send-email-pclouds@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: Christian Couder , =?UTF-8?q?Nguy=E1=BB=85n=20Th=C3=A1i=20Ng=E1=BB=8Dc=20Duy?= To: git@vger.kernel.org X-From: git-owner@vger.kernel.org Sun Nov 01 14:56:14 2015 Return-path: Envelope-to: gcvg-git-2@plane.gmane.org Received: from vger.kernel.org ([209.132.180.67]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1Zst7F-0002GJ-Mj for gcvg-git-2@plane.gmane.org; Sun, 01 Nov 2015 14:56:14 +0100 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752727AbbKAN4J convert rfc822-to-quoted-printable (ORCPT ); Sun, 1 Nov 2015 08:56:09 -0500 Received: from mail-wi0-f173.google.com ([209.85.212.173]:34142 "EHLO mail-wi0-f173.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752681AbbKAN4F (ORCPT ); Sun, 1 Nov 2015 08:56:05 -0500 Received: by wikq8 with SMTP id q8so36600171wik.1 for ; Sun, 01 Nov 2015 05:56:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; bh=RPlOD+Oxosbjq8h18vo5K6QM8NuivE1uN2A1zQTuqsU=; b=jXJmhi0iHcUMUdkf/b3bkh8hHNFqsPRFnX4HsjCt1Zod2zrigP/OtA4x04M+qlZk7f ptvJK4GgI05QqJauFebHpEXNNQ90MOxhtfZfr0ij5VQg5775F3+qKvG+5i3PfMCOiliP 9EVZ00dUu41Mxme/Gmmyk/kXtm4OCCmZq3bvDPTvCC7WfzOHwRpEYJrFae8Pg0dhU9VO gp6YmfS+JF4Vn2RTdJcRRi4hJR/lucPJViM4fEATOTQDtQBblsSm3sCeAgeGhCgWOxIU 8qTYoAo6MVRNXJiMirL7gv6OQwXX0Mqy3Q9NlAKiHai2/8gRpIi3t6d6Q3WI2YqMheT5 SlhQ== X-Received: by 10.195.11.98 with SMTP id eh2mr18671362wjd.59.1446386162851; Sun, 01 Nov 2015 05:56:02 -0800 (PST) Received: from duynguyen-vnpc.dek-tpc.internal.homenet.telecomitalia.it (host171-164-static.60-79-b.business.telecomitalia.it. [79.60.164.171]) by smtp.gmail.com with ESMTPSA id bk2sm17185525wjc.3.2015.11.01.05.56.00 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 01 Nov 2015 05:56:01 -0800 (PST) X-Mailer: git-send-email 2.2.0.513.g477eb31 In-Reply-To: <1446386146-10438-1-git-send-email-pclouds@gmail.com> Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Archived-At: The extension contains a bitmap, one bit for each entry in the index. If the n-th bit is zero, the n-th entry is considered unchanged, we can ce_mark_uptodate() it without refreshing. If the bit is non-zero and we found out the corresponding file is clean after refresh, we can clear the bit. The 'skipping refresh' bit is not in this patch yet as we would need watchman. More details in later patches. Signed-off-by: Nguy=E1=BB=85n Th=C3=A1i Ng=E1=BB=8Dc Duy --- cache.h | 4 ++++ read-cache.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++++++++= ++++++-- 2 files changed, 73 insertions(+), 2 deletions(-) diff --git a/cache.h b/cache.h index 9633acc..a05fd31 100644 --- a/cache.h +++ b/cache.h @@ -163,6 +163,8 @@ struct cache_entry { #define CE_VALID (0x8000) #define CE_STAGESHIFT 12 =20 +#define CE_NO_WATCH (0x0001) + /* * Range 0xFFFF0FFF in ce_flags is divided into * two parts: in-memory flags and on-disk ones. @@ -298,6 +300,7 @@ static inline unsigned int canon_mode(unsigned int = mode) #define CACHE_TREE_CHANGED (1 << 5) #define SPLIT_INDEX_ORDERED (1 << 6) #define UNTRACKED_CHANGED (1 << 7) +#define WATCHMAN_CHANGED (1 << 8) =20 struct split_index; struct untracked_cache; @@ -322,6 +325,7 @@ struct index_state { struct untracked_cache *untracked; void *mmap; size_t mmap_size; + char *last_update; }; =20 extern struct index_state the_index; diff --git a/read-cache.c b/read-cache.c index f609776..893223e 100644 --- a/read-cache.c +++ b/read-cache.c @@ -19,6 +19,7 @@ #include "split-index.h" #include "utf8.h" #include "shm.h" +#include "ewah/ewok.h" =20 static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, unsigned int options); @@ -41,11 +42,13 @@ static struct cache_entry *refresh_cache_entry(stru= ct cache_entry *ce, #define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* "REUC" */ #define CACHE_EXT_LINK 0x6c696e6b /* "link" */ #define CACHE_EXT_UNTRACKED 0x554E5452 /* "UNTR" */ +#define CACHE_EXT_WATCHMAN 0x57414D41 /* "WAMA" */ =20 /* changes that can be kept in $GIT_DIR/index (basically all extension= s) */ #define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \ CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \ - SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED) + SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED | \ + WATCHMAN_CHANGED) =20 struct index_state the_index; static const char *alternate_index_output; @@ -1224,8 +1227,13 @@ int refresh_index(struct index_state *istate, un= signed int flags, continue; =20 new =3D refresh_cache_ent(istate, ce, options, &cache_errno, &change= d); - if (new =3D=3D ce) + if (new =3D=3D ce) { + if (ce->ce_flags & CE_NO_WATCH) { + ce->ce_flags &=3D ~CE_NO_WATCH; + istate->cache_changed |=3D WATCHMAN_CHANGED; + } continue; + } if (!new) { const char *fmt; =20 @@ -1369,6 +1377,48 @@ static int verify_hdr(const struct cache_header = *hdr, unsigned long size) return 0; } =20 +static void mark_no_watchman(size_t pos, void *data) +{ + struct index_state *istate =3D data; + assert(pos < istate->cache_nr); + istate->cache[pos]->ce_flags |=3D CE_NO_WATCH; +} + +static int read_watchman_ext(struct index_state *istate, const void *d= ata, + unsigned long sz) +{ + struct ewah_bitmap *bitmap; + int ret, len; + + if (memchr(data, 0, sz) =3D=3D NULL) + return error("invalid extension"); + len =3D strlen(data) + 1; + bitmap =3D ewah_new(); + ret =3D ewah_read_mmap(bitmap, (const char *)data + len, sz - len); + if (ret !=3D sz - len) { + ewah_free(bitmap); + return error("fail to parse ewah bitmap"); + } + istate->last_update =3D xstrdup(data); + ewah_each_bit(bitmap, mark_no_watchman, istate); + ewah_free(bitmap); + return 0; +} + +static void write_watchman_ext(struct strbuf *sb, struct index_state* = istate) +{ + struct ewah_bitmap *bitmap; + int i; + + strbuf_add(sb, istate->last_update, strlen(istate->last_update) + 1); + bitmap =3D ewah_new(); + for (i =3D 0; i < istate->cache_nr; i++) + if (istate->cache[i]->ce_flags & CE_NO_WATCH) + ewah_set(bitmap, i); + ewah_serialize_strbuf(bitmap, sb); + ewah_free(bitmap); +} + static int read_index_extension(struct index_state *istate, const char *ext, void *data, unsigned long sz) { @@ -1386,6 +1436,11 @@ static int read_index_extension(struct index_sta= te *istate, case CACHE_EXT_UNTRACKED: istate->untracked =3D read_untracked_extension(data, sz); break; + + case CACHE_EXT_WATCHMAN: + read_watchman_ext(istate, data, sz); + break; + default: if (*ext < 'A' || 'Z' < *ext) return error("index uses %.4s extension, which we do not understand= ", @@ -1794,6 +1849,8 @@ int discard_index(struct index_state *istate) istate->untracked =3D NULL; istate->from_shm =3D 0; istate->to_shm =3D 0; + free(istate->last_update); + istate->last_update =3D NULL; return 0; } =20 @@ -2191,6 +2248,16 @@ static int do_write_index(struct index_state *is= tate, int newfd, if (err) return -1; } + if (!strip_extensions && istate->last_update) { + struct strbuf sb =3D STRBUF_INIT; + + write_watchman_ext(&sb, istate); + err =3D write_index_ext_header(&c, newfd, CACHE_EXT_WATCHMAN, sb.len= ) < 0 + || ce_write(&c, newfd, sb.buf, sb.len) < 0; + strbuf_release(&sb); + if (err) + return -1; + } =20 if (ce_flush(&c, newfd, istate->sha1) || fstat(newfd, &st)) return -1; --=20 2.2.0.513.g477eb31