* Re: [FYI] very large text files and their problems.
From: Nguyen Thai Ngoc Duy @ 2012-02-24 11:14 UTC (permalink / raw)
To: Ian Kumlien; +Cc: git
In-Reply-To: <20120224101121.GA9526@pomac.netswarm.net>
On Fri, Feb 24, 2012 at 5:11 PM, Ian Kumlien <pomac@vapor.com> wrote:
> I'm uncertain if you got my reply since i did it out of bounds - so i'll
> repeat myself - sorry... =)
yes I received it, just too busy this week.
>> > git needs to have atleast the same ammount of memory as the largest
>> > file free... Couldn't this be worked around?
>> >
>> > On a (32 bit) machine with 4GB memory - results in:
>> > fatal: Out of memory, malloc failed (tried to allocate 3310214313 bytes)
>> >
>> > (and i see how this could be a problem, but couldn't it be mitigated? or
>> > is it bydesign and intended behaviour?)
>>
>> I think that it's delta resolving that hogs all your memory. If your
>> files are smaller than 512M, try lower core.bigFileThreshold. The
>> topic jc/split-blob, which stores a big file are several smaller
>> pieces, might solve your problem. Unfortunately the topic is not
>> complete yet.
>
> Well, in this case it's just stream unpacking gzip data to disk, i
> understand if delta would be a problem... But wouldn't delta be a
> problem in the sence of <size_of_change>+<size_of_subdata>+<result> ?
>
> Ie, if the file is mmapped - it shouldn't have to be allocated, right?
We should not delta large files. I was worried that the large file
check could go wrong, But I guess your blob's not deltified in this
case.
When you receive a pack during a clone, the pack is streamed to
index-pack, not mmapped, and index-pack checks every object in there
in uncompressed form. I think I have found a way to avoid allocating
that much. Need some more check, then send out.
--
Duy
^ permalink raw reply
* [PATCH 1/2] Skip SHA-1 collision test on "index-pack --verify"
From: Nguyễn Thái Ngọc Duy @ 2012-02-24 12:23 UTC (permalink / raw)
To: git; +Cc: Ian Kumlien, Nguyễn Thái Ngọc Duy
index-pack --verify (or verify-pack) is about verifying the pack
itself. SHA-1 collision test is about outside (probably malicious)
objects with the same SHA-1 entering current repo.
SHA-1 collision test is currently done unconditionally. Which means if
you verify an in-repo pack, all objects from the pack will be checked
against objects in repo, which are themselves.
Skip this test for --verify, unless --strict is also specified.
linux-2.6 $ ls -sh .git/objects/pack/pack-e7732c98a8d54840add294c3c562840f78764196.pack
401M .git/objects/pack/pack-e7732c98a8d54840add294c3c562840f78764196.pack
Without the patch (and with another patch to cut out second pass in
index-pack):
linux-2.6 $ time ~/w/git/old index-pack -v --verify .git/objects/pack/pack-e7732c98a8d54840add294c3c562840f78764196.pack
Indexing objects: 100% (1944656/1944656), done.
fatal: pack has 1617280 unresolved deltas
real 1m1.223s
user 0m55.028s
sys 0m0.828s
With the patch:
linux-2.6 $ time ~/w/git/git index-pack -v --verify .git/objects/pack/pack-e7732c98a8d54840add294c3c562840f78764196.pack
Indexing objects: 100% (1944656/1944656), done.
fatal: pack has 1617280 unresolved deltas
real 0m41.714s
user 0m40.994s
sys 0m0.550s
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
builtin/index-pack.c | 5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index dd1c5c9..cee83b9 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -62,6 +62,7 @@ static int nr_resolved_deltas;
static int from_stdin;
static int strict;
+static int verify;
static int verbose;
static struct progress *progress;
@@ -461,7 +462,7 @@ static void sha1_object(const void *data, unsigned long size,
enum object_type type, unsigned char *sha1)
{
hash_sha1_file(data, size, typename(type), sha1);
- if (has_sha1_file(sha1)) {
+ if ((strict || !verify) && has_sha1_file(sha1)) {
void *has_data;
enum object_type has_type;
unsigned long has_size;
@@ -1078,7 +1079,7 @@ static void show_pack_info(int stat_only)
int cmd_index_pack(int argc, const char **argv, const char *prefix)
{
- int i, fix_thin_pack = 0, verify = 0, stat_only = 0, stat = 0;
+ int i, fix_thin_pack = 0, stat_only = 0, stat = 0;
const char *curr_pack, *curr_index;
const char *index_name = NULL, *pack_name = NULL;
const char *keep_name = NULL, *keep_msg = NULL;
--
1.7.8.36.g69ee2
^ permalink raw reply related
* [PATCH 2/2] index-pack: reduce memory usage when the pack has large blobs
From: Nguyễn Thái Ngọc Duy @ 2012-02-24 12:23 UTC (permalink / raw)
To: git; +Cc: Ian Kumlien, Nguyễn Thái Ngọc Duy
In-Reply-To: <1330086201-13916-1-git-send-email-pclouds@gmail.com>
This command unpacks every non-delta objects in order to:
1. calculate sha-1
2. do byte-to-byte sha-1 collision test if we happen to have objects
with the same sha-1
3. validate object content in strict mode
All this requires the entire object to stay in memory, a bad news for
giant blobs. This patch lowers memory consumption by not saving the
object in memory whenever possible, calculating SHA-1 while unpacking
the object.
This patch assumes that the collision test is rarely needed. The
collision test will be done later in second pass if necessary, which
puts the entire object back to memory again (We could even do the
collision test without putting the entire object back in memory, by
comparing as we unpack it).
In strict mode, it always keeps non-blob objects in memory for
validation (blobs do not need data validation). "--strict --verify"
also keeps blobs in memory.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
Does anybody do "git index-pack --stdin < .git/objects/pack/something"?
builtin/index-pack.c | 74 +++++++++++++++++++++++++++++++++++++++++---------
1 files changed, 61 insertions(+), 13 deletions(-)
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index cee83b9..0c1f915 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -277,30 +277,60 @@ static void unlink_base_data(struct base_data *c)
free_base_data(c);
}
-static void *unpack_entry_data(unsigned long offset, unsigned long size)
+static void *unpack_entry_data(unsigned long offset, unsigned long size,
+ enum object_type type, unsigned char *sha1)
{
+ static char fixed_buf[8192];
int status;
git_zstream stream;
- void *buf = xmalloc(size);
+ void *buf;
+ git_SHA_CTX c;
+
+ if (sha1) { /* do hash_sha1_file internally */
+ char hdr[32];
+ int hdrlen = sprintf(hdr, "%s %lu", typename(type), size)+1;
+ git_SHA1_Init(&c);
+ git_SHA1_Update(&c, hdr, hdrlen);
+
+ buf = fixed_buf;
+ } else {
+ buf = xmalloc(size);
+ }
memset(&stream, 0, sizeof(stream));
git_inflate_init(&stream);
stream.next_out = buf;
- stream.avail_out = size;
+ stream.avail_out = buf == fixed_buf ? sizeof(fixed_buf) : size;
do {
stream.next_in = fill(1);
stream.avail_in = input_len;
status = git_inflate(&stream, 0);
use(input_len - stream.avail_in);
+ if (sha1) {
+ git_SHA1_Update(&c, buf, stream.next_out - (unsigned char *)buf);
+ stream.next_out = buf;
+ stream.avail_out = sizeof(fixed_buf);
+ }
} while (status == Z_OK);
if (stream.total_out != size || status != Z_STREAM_END)
bad_object(offset, "inflate returned %d", status);
git_inflate_end(&stream);
+ if (sha1) {
+ git_SHA1_Final(sha1, &c);
+ buf = NULL;
+ }
return buf;
}
-static void *unpack_raw_entry(struct object_entry *obj, union delta_base *delta_base)
+static int is_delta_type(enum object_type type)
+{
+ return (type == OBJ_REF_DELTA || type == OBJ_OFS_DELTA);
+}
+
+static void *unpack_raw_entry(struct object_entry *obj,
+ union delta_base *delta_base,
+ unsigned char *sha1)
{
unsigned char *p;
unsigned long size, c;
@@ -360,7 +390,17 @@ static void *unpack_raw_entry(struct object_entry *obj, union delta_base *delta_
}
obj->hdr_size = consumed_bytes - obj->idx.offset;
- data = unpack_entry_data(obj->idx.offset, obj->size);
+ /*
+ * --verify --strict: sha1_object() does all collision test
+ * --strict: sha1_object() does all except blobs,
+ * blobs tested in second pass
+ * --verify : no collision test
+ * : all in second pass
+ */
+ if (is_delta_type(obj->type) ||
+ (strict && (verify || obj->type != OBJ_BLOB)))
+ sha1 = NULL; /* save unpacked object */
+ data = unpack_entry_data(obj->idx.offset, obj->size, obj->type, sha1);
obj->idx.crc32 = input_crc32;
return data;
}
@@ -461,8 +501,9 @@ static void find_delta_children(const union delta_base *base,
static void sha1_object(const void *data, unsigned long size,
enum object_type type, unsigned char *sha1)
{
- hash_sha1_file(data, size, typename(type), sha1);
- if ((strict || !verify) && has_sha1_file(sha1)) {
+ if (data)
+ hash_sha1_file(data, size, typename(type), sha1);
+ if (data && has_sha1_file(sha1)) {
void *has_data;
enum object_type has_type;
unsigned long has_size;
@@ -511,11 +552,6 @@ static void sha1_object(const void *data, unsigned long size,
}
}
-static int is_delta_type(enum object_type type)
-{
- return (type == OBJ_REF_DELTA || type == OBJ_OFS_DELTA);
-}
-
/*
* This function is part of find_unresolved_deltas(). There are two
* walkers going in the opposite ways.
@@ -702,7 +738,7 @@ static void parse_pack_objects(unsigned char *sha1)
nr_objects);
for (i = 0; i < nr_objects; i++) {
struct object_entry *obj = &objects[i];
- void *data = unpack_raw_entry(obj, &delta->base);
+ void *data = unpack_raw_entry(obj, &delta->base, obj->idx.sha1);
obj->real_type = obj->type;
if (is_delta_type(obj->type)) {
nr_deltas++;
@@ -744,6 +780,9 @@ static void parse_pack_objects(unsigned char *sha1)
* - if used as a base, uncompress the object and apply all deltas,
* recursively checking if the resulting object is used as a base
* for some more deltas.
+ * - if the same object exists in repository and we're not in strict
+ * mode, we skipped the sha-1 collision test in the first pass.
+ * Do it now.
*/
if (verbose)
progress = start_progress("Resolving deltas", nr_deltas);
@@ -753,6 +792,15 @@ static void parse_pack_objects(unsigned char *sha1)
if (is_delta_type(obj->type))
continue;
+
+ if (((!strict && !verify) ||
+ (strict && !verify && obj->type == OBJ_BLOB)) &&
+ has_sha1_file(obj->idx.sha1)) {
+ void *data = get_data_from_pack(obj);
+ sha1_object(data, obj->size, obj->type, obj->idx.sha1);
+ free(data);
+ }
+
base_obj->obj = obj;
base_obj->data = NULL;
find_unresolved_deltas(base_obj);
--
1.7.8.36.g69ee2
^ permalink raw reply related
* Re: [FYI] very large text files and their problems.
From: Ian Kumlien @ 2012-02-24 12:55 UTC (permalink / raw)
To: Nguyen Thai Ngoc Duy; +Cc: git
In-Reply-To: <CACsJy8Aq=ofaHo3RkOBWTYP3eehZAU0E=HeDoUpR0nh+seKxzA@mail.gmail.com>
On Fri, Feb 24, 2012 at 06:14:46PM +0700, Nguyen Thai Ngoc Duy wrote:
> On Fri, Feb 24, 2012 at 5:11 PM, Ian Kumlien <pomac@vapor.com> wrote:
> > I'm uncertain if you got my reply since i did it out of bounds - so i'll
> > repeat myself - sorry... =)
>
> yes I received it, just too busy this week.
Ah good, you never know what anti-spam measures people applies these
days... =)
And i have the same, so i totally understand.
> >> > git needs to have atleast the same ammount of memory as the largest
> >> > file free... Couldn't this be worked around?
> >> >
> >> > On a (32 bit) machine with 4GB memory - results in:
> >> > fatal: Out of memory, malloc failed (tried to allocate 3310214313 bytes)
> >> >
> >> > (and i see how this could be a problem, but couldn't it be mitigated? or
> >> > is it bydesign and intended behaviour?)
> >>
> >> I think that it's delta resolving that hogs all your memory. If your
> >> files are smaller than 512M, try lower core.bigFileThreshold. The
> >> topic jc/split-blob, which stores a big file are several smaller
> >> pieces, might solve your problem. Unfortunately the topic is not
> >> complete yet.
> >
> > Well, in this case it's just stream unpacking gzip data to disk, i
> > understand if delta would be a problem... But wouldn't delta be a
> > problem in the sence of <size_of_change>+<size_of_subdata>+<result> ?
> >
> > Ie, if the file is mmapped - it shouldn't have to be allocated, right?
>
> We should not delta large files. I was worried that the large file
> check could go wrong, But I guess your blob's not deltified in this
> case.
That would be correct
> When you receive a pack during a clone, the pack is streamed to
> index-pack, not mmapped, and index-pack checks every object in there
> in uncompressed form. I think I have found a way to avoid allocating
> that much. Need some more check, then send out.
Ah! That explains alot - do you have a publicly available version i
could look at?
> --
> Duy
^ permalink raw reply
* Re: Submodule status inside nested submodule fails
From: Charles Brossollet @ 2012-02-24 13:41 UTC (permalink / raw)
To: git
In-Reply-To: <loom.20120224T104003-230@post.gmane.org>
Charles Brossollet <chbrosso <at> lltech.fr> writes:
> Searching for message in git source I found the test triggering the message to
> be an empty result for "git rev-parse --show-cdup". Running the command in
> ext/submodule strangely returns the *absolute* path of ext/submodule, which
> should not happen because this command returns path of current dir relative to
> the containing working tree.
Things are OK if I run the commands from Windows' CMD, but it fails when run in
MinGW shell, so I believe the problem is related to MsysGit, discussion follows
up on their list.
^ permalink raw reply
* cvs2git multiple session for repository migration...
From: supadhyay @ 2012-02-24 14:08 UTC (permalink / raw)
To: git
Hi All,
I am migrating my CVS repository to GIt. As I have multiple repository I
want to start multiple session for cvs2git,but it failed to start. I
received cvs2svn.lock file error ? Is there any workaround to start multiple
cvs2git session (which migrate the CVS directory) ??
Thanks in advance...
--
View this message in context: http://git.661346.n2.nabble.com/cvs2git-multiple-session-for-repository-migration-tp7314909p7314909.html
Sent from the git mailing list archive at Nabble.com.
^ permalink raw reply
* Re: [PATCH 2/2] index-pack: reduce memory usage when the pack has large blobs
From: Ian Kumlien @ 2012-02-24 14:30 UTC (permalink / raw)
To: Nguyễn Thái Ngọc Duy; +Cc: git
In-Reply-To: <1330086201-13916-2-git-send-email-pclouds@gmail.com>
On Fri, Feb 24, 2012 at 07:23:21PM +0700, Nguyễn Thái Ngọc Duy wrote:
> This command unpacks every non-delta objects in order to:
>
> 1. calculate sha-1
> 2. do byte-to-byte sha-1 collision test if we happen to have objects
> with the same sha-1
> 3. validate object content in strict mode
>
> All this requires the entire object to stay in memory, a bad news for
> giant blobs. This patch lowers memory consumption by not saving the
> object in memory whenever possible, calculating SHA-1 while unpacking
> the object.
>
> This patch assumes that the collision test is rarely needed. The
> collision test will be done later in second pass if necessary, which
> puts the entire object back to memory again (We could even do the
> collision test without putting the entire object back in memory, by
> comparing as we unpack it).
>
> In strict mode, it always keeps non-blob objects in memory for
> validation (blobs do not need data validation). "--strict --verify"
> also keeps blobs in memory.
I applied both patches to git master, with some manual tinkering so i
might have missed some change that caused this to break.
But i get a segmentation fault and i just thought that i'd send you a
small trace before i even start trying to look in to this:
0xb7eb5b43 in SHA1_Update () from /lib/i686/cmov/libcrypto.so.0.9.8
(gdb) bt
#0 0xb7eb5b43 in SHA1_Update () from /lib/i686/cmov/libcrypto.so.0.9.8
#1 0x08116a2d in write_sha1_file_prepare
#2 0x08116a83 in hash_sha1_file
#3 0x0807c2a6 in sha1_object
#4 0x0807d74a in parse_pack_objects
#5 0x0807de6f in cmd_index_pack
#6 0x0804be97 in run_builtin
#7 handle_internal_command
#8 0x0804c0ad in run_argv
#9 main
Sorry about the censorship but i don't know how sensetive this data
is...
sha1_file.c:2343
---
static void write_sha1_file_prepare(const void *buf, unsigned long len,
const char *type, unsigned char *sha1,
char *hdr, int *hdrlen)
{
git_SHA_CTX c;
/* Generate the header */
*hdrlen = sprintf(hdr, "%s %lu", type, len)+1;
/* Sha1.. */
git_SHA1_Init(&c);
git_SHA1_Update(&c, hdr, *hdrlen);
git_SHA1_Update(&c, buf, len); <== this line fails.
git_HA1_Final(sha1, &c);
}
---
Just keep sending patches, i have atleast one git to test it on. ;)
^ permalink raw reply
* FW: question about merge in 1.7.10
From: Marlene Cote @ 2012-02-24 14:33 UTC (permalink / raw)
To: git@vger.kernel.org
I used merge with -no-commit -no-ff. So, I assume I won't see any change in behavior, since merge is not performing commits, right?
------------------
Regards,
Marlene Cote, Release Engineering
978-268-0821
^ permalink raw reply
* Re: [PATCH 2/2] index-pack: reduce memory usage when the pack has large blobs
From: Ian Kumlien @ 2012-02-24 14:40 UTC (permalink / raw)
To: Nguyễn Thái Ngọc Duy; +Cc: git
In-Reply-To: <1330086201-13916-2-git-send-email-pclouds@gmail.com>
On Fri, Feb 24, 2012 at 07:23:21PM +0700, Nguyễn Thái Ngọc Duy wrote:
> This command unpacks every non-delta objects in order to:
>
> 1. calculate sha-1
> 2. do byte-to-byte sha-1 collision test if we happen to have objects
> with the same sha-1
> 3. validate object content in strict mode
>
> All this requires the entire object to stay in memory, a bad news for
> giant blobs. This patch lowers memory consumption by not saving the
> object in memory whenever possible, calculating SHA-1 while unpacking
> the object.
>
> This patch assumes that the collision test is rarely needed. The
> collision test will be done later in second pass if necessary, which
> puts the entire object back to memory again (We could even do the
> collision test without putting the entire object back in memory, by
> comparing as we unpack it).
>
> In strict mode, it always keeps non-blob objects in memory for
> validation (blobs do not need data validation). "--strict --verify"
> also keeps blobs in memory.
>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Actually, nevermind my last report - i had missed a merge :(
And now that i merged that part it seems like it doesn't do much..
(No real output for 2+ minutes)
I think i should reapply the patches again and verify that everything is
correct before reporting any additional progress.
But, this might not be before monday, unfortunately... But *thanks* for
posting the patches!
^ permalink raw reply
* Re: [PATCH 2/2] index-pack: reduce memory usage when the pack has large blobs
From: Ian Kumlien @ 2012-02-24 15:37 UTC (permalink / raw)
To: Nguyễn Thái Ngọc Duy; +Cc: git
In-Reply-To: <1330086201-13916-2-git-send-email-pclouds@gmail.com>
On Fri, Feb 24, 2012 at 07:23:21PM +0700, Nguyễn Thái Ngọc Duy wrote:
> This command unpacks every non-delta objects in order to:
>
> 1. calculate sha-1
> 2. do byte-to-byte sha-1 collision test if we happen to have objects
> with the same sha-1
> 3. validate object content in strict mode
>
> All this requires the entire object to stay in memory, a bad news for
> giant blobs. This patch lowers memory consumption by not saving the
> object in memory whenever possible, calculating SHA-1 while unpacking
> the object.
>
> This patch assumes that the collision test is rarely needed. The
> collision test will be done later in second pass if necessary, which
> puts the entire object back to memory again (We could even do the
> collision test without putting the entire object back in memory, by
> comparing as we unpack it).
>
> In strict mode, it always keeps non-blob objects in memory for
> validation (blobs do not need data validation). "--strict --verify"
> also keeps blobs in memory.
>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Finally, reapplied the patches and so on:
remote: Counting objects: 1425, done.
remote: Compressing objects: 100% (617/617), done.
remote: Total 1425 (delta 790), reused 1425 (delta 790)
Receiving objects: 100% (1425/1425), 56.06 MiB | 3.97 MiB/s, done.
Resolving deltas: 100% (790/790), done.
real 1m57.742s
user 0m29.950s
sys 0m6.308s
*YAY*
I wonder how the hell i could have missed several parts of the patch =(
But there seems to be some issue in gerrit 2.1.8, will have to check
against a newer gerrit to verify if it's still a problem.
FYI - it seems to hang doing nothing.
As for your patches:
Tested-by: Ian Kumlien <pomac@vapor.com>
;)
^ permalink raw reply
* Re: [PATCH 2/2] index-pack: reduce memory usage when the pack has large blobs
From: Ian Kumlien @ 2012-02-24 16:16 UTC (permalink / raw)
To: Nguyễn Thái Ngọc Duy; +Cc: git
In-Reply-To: <1330086201-13916-2-git-send-email-pclouds@gmail.com>
On Fri, Feb 24, 2012 at 07:23:21PM +0700, Nguyễn Thái Ngọc Duy wrote:
> This command unpacks every non-delta objects in order to:
>
> 1. calculate sha-1
> 2. do byte-to-byte sha-1 collision test if we happen to have objects
> with the same sha-1
> 3. validate object content in strict mode
>
> All this requires the entire object to stay in memory, a bad news for
> giant blobs. This patch lowers memory consumption by not saving the
> object in memory whenever possible, calculating SHA-1 while unpacking
> the object.
>
> This patch assumes that the collision test is rarely needed. The
> collision test will be done later in second pass if necessary, which
> puts the entire object back to memory again (We could even do the
> collision test without putting the entire object back in memory, by
> comparing as we unpack it).
>
> In strict mode, it always keeps non-blob objects in memory for
> validation (blobs do not need data validation). "--strict --verify"
> also keeps blobs in memory.
>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Writing objects: 100% (1425/1425), 56.06 MiB | 4.62 MiB/s, done.
Total 1425 (delta 790), reused 1425 (delta 790)
fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
To ../test_data/
! [remote rejected] master -> master (missing necessary objects)
! [remote rejected] origin/HEAD -> origin/HEAD (missing necessary objects)
! [remote rejected] origin/master -> origin/master (missing necessary objects)
error: failed to push some refs to '../test_data/'
So there are additional code paths to look at... =(
^ permalink raw reply
* Re: git gui: how to fetch a single branch
From: Neal Kreitzinger @ 2012-02-24 16:57 UTC (permalink / raw)
To: Matt Seitz (matseitz); +Cc: git
In-Reply-To: <70952A932255A2489522275A628B97C31294E91C@xmb-sjc-233.amer.cisco.com>
On 2/23/2012 5:58 PM, Matt Seitz (matseitz) wrote:
> How can I use "git gui" to fetch a single branch from a remote
> repository?
>
> If I select Remote->Fetch From, and then a remote repository, it appears
> to always fetch all the branches from the remote repository with no
> option to select which branch I want to fetch.
>
One way you can do it is to create a remote to that branch (see -t
option of git remote manpage). Then fetching that remote only fetches
that branch. Of course, this is practical if you are interested in
fetching certain branches consistently, but not as practical for
fetching random single branches frequently.
v/r,
neal
^ permalink raw reply
* Re: measuring the % change between two commits
From: Sitaram Chamarty @ 2012-02-24 17:05 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Git Mailing List
In-Reply-To: <7vpqd52tqr.fsf@alter.siamese.dyndns.org>
On Fri, Feb 24, 2012 at 1:43 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Sitaram Chamarty <sitaramc@gmail.com> writes:
>
>> I could do a --numstat and then do a 'wc -l' on each file I guess, but
>> I was hoping to avoid that.
>>
>> --dirstat gives you a percentage but does not count the top level directory.
>
> Note that dirstat is not about "how much damage was caused to the entire
> codebase". It only measures "How is the damage this patch causes
> distributed across directories it touches". It was unclear from your "a %
> measure for the changes between two commits" which one you meant, but I am
> guessing from your "--numstat and wc -l" reference that you are asking for
> the former, e.g. we have 300,000 lines of code and between these two
> commits 10,000 lines changed, hence we updated 3% of the codebase during
> that period".
yes; I wanted an overall figure. Clearly I misunderstood dirstat
then. (Should have guessed from the "...may not total to 100%..."
comment somewhere.
Thanks
--
Sitaram
^ permalink raw reply
* Re: [BUG?] bulk checkin does not respect filters
From: Junio C Hamano @ 2012-02-24 18:48 UTC (permalink / raw)
To: Jeff King; +Cc: Junio C Hamano, git
In-Reply-To: <20120224075425.GA18688@sigill.intra.peff.net>
Jeff King <peff@peff.net> writes:
> Isn't write_entry the _other_ side of things. I.e., checking out, not
> checking in?
Ah, of course.
^ permalink raw reply
* git and SSL certificates
From: Edward Ned Harvey @ 2012-02-24 19:11 UTC (permalink / raw)
To: git
I have a git server hosted on https (github enterprise virtual appliance),
using a valid signed cert from startcom, which passes all the SSL checks for
any browser I use on any OS (IE, Firefox, Safari, Chrome, on Ubuntu, Mac
OSX, MS Win7) but when I connect to it using git, git complains about the
cert, but it's platform dependent, and it doesn't seem to make any sense...
Does git have its own set of SSL trusted root CA's compiled in at build time
or something? It seems weird that it's apparently not using the trusted
root CA's from the OS...
I have not tried re-signing my cert using a different CA. I see github uses
DigiCert. My clients do not complain about SSL cert when cloning from
github.
The test command is, simply:
git clone https://user@server.com/user/project.git
(Obviously, using a real username, a real servername, and a real project
name instead of the line above.)
** On OSX, it works no problem. This is OSX 10.7 Lion, upgraded from 10.6
SL, with 4.1 upgraded from XCode 3.2.6. Git version 1.7.4.4
** On ubuntu, oneiric x86_64, git version 1.7.5.4, it says:
error: server certificate verification failed. CAfile:
/etc/ssl/certs/ca-certificates.crt CRLfile: none while accessing
https://user@server.com/user/project.git/info/refs
fatal: HTTP request failed
This is annoying, because ... It names the location where it's searching for
the root certificates, so I thought maybe the startcom root CA wasn't in
there, so I went and looked, and confirmed it's there. Compared the actual
pem encoded root ca cert string to the one that signed my server's cert, and
it's definitely there.
On linux, users are able to workaround using GIT_SSL_NO_VERIFY=1, but that
kind of defeats the purpose. I don't want them doing this.
** On Win 7 64bit, tortoisegit 1.6.5.0 based on git 1.7.3.1, it says:
error: SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify
failed while accessing https://user@server.com/user/project.git/info/refs
fatal: HTTP request failed
Cloning into C:\workdir
I don't see any way to workaround, but haven't looked very hard for a
windows equivalent of GIT_SSL_NO_VERIFY
** On Win 7 64bit, cygwin git version 1.7.9, it says:
error: SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify
failed while accessing https://user@server.com/user/project.git/info/refs
fatal: HTTP request failed
Also, it ignores the presence of GIT_SSL_NO_VERIFY. So there isn't any
known workaround for cygwin.
^ permalink raw reply
* Re: [BugReport] git tag -a / git show
From: Zbigniew Jędrzejewski-Szmek @ 2012-02-24 19:27 UTC (permalink / raw)
To: "Romain Vimont (®om)"; +Cc: git
In-Reply-To: <b05f03b381140ca57a7d03a934f605bd@rom1v.com>
On 02/24/2012 11:24 AM, Romain Vimont (®om) wrote:
> $ git log --pretty=online
> 0ef41513d0b6d0ad28f21d0ac1da7096ad1dc6ff This is the last commit
> a4702c69c28484d357179166cf3b116764da20a4 This is a commit
>
> Now, I edit some files (for example in a config file "mock_data=true"),
> then I want to tag without commiting this change.
>
> $ git tag -a v0.1 -m 'My v0.1 with mock data'
> And it shows the diff between a4702c69c28484d357179166cf3b116764da20a4
> and 0ef41513d0b6d0ad28f21d0ac1da7096ad1dc6ff (the two last commits).
Hi Romain,
git tag attaches the tag to the last commit, 0ef41513 in your case.
Dirty changes in your tree are ignored by the tag command. You would
have to commit them first, and attach the tag to this new commit.
zbyszek
^ permalink raw reply
* Re: git and SSL certificates
From: Shawn Pearce @ 2012-02-24 19:27 UTC (permalink / raw)
To: Edward Ned Harvey; +Cc: git
In-Reply-To: <000501ccf328$1efe1070$5cfa3150$@nedharvey.com>
On Fri, Feb 24, 2012 at 11:11, Edward Ned Harvey <git@nedharvey.com> wrote:
> I have a git server hosted on https (github enterprise virtual appliance),
> using a valid signed cert from startcom, which passes all the SSL checks for
> any browser I use on any OS (IE, Firefox, Safari, Chrome, on Ubuntu, Mac
> OSX, MS Win7) but when I connect to it using git, git complains about the
> cert, but it's platform dependent, and it doesn't seem to make any sense...
> Does git have its own set of SSL trusted root CA's compiled in at build time
> or something? It seems weird that it's apparently not using the trusted
> root CA's from the OS...
Nope. Git uses the system's libcurl, which is probably using the
system's libssl or libgnutls, which is using the system's
certificates.
^ permalink raw reply
* Re: FW: question about merge in 1.7.10
From: Zbigniew Jędrzejewski-Szmek @ 2012-02-24 19:29 UTC (permalink / raw)
To: Marlene Cote; +Cc: git@vger.kernel.org
In-Reply-To: <1F026B57884A5841B330471696849DE9114503D7@MBX021-W4-CA-5.exch021.domain.local>
On 02/24/2012 03:33 PM, Marlene Cote wrote:
>
> I used merge with -no-commit -no-ff. So, I assume I won't see any change in behavior, since merge is not performing commits, right?
Hi Marlene,
--no-commit only stops the last step, i.e. making of the commit. The
tree (working files) is changed before that. So basically after a merge
with --no-commit, your tree will be dirty and git status will show
modified files.
zbyszek
^ permalink raw reply
* Re: Improving merge messages for 1.7.10 and making "pull" easier
From: Junio C Hamano @ 2012-02-24 19:39 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Git Mailing List, Thomas Rast, Linus Torvalds
In-Reply-To: <CACBZZX5UVq9k7jvSy3m+yMVj7_JbfLp8ugFWf2gGFdMz_8GPEA@mail.gmail.com>
Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
> Firstly (and as a more general thing) I think we should add a mention
> of "git merge --abort" to the message, just saving an empty file is
> not sufficient to fully clear the merge state:
Makes sense, but the new message does not quite parse.
>> "Lines starting with '#' will be ignored, and an empty message followed\n"
>> "by 'git merge --abort' the merge.\n");
Perhaps s/the merge/aborts &/ or something.
> Additionally, perhaps it would be a good idea to:
>
> * Detect if the user didn't run this explicitly but implicitly from a
> "git pull". We could pass some env var along or another option
> (e.g. --internal-from-porcelain=pull) and add this:
>
> You've merged implicitly via a "git pull", if you're just
> updating some local work in progress to keep up with upstream
> you may want to use "git pull --rebase" instead (or set the
> pull.rebase configuration variable) to rebase instead of merge.
Won't this message be given to _all_ users of "git pull", even to the ones
who already have decided correctly that "pull" is the right thing in their
situation? With a new advice.* settings to squelch it, perhaps.
> * Explicitly check if we're merging an updated upstream into the
> work-in-progress topic,...
It might be a worthy goal, but how would we detect it? A few examples
that we shouldn't give an unhelpful advice with a false positive are
merges into:
- The 'master' branch used by people who use Git as an improved CVS, when
they do an equivalent of 'cvs update'. Merging the updated 'master'
from the central repository into their 'master' that contains their
work that may or may not be ready to be pushed back is how their
project works. It is a norm for them to make such a merge, even though
more experienced people may prefer to see the history of their project
kept cleaner by suggesting their project participants to use their own
topic branches.
- Integration branches like my 'next', when it gets a merge from
'master'. This is "merging an updated upstream" but is done in order to
keep the promise that 'next' would contain everything in 'master'.
And what alternative would we offer? If we were to suggest "rebase", we
would also need to consider the topic of the other a-couple-of-days-old
thread to detect which part of history is no longer subject to rewrite.
> I work with a lot of inexperienced git users and a lot of them are
> going to be very confused by this change. I still think it's a good
> change to make, but we could do a lot more to mitigate the inevitable
> confusion.
What exact change are you talking about with "this change"? Earlier you
had a chance to edit the merge log only when it needed your help resolving
(hence you did a separate "git commit" to record it) but you had to "git
commit --amend" (or start with "git merge --no-commit") to edit the merge
log if it did not need any help resolving conflicts, but now you do not
have to. Is that the change you have in mind?
I would like to know how that would lead to an "inevitable confusion".
Admittedly, the original without any "# Please do X" comment, the user may
wonder what is being asked of him when he sees the editor for the first
time, but I thought Thomas's patch took care of that issue.
> One thing that would help these users in particular would be to have
> some easy to use replacement for their frequent use of "git
> pull".
After this part, I think you shifted into a different topic.
I have mixed feelings about "rebase your unpublished work and keep it
always a descendant of the upstream" workflow you seem to be advocating.
It _might_ deserve a bit more visibility, but I do not think rewording
this message done during "merge" is the place to do so.
> They don't often commit their work (because of git inexperience) so
> rebasing will error out because the tree is unclean.
That is a *good* thing, isn't it? There lies the perfect opportunity for
them to train their fingers to commit first and then rebase.
^ permalink raw reply
* RE: FW: question about merge in 1.7.10
From: Marlene Cote @ 2012-02-24 19:33 UTC (permalink / raw)
To: Zbigniew Jędrzejewski-Szmek; +Cc: git@vger.kernel.org
In-Reply-To: <4F47E51B.6080401@in.waw.pl>
I know that the tree will be dirty. The change to git in 1.7.10 says that the merge will require a commit message when the command is run. Since I am not committing, I need to supply a message, but only later when I commit. Not when I execute the merge command. This means my scripts don't have to change.
-----Original Message-----
From: Zbigniew Jędrzejewski-Szmek [mailto:zbyszek@in.waw.pl]
Sent: Friday, February 24, 2012 2:30 PM
To: Marlene Cote
Cc: git@vger.kernel.org
Subject: Re: FW: question about merge in 1.7.10
On 02/24/2012 03:33 PM, Marlene Cote wrote:
>
> I used merge with -no-commit -no-ff. So, I assume I won't see any change in behavior, since merge is not performing commits, right?
Hi Marlene,
--no-commit only stops the last step, i.e. making of the commit. The tree (working files) is changed before that. So basically after a merge with --no-commit, your tree will be dirty and git status will show modified files.
zbyszek
^ permalink raw reply
* Re: [Not A BugReport] git tag -a / git show
From: Junio C Hamano @ 2012-02-24 19:50 UTC (permalink / raw)
To: Romain Vimont (®om); +Cc: git
In-Reply-To: <b05f03b381140ca57a7d03a934f605bd@rom1v.com>
"Romain Vimont (®om)" <rom@rom1v.com> writes:
> Now, I edit some files (for example in a config file
> "mock_data=true"), then I want to tag without commiting this change.
Tag applies to an existing commit [*1*]. Your change in the working tree
is purely ephemeral until it is committed.
In other words, you don't "tag without committing".
> $ git tag -a v0.1 -m 'My v0.1 with mock data'
By omitting the [<head>] part from your command line for a command whose
usage is:
usage: git tag [-a|-s|-u <key-id>] [-f] [-m <msg>|-F <file>] <tagname> [<head>]
you asked <head> to default to HEAD, the most recent commit, so the tag
points at your 0ef41513d0b6 (This is the last commit). The tag message
should say "My v0.1" without anything else.
And show naturally shows the patch to bring its parent to that tagged
commit.
If you wanted to keep your mainline pristine without mock data, and want
to have a playpen that uses mock data, a way to do so is to use a separate
branch, e.g.
$ git checkout -b playpen
Now, you are on your 'playpen' branch that was forked from the tip of
whatever branch you were on, perhaps 'master'. Then commit that state
with whatever change that is specific to the playpen you want to keep out
of the mainline:
$ edit config.txt ;# set mock_data=true
$ git commit -a -m 'With mock data'
You can optionally tag the resulting commit if you want to. You are still
on the 'playpen' branch, so you probably would want to come back to the
previous branch after you are done.
[Footnote]
*1* technically, tag can apply to any type of object, but it is most
common to apply to a commit.
^ permalink raw reply
* Re: [BugReport] git tag -a / git show
From: Romain Vimont (®om) @ 2012-02-24 19:55 UTC (permalink / raw)
To: Zbigniew Jędrzejewski-Szmek; +Cc: git
In-Reply-To: <4F47E48D.4080501@in.waw.pl>
Thank you for your answer.
After my message this morning, that's what I did: I commited with the
mock data then tag.
Tonight, I just tried something which do exactly what I wanted to do
this morning:
$ git checkout -b temp
$ git commit -a -m 'My config file with mock_data=true'
$ git tag -a v0.1 -m v0.1
$ git checkout master
$ git branch -D temp
With these commands, the tag is associated to a commit which is not in
any branch.
Regards,
©om
Le vendredi 24 février 2012 à 20:27 +0100, Zbigniew Jędrzejewski-Szmek a
écrit :
> On 02/24/2012 11:24 AM, Romain Vimont (®om) wrote:
> > $ git log --pretty=online
> > 0ef41513d0b6d0ad28f21d0ac1da7096ad1dc6ff This is the last commit
> > a4702c69c28484d357179166cf3b116764da20a4 This is a commit
> >
> > Now, I edit some files (for example in a config file "mock_data=true"),
> > then I want to tag without commiting this change.
> >
> > $ git tag -a v0.1 -m 'My v0.1 with mock data'
>
> > And it shows the diff between a4702c69c28484d357179166cf3b116764da20a4
> > and 0ef41513d0b6d0ad28f21d0ac1da7096ad1dc6ff (the two last commits).
>
> Hi Romain,
> git tag attaches the tag to the last commit, 0ef41513 in your case.
> Dirty changes in your tree are ignored by the tag command. You would
> have to commit them first, and attach the tag to this new commit.
>
> zbyszek
>
^ permalink raw reply
* Re: [Not A BugReport] git tag -a / git show
From: Romain Vimont (®om) @ 2012-02-24 19:58 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vsji0yprw.fsf@alter.siamese.dyndns.org>
Thank you for the details.
> *1* technically, tag can apply to any type of object, but it is most
common to apply to a commit.
To what other type of object can you apply a tag ?
Good evening.
Le vendredi 24 février 2012 à 11:50 -0800, Junio C Hamano a écrit :
> "Romain Vimont (®om)" <rom@rom1v.com> writes:
>
> > Now, I edit some files (for example in a config file
> > "mock_data=true"), then I want to tag without commiting this change.
>
> Tag applies to an existing commit [*1*]. Your change in the working tree
> is purely ephemeral until it is committed.
>
> In other words, you don't "tag without committing".
>
> > $ git tag -a v0.1 -m 'My v0.1 with mock data'
>
> By omitting the [<head>] part from your command line for a command whose
> usage is:
>
> usage: git tag [-a|-s|-u <key-id>] [-f] [-m <msg>|-F <file>] <tagname> [<head>]
>
> you asked <head> to default to HEAD, the most recent commit, so the tag
> points at your 0ef41513d0b6 (This is the last commit). The tag message
> should say "My v0.1" without anything else.
>
> And show naturally shows the patch to bring its parent to that tagged
> commit.
>
> If you wanted to keep your mainline pristine without mock data, and want
> to have a playpen that uses mock data, a way to do so is to use a separate
> branch, e.g.
>
> $ git checkout -b playpen
>
> Now, you are on your 'playpen' branch that was forked from the tip of
> whatever branch you were on, perhaps 'master'. Then commit that state
> with whatever change that is specific to the playpen you want to keep out
> of the mainline:
>
> $ edit config.txt ;# set mock_data=true
> $ git commit -a -m 'With mock data'
>
> You can optionally tag the resulting commit if you want to. You are still
> on the 'playpen' branch, so you probably would want to come back to the
> previous branch after you are done.
>
>
> [Footnote]
>
> *1* technically, tag can apply to any type of object, but it is most
> common to apply to a commit.
>
^ permalink raw reply
* Re: FW: question about merge in 1.7.10
From: Junio C Hamano @ 2012-02-24 20:00 UTC (permalink / raw)
To: Marlene Cote; +Cc: git@vger.kernel.org
In-Reply-To: <1F026B57884A5841B330471696849DE9114503D7@MBX021-W4-CA-5.exch021.domain.local>
Marlene Cote <Marlene_Cote@affirmednetworks.com> writes:
> I used merge with -no-commit -no-ff. So, I assume I won't see any
> change in behavior, since merge is not performing commits, right?
The _only_ difference is when
(1) "git merge" is run interactively;
(2) the merge goes cleanly that it does not need to ask help from the
user to resolve conflicts; and
(3) it records the result by creating a commit.
Because your use case with "--no-commit" does not satisfy the last
criteria, I do not expect you to see any difference. Otherwise you may
have found a bug.
As noted in
http://git-blame.blogspot.com/2012/02/anticipating-git-1710.html
please try out the version from the 'master' branch before it gets
released, so that you can help us avoid surprises in the corner cases.
Thanks.
^ permalink raw reply
* RE: git and SSL certificates
From: Edward Ned Harvey @ 2012-02-24 20:01 UTC (permalink / raw)
To: 'Shawn Pearce'; +Cc: git
In-Reply-To: <CAJo=hJuyHv_L_zajW-MWEj6fDGggrVDkRWT32mD5TBzD_GzReQ@mail.gmail.com>
> From: Shawn Pearce [mailto:spearce@spearce.org]
> Sent: Friday, February 24, 2012 2:27 PM
>
> Nope. Git uses the system's libcurl, which is probably using the
> system's libssl or libgnutls, which is using the system's
> certificates.
Thanks, this gives me more fuel to go on, because now I know I can reproduce
the problem using any other tool I want - curl for example. Where I'm able
to specify -v for verbose, and get its cert search path.
It's still really bizarre, because the Startcom root CA is indeed present in
the search path, and it is indeed the same root CA that was used to sign my
server cert. So now I'll go ask startcom what they think about it...
If anyone is interested, please say so, and I'll report back here.
Otherwise, I'll probably just let the thread die.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox