* Re: [BUG] multi-commit cherry-pick messes up the order of commits
From: Jonathan Nieder @ 2012-01-12 20:11 UTC (permalink / raw)
To: Jeff King
Cc: Ramkumar Ramachandra, Junio C Hamano, SZEDER Gábor,
Christian Couder, Christian Couder, git
In-Reply-To: <20120112194710.GA28148@sigill.intra.peff.net>
Jeff King wrote:
> I am tempted to suggest
[...]
> That would make all of these work as most people would
> expect:
>
> git cherry-pick A B C
> git cherry-pick A..B
> git cherry-pick A..B B..C
>
> but would be a regression for:
>
> git cherry-pick B ^A
>
> versus the current code. I suspect that the latter form is not all that
> commonly used, though, and certainly I would accept it as a casualty of
> making the "A B C" form work. My only hesitation is that it is in fact a
> regression.
I find myself using such complicated expressions as
list-revs-to-skip |
xargs git cherry-pick --cherry-pick --right-only HEAD...topic --not
so yeah, that would be a pretty serious loss in functionality.
However, moving to something like the far future semantics that Junio
hinted at, for cherry-pick/revert and other --no-walk style commands
only, would not be a regression for me. The multi-pick feature is
still young, and I _suspect_ changing the meaning of A..B B..C for it
would not inconvenience anybody.
I would even welcome a change in the meaning of B ^A: the most
intuitive thing for it to do would be to cherry-pick the single commit B
when and only when it is not an ancestor of A.
Jonathan
^ permalink raw reply
* Re: [BUG] multi-commit cherry-pick messes up the order of commits
From: Junio C Hamano @ 2012-01-12 20:11 UTC (permalink / raw)
To: Ramkumar Ramachandra
Cc: Jeff King, SZEDER Gábor, Christian Couder, Christian Couder,
git, Jonathan Nieder
In-Reply-To: <CALkWK0kk0mVNaetr=triuVYva7inyx2aZvam81qTVA9=Q=UzGw@mail.gmail.com>
Ramkumar Ramachandra <artagnon@gmail.com> writes:
> Okay, just to make sure I understand this correctly: if more than one
> argument is literally specified, I should not set up the revision
> walker and pick the commits listed in revs->pending, correct?
Not really.
A rough approximation would be that if you see any negative ones (either
coming from A..B or ^A), you would always want to walk, giving everything
to prepare_revision_walk()-and-then-get_revision() machiery.
Otherwise you have only zero [*1*] or more positive ones, and you pick
them in the order you find in the pending list, without bothering the
revision traversal machinery at all [*2*].
> when I encounter the following command,
>
> $ git cherry-pick maint ^master
>
> I should just pick two commits: maint, and ^master.
So the answer is aboslutely no. "maint ^master" is the same as saying
"master..maint".
[Footnote]
*1* You would probably want to error out if you got zero positive ones in
this case (i.e. absolutely nothing was given, neither positive nor
negative).
*2* The reason this is "rough" approximation is because we would probably
want to do Jonathan's "maint..master master..next" someday, and when that
happens, we would need to do much more than "do we have any negative? then
send it through to prepare_revision_walk()". But we are not there yet, so
I think the above is actually more or less the complete implementation.
^ permalink raw reply
* Re: [BUG] multi-commit cherry-pick messes up the order of commits
From: Jeff King @ 2012-01-12 20:17 UTC (permalink / raw)
To: Jonathan Nieder
Cc: Ramkumar Ramachandra, Junio C Hamano, SZEDER Gábor,
Christian Couder, Christian Couder, git
In-Reply-To: <20120112201122.GE6038@burratino>
On Thu, Jan 12, 2012 at 02:11:22PM -0600, Jonathan Nieder wrote:
> > I am tempted to suggest
> [...]
> > That would make all of these work as most people would
> > expect:
> >
> > git cherry-pick A B C
> > git cherry-pick A..B
> > git cherry-pick A..B B..C
> >
> > but would be a regression for:
> >
> > git cherry-pick B ^A
> >
> > versus the current code. I suspect that the latter form is not all that
> > commonly used, though, and certainly I would accept it as a casualty of
> > making the "A B C" form work. My only hesitation is that it is in fact a
> > regression.
>
> I find myself using such complicated expressions as
>
> list-revs-to-skip |
> xargs git cherry-pick --cherry-pick --right-only HEAD...topic --not
>
> so yeah, that would be a pretty serious loss in functionality.
That's gross. :)
But thank you for providing a real-world example. I had a vague notion
that the full power of the revision parser was not actually useful to
people, but clearly not.
OTOH, if cherry-pick were more simplistic, you could perhaps get by
with:
list-revs-to-skip |
xargs git rev-list --cherry-pick --right-only HEAD...topic --not |
git cherry-pick --stdin
-Peff
^ permalink raw reply
* Re: [PATCH v2 2/3] index-pack: eliminate recursion in find_unresolved_deltas
From: Junio C Hamano @ 2012-01-12 20:32 UTC (permalink / raw)
To: Nguyen Thai Ngoc Duy; +Cc: git, Shawn O. Pearce
In-Reply-To: <CACsJy8Cz-qWs2wrOYTjDMPjJH0wRQCFy9u6OFVPzn6YV0a6WaQ@mail.gmail.com>
Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:
> 2012/1/10 Junio C Hamano <gitster@pobox.com>:
>> Nguyễn Thái Ngọc Duy <pclouds@gmail.com> writes:
>>
>>> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
>>
>> I find both the original and the updated code rather dense to read without
>> annotation, but from a cursory look all changes look good.
>
> Maybe I stared at it for too long it seems obvious to me (hence no
> further description in commit message). Let me describe it (and put in
> commit message later if it makes sense)
>
> Current code already links all bases together in a form of tree, using
> struct base_data, with prev_base pointer to point to parent node. The
> only problem is that struct base_data is all allocated on stack. So we
> need to put all on heap (parse_pack_objects and
> fix_unresolved_deltas). After that, it's simple depth-first traversal
> where each node also maintains its own state (ofs and ref indices to
> iterate over all children nodes).
>
> So we process one node:
>
> - if it returns a new (child) node (a parent base), we link it to our
> tree, then process the new node.
> - if it returns nothing, the node is done, free it. We go back to
> parent node and resume whatever it's doing.
>
> and do it until we have no nodes to process.
If you have the current path (base to another that is recorded as a delta
to it to yet another that is recorded as a delta to that delitified
object) on the stack, it is obvious that as you have done with the objects
on the deeper end of the delta chain, the data that becomes unnecessary
will be gone by simply returning from the recursion, but if you "put all
on heap", you would have to do the same freeing as part of the hand-rolled
recursion. It is unclear if, where and how the patch takes care of that
in the above.
Other than that, I find the description very readable.
Thanks.
^ permalink raw reply
* Re: [PATCH/RFC] Restore line limit option in post-receive-email
From: Cheng Leong @ 2012-01-12 21:01 UTC (permalink / raw)
To: git
In-Reply-To: <CACor6wGkDPSkA_G117YtnsQA7H_cROKKW69m-jW_rb2SrVWo1w@mail.gmail.com>
> Would you like me to reroll a patch with both or is this a trivial fixup?
Bump. What do I need to do to contribute this patch?
Thanks,
Cheng
^ permalink raw reply
* Re: SVN -> Git *but* with special changes
From: Abscissa @ 2012-01-12 21:52 UTC (permalink / raw)
To: git
In-Reply-To: <1326065910362-7166084.post@n2.nabble.com>
I'm trying to do it manually with git-svn, but I'm getting the same problem.
I have two different SVN repos I want to do (turns out neither have
branches, I had thought they did, but one of them does have tags). Doing
this:
git svn clone {the url to the SVN repo} --prefix=svn/ --preserve-empty-dirs
--authors-file={already prepared authors file} --trunk=trunk --tags=tags
It gets partway through, and then gives me:
Failed to strip path '{some path}/.gitignore' ((?-xism:^trunk(/|$)))
Where {some path} is an empty dir in the trunk of one repo, and a completely
non-existent path in the other. (In both cases I'm looking at the revision
git-svn had gotten to when bailing, plus the revision before and the
revision after).
I don't have a clue what that error message is trying to tell me or what is
going wrong. :(
If it helps, these are the repos I'm trying to convert:
http://svn.dsource.org/projects/semitwist (git svn clone fails at r46 (out
of 242) on 'src/nonFatalAssertTest/.gitignore')
http://svn.dsource.org/projects/goldie (git svn clone fails at r85 (out of
557) on 'bin/lang/.gitignore')
(Yes, they do have a non-standard top-level "downloads" directory, but
that's just how that host does file downloads, and I *don't* need to
preserve it)
--
View this message in context: http://git.661346.n2.nabble.com/SVN-Git-but-with-special-changes-tp6840904p7181897.html
Sent from the git mailing list archive at Nabble.com.
^ permalink raw reply
* thin packs ending up fat
From: Jeff King @ 2012-01-12 22:15 UTC (permalink / raw)
To: git; +Cc: Nicolas Pitre
[This ended up long; the gist of it is that we often fail to find thin
deltas, making small fetches much bigger than they could be. Read on
for details].
Recently I was doing some work related to bundles containing thin packs,
and I wanted to generate some thin packs. Much to my surprise, all of my
packs ended up having no thin objects!
It turns out that when packing a subset of a fully packed repo (as we do
for a bundle or for a fetch), we tend not to make thin packs at all.
The culprit is this logic in try_delta:
/*
* We do not bother to try a delta that we discarded
* on an earlier try, but only when reusing delta data.
*/
if (reuse_delta && trg_entry->in_pack &&
trg_entry->in_pack == src_entry->in_pack &&
trg_entry->in_pack_type != OBJ_REF_DELTA &&
trg_entry->in_pack_type != OBJ_OFS_DELTA)
return 0;
This does not try to delta objects which are found in the same pack,
under the assumption that we would have tried them last time and found
that they didn't work. For a complete repack, it works well. We don't
miss useful deltas, and it saves a lot of CPU time.
For a thin pack, though, it means we'll typically fail to actually use
the boundary objects. I think what happens is this:
1. You start at commit C which contains a file with content A. You
build commit C', which contains content A'.
2. You repack the repository, putting A and A' in the pack. Our delta
rules tend to create deltas in reverse-recency order, so A
typically is stored as a delta based on A', and A' is stored
whole.
3. You try to create a thin pack (e.g., with "git bundle create
foo.bundle C..C'"). We know that we want to send A', and that we
can assume the other side has A to use as a base in our thin pack.
But the logic above says "A and A' are both in the same pack, and
we didn't find a delta for A'. They must not be a good match". But
in this case it is because our deltas go in the reverse direction,
and we never tried them.
So I did a little experimenting (scripts and inputs are at the end of
the email). I generated a thin pack for a subset of objects from my
fully packed repo, measuring the size of the resulting pack and the CPU
time used. I did this both with stock git, and a version of git with the
above optimization removed.
I used two datasets to generate the input to pack-objects. In the first,
I used the reflog entries from my refs/remotes/origin/master ref. That
simulates the workload of upload-packs performed for me by kernel.org
over the last month or so. In the second dataset, I took packed the
objects between git v1.0.0 and v1.1.0, between v1.1.0 and v1.2.0, and so
on. This was an attempt to simulate somebody who fetches much less
frequently, or perhaps somebody who generates bundles to sneaker-net
individual versions.
For each dataset, I calculated the average size (in bytes) over all of
the generated packs in the dataset and the average CPU time used per
pack (in seconds).
dataset
| fetches | tags
---------------------------------
before | 53358 | 2750977
size after | 32315 | 2652939
change | -39% | -4%
---------------------------------
before | 0.18 | 1.12
CPU after | 0.19 | 1.50
change | +5% | +33%
For the case of small, frequent packs, we saved a lot of space for a
little bit of extra CPU. But for larger packs, our savings were much
more limited, and we spent a lot more CPU time. That makes sense; this
is really beneficial for finding extra matches at the boundaries, and
for cases where you can avoid sending an entire undeltified object at
all for a given path.
So I think this is worth pursuing as an optimization to save bandwidth
on transfers, but it would require figuring out the right times to turn
it on. My initial idea was that we would want it off whenever we were
doing a thin pack. But the "tags" dataset shows that it's probably not
worth the trouble for large packs due to the extra CPU time.
I have a feeling that the case it is generally catching is this "reverse
delta" case. I.e., A is based off of B, we are including B, but A is
available as a thin base. But that can probably be extended into a delta
chain (A is based from B which is based on C; we want to include B and
C, but can build off of A as a base. We'll keep the delta from B to C,
but we would want to delta C off of A). So detecting that they are part
of the same chain might require walking the chains.
Maybe it is enough to simply turn off this optimization if the potential
delta source is not being included in the pack (i.e., we are using
--thin and it is a boundary object). Because if both objects are being
sent, we will just end up reusing the delta that goes in the reverse
direction anyway.
Or maybe there is some other clever way of detecting this situation. I'm
open to suggestions.
-Peff
---
This is the test script I used (feed the dataset files on stdin, get
lines of sizes and CPU times on stdout):
-- >8 --
#!/bin/sh
while read revs; do
for i in $revs; do
echo $i
done >input
echo >&2 "==> $revs"
time -o time.out -f "%e" \
git pack-objects --thin --revs --stdout <input >pack.out
echo "`stat --format=%s pack.out` `cat time.out`"
done
-- 8< --
and the "fetch" dataset:
-- >8 --
8b0e15fa95e11965f18c8d2585dc8ffd9bfc9356 ^7f41b6bbe3181dc4d1687db036bf22316997a1bf
34c4461ae3b353e8c931565d5527b98ed12e3735 ^8b0e15fa95e11965f18c8d2585dc8ffd9bfc9356
463b0ea22b5b9a882e8140d0308433d8cbd0d1fe ^34c4461ae3b353e8c931565d5527b98ed12e3735
288396994f077eec7e7db0017838a5afbfbf81e3 ^463b0ea22b5b9a882e8140d0308433d8cbd0d1fe
05f6edcd2a418a88eeb953d51408a6aeef312f5c ^288396994f077eec7e7db0017838a5afbfbf81e3
08cfdbb88cd6225b4fc4b8a3cecd0e01758c835d ^05f6edcd2a418a88eeb953d51408a6aeef312f5c
87009edcbd0b4987ccb7ba050a1efe368a315753 ^08cfdbb88cd6225b4fc4b8a3cecd0e01758c835d
8963314c77af9a4eda5dcbdbab3d4001af83ad81 ^87009edcbd0b4987ccb7ba050a1efe368a315753
10b2a48113b8ab6b8f48229eb40fc3637ce025ae ^8963314c77af9a4eda5dcbdbab3d4001af83ad81
997a1946a55cafb992c4ba8e5e0795aa73f5a4a9 ^10b2a48113b8ab6b8f48229eb40fc3637ce025ae
e8e1c29021da446d0c50573ef9619bf74f515c20 ^997a1946a55cafb992c4ba8e5e0795aa73f5a4a9
87bf9a7048c623b3567f612ca3e67a0d412fc83d ^e8e1c29021da446d0c50573ef9619bf74f515c20
ee6dfb2d83ba1b057943e705f707fa27e34e47f9 ^87bf9a7048c623b3567f612ca3e67a0d412fc83d
4cb6764227173a6483edbdad09121651bc0b01c3 ^ee6dfb2d83ba1b057943e705f707fa27e34e47f9
8a042478967679b0dab3137f5f18367a0ffdc48a ^4cb6764227173a6483edbdad09121651bc0b01c3
248dbbe83256202f0edd6e1468d01cfbe27fd733 ^8a042478967679b0dab3137f5f18367a0ffdc48a
bc1bbe0c19a6ff39522b4fa3259f34150e308e1f ^248dbbe83256202f0edd6e1468d01cfbe27fd733
4d2440fe0daa9ad1556dfd220af8b3a883cf849d ^bc1bbe0c19a6ff39522b4fa3259f34150e308e1f
f56ef114eeefee755f422e6cbef2d83974cb90f1 ^4d2440fe0daa9ad1556dfd220af8b3a883cf849d
e14d63198867c545d0662afc00bf7be048bf2231 ^f56ef114eeefee755f422e6cbef2d83974cb90f1
017d1e134545db0d162908f3538077eaa1f34fb6 ^e14d63198867c545d0662afc00bf7be048bf2231
fc14b89a7e6899b8ac3b5751ec2d8c98203ea4c2 ^017d1e134545db0d162908f3538077eaa1f34fb6
406da7803217998ff6bf5dc69c55b1613556c2f4 ^fc14b89a7e6899b8ac3b5751ec2d8c98203ea4c2
7e02a6c63a183270b726bb21640059ae16fa48ae ^406da7803217998ff6bf5dc69c55b1613556c2f4
4cb5d10b14dcbe0155bed9c45ccb94e83bd4c599 ^7e02a6c63a183270b726bb21640059ae16fa48ae
9859a023fef30ffebdd22ad9639c587ac720b8b6 ^4cb5d10b14dcbe0155bed9c45ccb94e83bd4c599
57526fde5df201a99afa6d122c3266b3a1c5673a ^9859a023fef30ffebdd22ad9639c587ac720b8b6
73c6b3575bc638b7096ec913bd91193707e2265d ^57526fde5df201a99afa6d122c3266b3a1c5673a
10f4eb652ee4e592f91f638e579d1afcb96c0408 ^73c6b3575bc638b7096ec913bd91193707e2265d
ee228024933069b93ce23a1bd5eeb7ae12c792f2 ^10f4eb652ee4e592f91f638e579d1afcb96c0408
d16520499d2652b5b59dfb25f9cf2d56a4c6913a ^ee228024933069b93ce23a1bd5eeb7ae12c792f2
876a6f4991abdd72ea707b193b4f2b831096ad3c ^d16520499d2652b5b59dfb25f9cf2d56a4c6913a
8d68493f20db71abeb77adc251b2a116fe45cdaa ^876a6f4991abdd72ea707b193b4f2b831096ad3c
3daff7c31998faedbe0dd7e2b8651e351be40d64 ^8d68493f20db71abeb77adc251b2a116fe45cdaa
e443bdfe1e8e1ef8b3665cfd1c1295bd73e13773 ^3daff7c31998faedbe0dd7e2b8651e351be40d64
5d6dfc7cb140a6eb90138334fab2245b69bc8bc4 ^e443bdfe1e8e1ef8b3665cfd1c1295bd73e13773
ec330158ec04849fe5ff2cb8749797cd63ae592b ^5d6dfc7cb140a6eb90138334fab2245b69bc8bc4
17b4e93d5b849293e6a3659bbc4075ed8a6e97e2 ^ec330158ec04849fe5ff2cb8749797cd63ae592b
4570aeb0d85f3b5ff274b6d5a651c2ee06d25d76 ^17b4e93d5b849293e6a3659bbc4075ed8a6e97e2
247f9d23da8cfd255533433ad2aa07d172afac0b ^4570aeb0d85f3b5ff274b6d5a651c2ee06d25d76
eac2d83247ea0a265d923518c26873bb12c33778 ^247f9d23da8cfd255533433ad2aa07d172afac0b
beecc7ab65b31c5471331e64acaa3f722125ea67 ^eac2d83247ea0a265d923518c26873bb12c33778
7e521640c80b4bb871bca7a9259621a7abb303e7 ^beecc7ab65b31c5471331e64acaa3f722125ea67
0e1cfc52de002e2d9b0e6562e8672fee3bf45a67 ^7e521640c80b4bb871bca7a9259621a7abb303e7
-- 8< --
and the "tags" dataset:
-- >8 --
v1.1.0 ^v1.0.0
v1.2.0 ^v1.1.0
v1.3.0 ^v1.2.0
v1.4.0 ^v1.3.0
v1.5.0 ^v1.4.0
v1.6.0 ^v1.5.0
v1.7.0 ^v1.6.0
-- 8< --
^ permalink raw reply
* Re: thin packs ending up fat
From: Jeff King @ 2012-01-12 22:32 UTC (permalink / raw)
To: git; +Cc: Nicolas Pitre
In-Reply-To: <20120112221523.GA3663@sigill.intra.peff.net>
On Thu, Jan 12, 2012 at 05:15:23PM -0500, Jeff King wrote:
> It turns out that when packing a subset of a fully packed repo (as we do
> for a bundle or for a fetch), we tend not to make thin packs at all.
> The culprit is this logic in try_delta:
>
> /*
> * We do not bother to try a delta that we discarded
> * on an earlier try, but only when reusing delta data.
> */
> if (reuse_delta && trg_entry->in_pack &&
> trg_entry->in_pack == src_entry->in_pack &&
> trg_entry->in_pack_type != OBJ_REF_DELTA &&
> trg_entry->in_pack_type != OBJ_OFS_DELTA)
> return 0;
> [...]
> Maybe it is enough to simply turn off this optimization if the potential
> delta source is not being included in the pack (i.e., we are using
> --thin and it is a boundary object). Because if both objects are being
> sent, we will just end up reusing the delta that goes in the reverse
> direction anyway.
Hmm. It turns out this is really easy, because we have already marked
such objects as preferred bases.
So with this patch:
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 96c1680..d05e228 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1439,6 +1439,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,
*/
if (reuse_delta && trg_entry->in_pack &&
trg_entry->in_pack == src_entry->in_pack &&
+ !src_entry->preferred_base &&
trg_entry->in_pack_type != OBJ_REF_DELTA &&
trg_entry->in_pack_type != OBJ_OFS_DELTA)
return 0;
here are the numbers I get:
dataset
| fetches | tags
---------------------------------
before | 53358 | 2750977
size after | 32398 | 2668479
change | -39% | -3%
---------------------------------
before | 0.18 | 1.12
CPU after | 0.18 | 1.15
change | +0% | +3%
So nearly all of the size benefit, but very little CPU change (even the
3% on the larger-pack case is close to the levels of run-to-run noise).
Obviously the size benefit in the larger-pack case isn't impressive, but
I think the "fetches" case is much more indicative of a real server
load.
-Peff
^ permalink raw reply related
* Re: thin packs ending up fat
From: Nicolas Pitre @ 2012-01-12 23:54 UTC (permalink / raw)
To: Jeff King; +Cc: git
In-Reply-To: <20120112223234.GA4949@sigill.intra.peff.net>
On Thu, 12 Jan 2012, Jeff King wrote:
> On Thu, Jan 12, 2012 at 05:15:23PM -0500, Jeff King wrote:
>
> > It turns out that when packing a subset of a fully packed repo (as we do
> > for a bundle or for a fetch), we tend not to make thin packs at all.
> > The culprit is this logic in try_delta:
> >
> > /*
> > * We do not bother to try a delta that we discarded
> > * on an earlier try, but only when reusing delta data.
> > */
> > if (reuse_delta && trg_entry->in_pack &&
> > trg_entry->in_pack == src_entry->in_pack &&
> > trg_entry->in_pack_type != OBJ_REF_DELTA &&
> > trg_entry->in_pack_type != OBJ_OFS_DELTA)
> > return 0;
> > [...]
> > Maybe it is enough to simply turn off this optimization if the potential
> > delta source is not being included in the pack (i.e., we are using
> > --thin and it is a boundary object). Because if both objects are being
> > sent, we will just end up reusing the delta that goes in the reverse
> > direction anyway.
>
> Hmm. It turns out this is really easy, because we have already marked
> such objects as preferred bases.
That's exactly what I was about to suggest after reading your first
email.
> So with this patch:
>
> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> index 96c1680..d05e228 100644
> --- a/builtin/pack-objects.c
> +++ b/builtin/pack-objects.c
> @@ -1439,6 +1439,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,
> */
> if (reuse_delta && trg_entry->in_pack &&
> trg_entry->in_pack == src_entry->in_pack &&
> + !src_entry->preferred_base &&
> trg_entry->in_pack_type != OBJ_REF_DELTA &&
> trg_entry->in_pack_type != OBJ_OFS_DELTA)
> return 0;
Acked-by: Nicolas Pitre <nico@fluxnic.net>
> here are the numbers I get:
>
> dataset
> | fetches | tags
> ---------------------------------
> before | 53358 | 2750977
> size after | 32398 | 2668479
> change | -39% | -3%
> ---------------------------------
> before | 0.18 | 1.12
> CPU after | 0.18 | 1.15
> change | +0% | +3%
>
> So nearly all of the size benefit, but very little CPU change (even the
> 3% on the larger-pack case is close to the levels of run-to-run noise).
> Obviously the size benefit in the larger-pack case isn't impressive, but
> I think the "fetches" case is much more indicative of a real server
> load.
Indeed. Please make sure to capture those numbers in the commit log.
Nicolas
^ permalink raw reply
* Re: thin packs ending up fat
From: Junio C Hamano @ 2012-01-13 0:14 UTC (permalink / raw)
To: Jeff King; +Cc: git, Nicolas Pitre
In-Reply-To: <20120112223234.GA4949@sigill.intra.peff.net>
Jeff King <peff@peff.net> writes:
> Hmm. It turns out this is really easy, because we have already marked
> such objects as preferred bases.
>
> So with this patch:
>
> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> index 96c1680..d05e228 100644
> --- a/builtin/pack-objects.c
> +++ b/builtin/pack-objects.c
> @@ -1439,6 +1439,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,
> */
> if (reuse_delta && trg_entry->in_pack &&
> trg_entry->in_pack == src_entry->in_pack &&
> + !src_entry->preferred_base &&
> trg_entry->in_pack_type != OBJ_REF_DELTA &&
> trg_entry->in_pack_type != OBJ_OFS_DELTA)
> return 0;
Yeah, I was wondering why this one-liner was not in the original message
;-)
> here are the numbers I get:
>
> dataset
> | fetches | tags
> ---------------------------------
> before | 53358 | 2750977
> size after | 32398 | 2668479
> change | -39% | -3%
> ---------------------------------
> before | 0.18 | 1.12
> CPU after | 0.18 | 1.15
> change | +0% | +3%
Looks good.
^ permalink raw reply
* Re: thin packs ending up fat
From: Junio C Hamano @ 2012-01-13 1:31 UTC (permalink / raw)
To: Jeff King; +Cc: git, Nicolas Pitre
In-Reply-To: <20120112223234.GA4949@sigill.intra.peff.net>
From: Jeff King <peff@peff.net>
Subject: [PATCH] thin-pack: try harder to create delta against preferred base
When creating a pack using objects that reside in existing packs, we try
to avoid recomputing futile delta between an object (trg) and a candidate
for its base object (src) if they are stored in the same packfile, and trg
is not recorded as a delta already. This heuristics makes sense because it
is likely that we tried to express trg as a delta based on src but it did
not produce a good delta when we created the existing pack.
As the pack heuristics prefer producing delta to remove data, and Linus's
law dictates that the size of a file grows over time, we tend to record
the newest version of the file as inflated, and older ones as delta
against it.
When creating a thin-pack to transfer recent history, it is likely that we
will try to send an object that is recorded in full, as it is newer. But
the heuristics to avoid recomputing futile delta effectively forbids us
from attempting to express such an object as a delta based on another
object. Sending an object in full is often more expensive than sending a
suboptimal delta based on other objects, and it is even more so if we
could use an object we know the receiving end already has (i.e. referred
base object) as the delta base.
Tweak the recomputation avoidance logic, so that we do not punt on
computing delta against a preferred base object.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
builtin/pack-objects.c | 9 +++++++--
1 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index c6e2d87..8bfe3a6 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1248,11 +1248,16 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,
return -1;
/*
- * We do not bother to try a delta that we discarded
- * on an earlier try, but only when reusing delta data.
+ * We do not bother to try a delta that we discarded on an
+ * earlier try, but only when reusing delta data. Note that
+ * src_entry that is marked as the preferred_base should always
+ * be considered, as even if we produce a suboptimal delta against
+ * it, we will still save the transfer cost, as we already know
+ * the other side has it and we won't send src_entry at all.
*/
if (reuse_delta && trg_entry->in_pack &&
trg_entry->in_pack == src_entry->in_pack &&
+ !src_entry->preferred_base &&
trg_entry->in_pack_type != OBJ_REF_DELTA &&
trg_entry->in_pack_type != OBJ_OFS_DELTA)
return 0;
--
1.7.9.rc0.53.gc8bc2
^ permalink raw reply related
* Re: thin packs ending up fat
From: Jeff King @ 2012-01-13 1:51 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, Nicolas Pitre
In-Reply-To: <7vwr8wz8u9.fsf@alter.siamese.dyndns.org>
On Thu, Jan 12, 2012 at 05:31:42PM -0800, Junio C Hamano wrote:
> From: Jeff King <peff@peff.net>
> Subject: [PATCH] thin-pack: try harder to create delta against preferred base
I just sat down to write a nicer commit message, and behold, it was done
for me. Thanks.
> When creating a thin-pack to transfer recent history, it is likely that we
> will try to send an object that is recorded in full, as it is newer. But
> the heuristics to avoid recomputing futile delta effectively forbids us
> from attempting to express such an object as a delta based on another
> object. Sending an object in full is often more expensive than sending a
> suboptimal delta based on other objects, and it is even more so if we
> could use an object we know the receiving end already has (i.e. referred
> base object) as the delta base.
s/referred/preferred/
> Tweak the recomputation avoidance logic, so that we do not punt on
> computing delta against a preferred base object.
>
> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Other than that, it looks good to me.
Signed-off-by: Jeff King <peff@peff.net>
I'll try to deploy this to GitHub in the near future. I doubt we'll see
much of a dent in our bandwidth, though, as small fetches that this
helps are probably lost in the noise of clones.
-Peff
^ permalink raw reply
* Re: thin packs ending up fat
From: Jeff King @ 2012-01-13 1:59 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, Nicolas Pitre
In-Reply-To: <20120113015117.GA8245@sigill.intra.peff.net>
On Thu, Jan 12, 2012 at 08:51:17PM -0500, Jeff King wrote:
> > Tweak the recomputation avoidance logic, so that we do not punt on
> > computing delta against a preferred base object.
> >
> > Signed-off-by: Junio C Hamano <gitster@pobox.com>
>
> Other than that, it looks good to me.
>
> Signed-off-by: Jeff King <peff@peff.net>
Actually, Nicolas asked for the numbers to go into the
commit message (and gave his acked-by). So maybe append:
The effect of this patch can be seen on two simulated
upload-pack workloads. The first is based on 44 reflog
entries from my git.git origin/master reflog, and represents
the packs that kernel.org produced to send me git updates
for the past month or two. The second workload represents
much larger fetches, going from git's v1.0.0 tag to v1.1.0,
then v1.1.0 to v1.2.0, and so on.
The table below shows the average generated pack size and
the average CPU time consumed for each dataset, both before
and after the patch:
dataset
| reflog | tags
---------------------------------
before | 53358 | 2750977
size after | 32398 | 2668479
change | -39% | -3%
---------------------------------
before | 0.18 | 1.12
CPU after | 0.18 | 1.15
change | +0% | +3%
This patch makes a much bigger difference for packs with a
shorter slice of history (since its effect is seen at the
boundaries of the pack) though it has some benefit even for
larger packs.
Acked-by: Nicolas Pitre <nico@fluxnic.net>
^ permalink raw reply
* Re: thin packs ending up fat
From: Nicolas Pitre @ 2012-01-13 2:19 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Jeff King, git
In-Reply-To: <7vwr8wz8u9.fsf@alter.siamese.dyndns.org>
On Thu, 12 Jan 2012, Junio C Hamano wrote:
> From: Jeff King <peff@peff.net>
> Subject: [PATCH] thin-pack: try harder to create delta against preferred base
>
> When creating a pack using objects that reside in existing packs, we try
> to avoid recomputing futile delta between an object (trg) and a candidate
> for its base object (src) if they are stored in the same packfile, and trg
> is not recorded as a delta already. This heuristics makes sense because it
> is likely that we tried to express trg as a delta based on src but it did
> not produce a good delta when we created the existing pack.
>
> As the pack heuristics prefer producing delta to remove data, and Linus's
> law dictates that the size of a file grows over time, we tend to record
> the newest version of the file as inflated, and older ones as delta
> against it.
>
> When creating a thin-pack to transfer recent history, it is likely that we
> will try to send an object that is recorded in full, as it is newer. But
> the heuristics to avoid recomputing futile delta effectively forbids us
> from attempting to express such an object as a delta based on another
> object. Sending an object in full is often more expensive than sending a
> suboptimal delta based on other objects, and it is even more so if we
> could use an object we know the receiving end already has (i.e. referred
> base object) as the delta base.
>
> Tweak the recomputation avoidance logic, so that we do not punt on
> computing delta against a preferred base object.
>
> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Acked-by: Nicolas Pitre <nico@fluxnic.net>
> ---
>
> builtin/pack-objects.c | 9 +++++++--
> 1 files changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> index c6e2d87..8bfe3a6 100644
> --- a/builtin/pack-objects.c
> +++ b/builtin/pack-objects.c
> @@ -1248,11 +1248,16 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,
> return -1;
>
> /*
> - * We do not bother to try a delta that we discarded
> - * on an earlier try, but only when reusing delta data.
> + * We do not bother to try a delta that we discarded on an
> + * earlier try, but only when reusing delta data. Note that
> + * src_entry that is marked as the preferred_base should always
> + * be considered, as even if we produce a suboptimal delta against
> + * it, we will still save the transfer cost, as we already know
> + * the other side has it and we won't send src_entry at all.
> */
> if (reuse_delta && trg_entry->in_pack &&
> trg_entry->in_pack == src_entry->in_pack &&
> + !src_entry->preferred_base &&
> trg_entry->in_pack_type != OBJ_REF_DELTA &&
> trg_entry->in_pack_type != OBJ_OFS_DELTA)
> return 0;
> --
> 1.7.9.rc0.53.gc8bc2
>
^ permalink raw reply
* Re: problem with updating a shallow clone via a branch fetch
From: Rahul Jain @ 2012-01-13 4:09 UTC (permalink / raw)
To: git
In-Reply-To: <AANLkTimSRG+UkD1vJszUf+3j0nV+_pPWSE3N2Dy9SfTa@mail.gmail.com>
I was also getting the same error. I went to .git/shallow file and deleted
<hash>. After that everything was working fine.
^ permalink raw reply
* Re: git svn clone terminating prematurely (I think)
From: Steven Michalske @ 2012-01-13 7:10 UTC (permalink / raw)
To: Steven Line; +Cc: git
In-Reply-To: <CAJ1a7SpRP5GymPrpEchuNk3xwkJLHBKjsXY_85Xs_LAzXtWYuw@mail.gmail.com>
Steven,
In this case nohup seems like a pain. There is a utility called "screen" available on most unixes.
It allows you to create a virtual terminal that you can detach from. After you detach you can log out, some time later log back in and then reattach.
Steve
On Jan 12, 2012, at 10:25 AM, Steven Line wrote:
> Thank you Jonathan.
>
> I had a breakthrough yesterday on this problem. To make a long story
> short my ssh connection to the server where I was running 'nohup git
> svn clone' was timing out. Additionally the nohup I was using wasn't
> really protecting the git svn clone process so an hour or two after
> the ssh disconnected, the git would terminate leaving me with an
> imcomplete repository. I don't understand why the nohup wasn't
> working yet. So my problem wasn't due to git itself.
>
> I started my most recent git svn clone and it's now been running for
> 18 hours. I'm optimistic that I've solved the problem. If my git
> does terminate early then I'll try a 'git svn fetch' to complete the
> clone, based on what you said in your post.
>
> --
> Steven Line
> 303-910-1212
> sline00@gmail.com
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: thin packs ending up fat
From: Junio C Hamano @ 2012-01-13 7:19 UTC (permalink / raw)
To: Jeff King; +Cc: git, Nicolas Pitre
In-Reply-To: <20120113015117.GA8245@sigill.intra.peff.net>
Jeff King <peff@peff.net> writes:
> On Thu, Jan 12, 2012 at 05:31:42PM -0800, Junio C Hamano wrote:
>
>> From: Jeff King <peff@peff.net>
>> Subject: [PATCH] thin-pack: try harder to create delta against preferred base
>
> I just sat down to write a nicer commit message, and behold, it was done
> for me. Thanks.
Thank _you_ for a quick response and correction.
I wanted to make sure I understood the root cause of the issue and the
approach the patch takes to address it, instead of committing something
that smelled correct. And the only way I know to do so is to write it
down.
Especiallly, before coming up with the description, I was wondering if
this kind of symptom appears in non-thin cases, but after writing down the
justification for this patch, it became clear that we wouldn't have to
worry too much about that case. In a non-thin pack, we need to record one
object at least in a delta family in inflated base form, so we may as well
send that one near the tip that is already in that form for that, letting
the existing "avoid futile delta" heuristics to kick in. Other objects in
the same delta family will delta against it.
^ permalink raw reply
* [PATCH v4 00/10] nd/clone-detached
From: Nguyễn Thái Ngọc Duy @ 2012-01-13 7:21 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
In-Reply-To: <1326189427-20800-1-git-send-email-pclouds@gmail.com>
Some comment updates after discussion and squash in the fixup patch.
The code is exactly the same as nd/clone-detached in pu.
Nguyễn Thái Ngọc Duy (10):
t5601: add missing && cascade
clone: write detached HEAD in bare repositories
clone: factor out checkout code
clone: factor out HEAD update code
clone: factor out remote ref writing
clone: delay cloning until after remote HEAD checking
clone: --branch=<branch> always means refs/heads/<branch>
clone: refuse to clone if --branch points to bogus ref
clone: allow --branch to take a tag
clone: print advice on checking out detached HEAD
Documentation/git-clone.txt | 5 +-
advice.c | 14 +++
advice.h | 1 +
builtin/checkout.c | 16 +---
builtin/clone.c | 256 +++++++++++++++++++++++++------------------
t/t5601-clone.sh | 40 ++++++-
t/t5706-clone-branch.sh | 8 +-
transport.c | 5 +-
8 files changed, 211 insertions(+), 134 deletions(-)
--
1.7.3.1.256.g2539c.dirty
^ permalink raw reply
* [PATCH v4 01/10] t5601: add missing && cascade
From: Nguyễn Thái Ngọc Duy @ 2012-01-13 7:21 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
In-Reply-To: <1326189427-20800-1-git-send-email-pclouds@gmail.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
t/t5601-clone.sh | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh
index 87ee016..49821eb 100755
--- a/t/t5601-clone.sh
+++ b/t/t5601-clone.sh
@@ -9,9 +9,9 @@ test_expect_success setup '
rm -fr .git &&
test_create_repo src &&
(
- cd src
- >file
- git add file
+ cd src &&
+ >file &&
+ git add file &&
git commit -m initial
)
--
1.7.3.1.256.g2539c.dirty
^ permalink raw reply related
* [PATCH v4 02/10] clone: write detached HEAD in bare repositories
From: Nguyễn Thái Ngọc Duy @ 2012-01-13 7:21 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
In-Reply-To: <1326189427-20800-1-git-send-email-pclouds@gmail.com>
If we don't write, HEAD is still at refs/heads/master as initialized
by init-db, which may or may not match remote's HEAD.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
builtin/clone.c | 9 +++------
t/t5601-clone.sh | 25 ++++++++++++++++++++++++-
2 files changed, 27 insertions(+), 7 deletions(-)
diff --git a/builtin/clone.c b/builtin/clone.c
index 86db954..8dde1ea 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -720,12 +720,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
}
} else if (remote_head) {
/* Source had detached HEAD pointing somewhere. */
- if (!option_bare) {
- update_ref(reflog_msg.buf, "HEAD",
- remote_head->old_sha1,
- NULL, REF_NODEREF, DIE_ON_ERR);
- our_head_points_at = remote_head;
- }
+ update_ref(reflog_msg.buf, "HEAD", remote_head->old_sha1,
+ NULL, REF_NODEREF, DIE_ON_ERR);
+ our_head_points_at = remote_head;
} else {
/* Nothing to checkout out */
if (!option_no_checkout)
diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh
index 49821eb..e0b8db6 100755
--- a/t/t5601-clone.sh
+++ b/t/t5601-clone.sh
@@ -12,7 +12,10 @@ test_expect_success setup '
cd src &&
>file &&
git add file &&
- git commit -m initial
+ git commit -m initial &&
+ echo 1 >file &&
+ git add file &&
+ git commit -m updated
)
'
@@ -88,6 +91,26 @@ test_expect_success 'clone --mirror' '
'
+test_expect_success 'clone --mirror with detached HEAD' '
+
+ ( cd src && git checkout HEAD^ && git rev-parse HEAD >../expected ) &&
+ git clone --mirror src mirror.detached &&
+ ( cd src && git checkout - ) &&
+ GIT_DIR=mirror.detached git rev-parse HEAD >actual &&
+ test_cmp expected actual
+
+'
+
+test_expect_success 'clone --bare with detached HEAD' '
+
+ ( cd src && git checkout HEAD^ && git rev-parse HEAD >../expected ) &&
+ git clone --bare src bare.detached &&
+ ( cd src && git checkout - ) &&
+ GIT_DIR=bare.detached git rev-parse HEAD >actual &&
+ test_cmp expected actual
+
+'
+
test_expect_success 'clone --bare names the local repository <name>.git' '
git clone --bare src &&
--
1.7.3.1.256.g2539c.dirty
^ permalink raw reply related
* [PATCH v4 03/10] clone: factor out checkout code
From: Nguyễn Thái Ngọc Duy @ 2012-01-13 7:21 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
In-Reply-To: <1326189427-20800-1-git-send-email-pclouds@gmail.com>
Read HEAD from disk instead of relying on local variable
our_head_points_at, so that if earlier code fails to make HEAD
properly, it'll be detected.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
builtin/clone.c | 101 +++++++++++++++++++++++++++++++-----------------------
1 files changed, 58 insertions(+), 43 deletions(-)
diff --git a/builtin/clone.c b/builtin/clone.c
index 8dde1ea..100056d 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -448,6 +448,63 @@ static void write_remote_refs(const struct ref *local_refs)
clear_extra_refs();
}
+static int checkout(void)
+{
+ unsigned char sha1[20];
+ char *head;
+ struct lock_file *lock_file;
+ struct unpack_trees_options opts;
+ struct tree *tree;
+ struct tree_desc t;
+ int err = 0, fd;
+
+ if (option_no_checkout)
+ return 0;
+
+ head = resolve_refdup("HEAD", sha1, 1, NULL);
+ if (!head) {
+ warning(_("remote HEAD refers to nonexistent ref, "
+ "unable to checkout.\n"));
+ return 0;
+ }
+ if (strcmp(head, "HEAD")) {
+ if (prefixcmp(head, "refs/heads/"))
+ die(_("HEAD not found below refs/heads!"));
+ }
+ free(head);
+
+ /* We need to be in the new work tree for the checkout */
+ setup_work_tree();
+
+ lock_file = xcalloc(1, sizeof(struct lock_file));
+ fd = hold_locked_index(lock_file, 1);
+
+ memset(&opts, 0, sizeof opts);
+ opts.update = 1;
+ opts.merge = 1;
+ opts.fn = oneway_merge;
+ opts.verbose_update = (option_verbosity > 0);
+ opts.src_index = &the_index;
+ opts.dst_index = &the_index;
+
+ tree = parse_tree_indirect(sha1);
+ parse_tree(tree);
+ init_tree_desc(&t, tree->buffer, tree->size);
+ unpack_trees(1, &t, &opts);
+
+ if (write_cache(fd, active_cache, active_nr) ||
+ commit_locked_index(lock_file))
+ die(_("unable to write new index file"));
+
+ err |= run_hook(NULL, "post-checkout", sha1_to_hex(null_sha1),
+ sha1_to_hex(sha1), "1", NULL);
+
+ if (!err && option_recursive)
+ err = run_command_v_opt(argv_submodule, RUN_GIT_CMD);
+
+ return err;
+}
+
static int write_one_config(const char *key, const char *value, void *data)
{
return git_config_set_multivar(key, value ? value : "true", "^$", 0);
@@ -722,13 +779,6 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
/* Source had detached HEAD pointing somewhere. */
update_ref(reflog_msg.buf, "HEAD", remote_head->old_sha1,
NULL, REF_NODEREF, DIE_ON_ERR);
- our_head_points_at = remote_head;
- } else {
- /* Nothing to checkout out */
- if (!option_no_checkout)
- warning(_("remote HEAD refers to nonexistent ref, "
- "unable to checkout.\n"));
- option_no_checkout = 1;
}
if (transport) {
@@ -736,42 +786,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
transport_disconnect(transport);
}
- if (!option_no_checkout) {
- struct lock_file *lock_file = xcalloc(1, sizeof(struct lock_file));
- struct unpack_trees_options opts;
- struct tree *tree;
- struct tree_desc t;
- int fd;
-
- /* We need to be in the new work tree for the checkout */
- setup_work_tree();
-
- fd = hold_locked_index(lock_file, 1);
-
- memset(&opts, 0, sizeof opts);
- opts.update = 1;
- opts.merge = 1;
- opts.fn = oneway_merge;
- opts.verbose_update = (option_verbosity > 0);
- opts.src_index = &the_index;
- opts.dst_index = &the_index;
-
- tree = parse_tree_indirect(our_head_points_at->old_sha1);
- parse_tree(tree);
- init_tree_desc(&t, tree->buffer, tree->size);
- unpack_trees(1, &t, &opts);
-
- if (write_cache(fd, active_cache, active_nr) ||
- commit_locked_index(lock_file))
- die(_("unable to write new index file"));
-
- err |= run_hook(NULL, "post-checkout", sha1_to_hex(null_sha1),
- sha1_to_hex(our_head_points_at->old_sha1), "1",
- NULL);
-
- if (!err && option_recursive)
- err = run_command_v_opt(argv_submodule, RUN_GIT_CMD);
- }
+ err = checkout();
strbuf_release(&reflog_msg);
strbuf_release(&branch_top);
--
1.7.3.1.256.g2539c.dirty
^ permalink raw reply related
* [PATCH v4 04/10] clone: factor out HEAD update code
From: Nguyễn Thái Ngọc Duy @ 2012-01-13 7:21 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
In-Reply-To: <1326189427-20800-1-git-send-email-pclouds@gmail.com>
While at it, update the comment at "if (remote_head)"
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
builtin/clone.c | 41 ++++++++++++++++++++++++-----------------
1 files changed, 24 insertions(+), 17 deletions(-)
diff --git a/builtin/clone.c b/builtin/clone.c
index 100056d..baed5ae 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -448,6 +448,29 @@ static void write_remote_refs(const struct ref *local_refs)
clear_extra_refs();
}
+static void update_head(const struct ref *our, const struct ref *remote,
+ const char *msg)
+{
+ if (our) {
+ /* Local default branch link */
+ create_symref("HEAD", our->name, NULL);
+ if (!option_bare) {
+ const char *head = skip_prefix(our->name, "refs/heads/");
+ update_ref(msg, "HEAD", our->old_sha1, NULL, 0, DIE_ON_ERR);
+ install_branch_config(0, head, option_origin, our->name);
+ }
+ } else if (remote) {
+ /*
+ * We know remote HEAD points to a non-branch, or
+ * HEAD points to a branch but we don't know which one, or
+ * we asked for a specific branch but it did not exist.
+ * Detach HEAD in all these cases.
+ */
+ update_ref(msg, "HEAD", remote->old_sha1,
+ NULL, REF_NODEREF, DIE_ON_ERR);
+ }
+}
+
static int checkout(void)
{
unsigned char sha1[20];
@@ -763,23 +786,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
reflog_msg.buf);
}
- if (our_head_points_at) {
- /* Local default branch link */
- create_symref("HEAD", our_head_points_at->name, NULL);
- if (!option_bare) {
- const char *head = skip_prefix(our_head_points_at->name,
- "refs/heads/");
- update_ref(reflog_msg.buf, "HEAD",
- our_head_points_at->old_sha1,
- NULL, 0, DIE_ON_ERR);
- install_branch_config(0, head, option_origin,
- our_head_points_at->name);
- }
- } else if (remote_head) {
- /* Source had detached HEAD pointing somewhere. */
- update_ref(reflog_msg.buf, "HEAD", remote_head->old_sha1,
- NULL, REF_NODEREF, DIE_ON_ERR);
- }
+ update_head(our_head_points_at, remote_head, reflog_msg.buf);
if (transport) {
transport_unlock_pack(transport);
--
1.7.3.1.256.g2539c.dirty
^ permalink raw reply related
* [PATCH v4 05/10] clone: factor out remote ref writing
From: Nguyễn Thái Ngọc Duy @ 2012-01-13 7:21 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
In-Reply-To: <1326189427-20800-1-git-send-email-pclouds@gmail.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
builtin/clone.c | 36 ++++++++++++++++++++++++------------
1 files changed, 24 insertions(+), 12 deletions(-)
diff --git a/builtin/clone.c b/builtin/clone.c
index baed5ae..d1d6dca 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -448,6 +448,28 @@ static void write_remote_refs(const struct ref *local_refs)
clear_extra_refs();
}
+static void update_remote_refs(const struct ref *refs,
+ const struct ref *mapped_refs,
+ const struct ref *remote_head_points_at,
+ const char *branch_top,
+ const char *msg)
+{
+ if (refs) {
+ clear_extra_refs();
+ write_remote_refs(mapped_refs);
+ }
+
+ if (remote_head_points_at && !option_bare) {
+ struct strbuf head_ref = STRBUF_INIT;
+ strbuf_addstr(&head_ref, branch_top);
+ strbuf_addstr(&head_ref, "HEAD");
+ create_symref(head_ref.buf,
+ remote_head_points_at->peer_ref->name,
+ msg);
+ }
+
+}
+
static void update_head(const struct ref *our, const struct ref *remote,
const char *msg)
{
@@ -740,10 +762,6 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
}
if (refs) {
- clear_extra_refs();
-
- write_remote_refs(mapped_refs);
-
remote_head = find_ref_by_name(refs, "HEAD");
remote_head_points_at =
guess_remote_head(remote_head, mapped_refs, 0);
@@ -777,14 +795,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
"refs/heads/master");
}
- if (remote_head_points_at && !option_bare) {
- struct strbuf head_ref = STRBUF_INIT;
- strbuf_addstr(&head_ref, branch_top.buf);
- strbuf_addstr(&head_ref, "HEAD");
- create_symref(head_ref.buf,
- remote_head_points_at->peer_ref->name,
- reflog_msg.buf);
- }
+ update_remote_refs(refs, mapped_refs, remote_head_points_at,
+ branch_top.buf, reflog_msg.buf);
update_head(our_head_points_at, remote_head, reflog_msg.buf);
--
1.7.3.1.256.g2539c.dirty
^ permalink raw reply related
* [PATCH v4 06/10] clone: delay cloning until after remote HEAD checking
From: Nguyễn Thái Ngọc Duy @ 2012-01-13 7:21 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
In-Reply-To: <1326189427-20800-1-git-send-email-pclouds@gmail.com>
This gives us an opportunity to abort the command during remote HEAD
check without wasting much bandwidth.
Cloning with remote-helper remains before the check because the remote
helper updates mapped_refs, which is necessary for remote ref checks.
foreign_vcs field is used to indicate the transport is handled by
remote helper.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
builtin/clone.c | 54 +++++++++++++++++++++++++++---------------------------
transport.c | 5 ++++-
2 files changed, 31 insertions(+), 28 deletions(-)
diff --git a/builtin/clone.c b/builtin/clone.c
index d1d6dca..a9d06a4 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -361,13 +361,8 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,
closedir(dir);
}
-static const struct ref *clone_local(const char *src_repo,
- const char *dest_repo)
+static void clone_local(const char *src_repo, const char *dest_repo)
{
- const struct ref *ret;
- struct remote *remote;
- struct transport *transport;
-
if (option_shared) {
struct strbuf alt = STRBUF_INIT;
strbuf_addf(&alt, "%s/objects", src_repo);
@@ -383,13 +378,8 @@ static const struct ref *clone_local(const char *src_repo,
strbuf_release(&dest);
}
- remote = remote_get(src_repo);
- transport = transport_get(remote, src_repo);
- ret = transport_get_remote_refs(transport);
- transport_disconnect(transport);
if (0 <= option_verbosity)
printf(_("done.\n"));
- return ret;
}
static const char *junk_work_tree;
@@ -581,6 +571,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
struct strbuf branch_top = STRBUF_INIT, reflog_msg = STRBUF_INIT;
struct transport *transport = NULL;
char *src_ref_prefix = "refs/heads/";
+ struct remote *remote;
int err = 0;
struct refspec *refspec;
@@ -732,13 +723,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
strbuf_reset(&value);
- if (is_local) {
- refs = clone_local(path, git_dir);
- mapped_refs = wanted_peer_refs(refs, refspec);
- } else {
- struct remote *remote = remote_get(option_origin);
- transport = transport_get(remote, remote->url[0]);
+ remote = remote_get(option_origin);
+ transport = transport_get(remote, remote->url[0]);
+ if (!is_local) {
if (!transport->get_refs_list || !transport->fetch)
die(_("Don't know how to clone %s"), transport->url);
@@ -753,14 +741,23 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
if (option_upload_pack)
transport_set_option(transport, TRANS_OPT_UPLOADPACK,
option_upload_pack);
-
- refs = transport_get_remote_refs(transport);
- if (refs) {
- mapped_refs = wanted_peer_refs(refs, refspec);
- transport_fetch_refs(transport, mapped_refs);
- }
}
+ refs = transport_get_remote_refs(transport);
+ mapped_refs = refs ? wanted_peer_refs(refs, refspec) : NULL;
+
+ /*
+ * mapped_refs may be updated if transport-helper is used so
+ * we need fetch it early because remote_head code below
+ * relies on it.
+ *
+ * for normal clones, transport_get_remote_refs() should
+ * return reliable ref set, we can delay cloning until after
+ * remote HEAD check.
+ */
+ if (!is_local && remote->foreign_vcs && refs)
+ transport_fetch_refs(transport, mapped_refs);
+
if (refs) {
remote_head = find_ref_by_name(refs, "HEAD");
remote_head_points_at =
@@ -795,15 +792,18 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
"refs/heads/master");
}
+ if (is_local)
+ clone_local(path, git_dir);
+ else if (refs && !remote->foreign_vcs)
+ transport_fetch_refs(transport, mapped_refs);
+
update_remote_refs(refs, mapped_refs, remote_head_points_at,
branch_top.buf, reflog_msg.buf);
update_head(our_head_points_at, remote_head, reflog_msg.buf);
- if (transport) {
- transport_unlock_pack(transport);
- transport_disconnect(transport);
- }
+ transport_unlock_pack(transport);
+ transport_disconnect(transport);
err = checkout();
diff --git a/transport.c b/transport.c
index a99b7c9..4366639 100644
--- a/transport.c
+++ b/transport.c
@@ -895,8 +895,10 @@ struct transport *transport_get(struct remote *remote, const char *url)
while (is_urlschemechar(p == url, *p))
p++;
- if (!prefixcmp(p, "::"))
+ if (!prefixcmp(p, "::")) {
helper = xstrndup(url, p - url);
+ remote->foreign_vcs = helper;
+ }
}
if (helper) {
@@ -938,6 +940,7 @@ struct transport *transport_get(struct remote *remote, const char *url)
char *handler = xmalloc(len + 1);
handler[len] = 0;
strncpy(handler, url, len);
+ remote->foreign_vcs = handler;
transport_helper_init(ret, handler);
}
--
1.7.3.1.256.g2539c.dirty
^ permalink raw reply related
* [PATCH v4 07/10] clone: --branch=<branch> always means refs/heads/<branch>
From: Nguyễn Thái Ngọc Duy @ 2012-01-13 7:21 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
In-Reply-To: <1326189427-20800-1-git-send-email-pclouds@gmail.com>
It does not make sense to look outside refs/heads for HEAD's target
(src_ref_prefix can be set to "refs/" if --mirror is used) because ref
code only allows symref in form refs/heads/...
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
builtin/clone.c | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/builtin/clone.c b/builtin/clone.c
index a9d06a4..89ba6f0 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -463,7 +463,7 @@ static void update_remote_refs(const struct ref *refs,
static void update_head(const struct ref *our, const struct ref *remote,
const char *msg)
{
- if (our) {
+ if (our && !prefixcmp(our->name, "refs/heads/")) {
/* Local default branch link */
create_symref("HEAD", our->name, NULL);
if (!option_bare) {
@@ -765,7 +765,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
if (option_branch) {
struct strbuf head = STRBUF_INIT;
- strbuf_addstr(&head, src_ref_prefix);
+ strbuf_addstr(&head, "refs/heads/");
strbuf_addstr(&head, option_branch);
our_head_points_at =
find_ref_by_name(mapped_refs, head.buf);
--
1.7.3.1.256.g2539c.dirty
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox