* Re: sparse support in pu
From: skillzero @ 2009-08-17 8:49 UTC (permalink / raw)
To: Nguyen Thai Ngoc Duy; +Cc: git
In-Reply-To: <fcaeb9bf0908170117v67e9f8b1ga56edcda14821e91@mail.gmail.com>
On Mon, Aug 17, 2009 at 1:17 AM, Nguyen Thai Ngoc Duy<pclouds@gmail.com> wrote:
> On Mon, Aug 17, 2009 at 1:09 PM, <skillzero@gmail.com> wrote:
>> 1. Have people decided whether it should be on by default if you have
>> a .git/info/sparse file? I'd definitely like it to be on by default.
>> When I first tried it, I didn't realize I had to use --sparse to git
>> checkout to get it to use the sparse rules. The same goes for a merge
>> I did that happened to have a file in the excluded area (it included
>> it because I didn't use --sparse to git merge).
>
> I tend to make it enabled by default too. I have made it stricter to
> trigger reading sparse in unpack_trees() -- only do it when
> unpack_opts.update is TRUE. This should make it safer to be enabled by
> default.
Other than it being new and not-widely-tested code, is there any
additional risk to having it enabled by default if there are no sparse
patterns defined?
It would be nice if .git/info/sparse is there by default (like
.git/info/exclude) with some commented out instructions (also like
.git/info/exclude).
>> 2. Is it not hooked up to git reset yet? I did a git checkout --sparse
>> and things look liked I expected then I did a git reset --hard
>> origin/master and it started checking out all the stuff previously
>> excluded via .git/info/sparse. I tried --sparse, but it didn't know
>> about that option.
>
> Because sparse was disabled by default, and "git reset" did not enable
> it. It'd be interesting to see what "git reset --hard" should do in
> this case: will it apply .git/info/sparse or not, which brings us back
> to the "default or not" question, hmm..
It seems like if it's going to be off by default (which it hopefully
won't be) then git reset would need to support --sparse since
otherwise, you'd never be able to git reset without undoing your
sparse options (unless you did a subsequent git checkout --sparse).
I also noticed that after I did the git reset, the sparse stuff seemed
to get into a weird state such that sparse patterns weren't working
reliably. For example, I couldn't get it to exclude a directory.
>> 3. One thing that was confusing is that I needed a trailing slash on
>> directories in .git/info/sparse to get them excluded. This seems
>> different than .gitignore, which works for me without the trailing
>> slash.
>
> Hmm.. probably because Git feeds directories to .gitignore handling
> functions. There is not much I can do, index does not have
> directories. I don't know if it's worth generating "directories" from
> index.
Maybe just add a note in the documentation? If there's a default
.git/info/sparse file then it might be good place to put a note as
well.
^ permalink raw reply
* Re: [msysGit] quick question about __stdcall at run-command.c mingw.c
From: Johannes Sixt @ 2009-08-17 8:43 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: Pat Thoyts, Frank Li, git, msysGit
In-Reply-To: <alpine.DEB.1.00.0908171018490.4991@intel-tinevez-2-302>
Johannes Schindelin schrieb:
> On Mon, 17 Aug 2009, Johannes Sixt wrote:
>> Pat Thoyts schrieb:
>>> 2009/8/17 Frank Li <lznuaa@gmail.com>:
>>>> I am tring to clear VC build patch.
>>>>
>>>> I found __stdcall position break MSVC build.
>>>>
>>>> static __stdcall unsigned run_thread(void *data)
>>>>
>>>> MSVC require __stdcall should be between return type and function
>>>> name. like static unsigned __stdcall run_thread(void *data)
>>>>
>>>> I think msys gcc should support MSVC format.
>
> I think that it does.
>
> But it is _your_ duty to check.
Cool down. Asking for "please could you check whether this works" *if* you
don't have the infrastructure to test it yourself is certainly dutyful enough.
Do you have an Irix, Solaris, HP box on your desk next to your Linux, so
that you don't have to ask others to test your patches?
-- Hannes
^ permalink raw reply
* Re: Linus' sha1 is much faster!
From: Andreas Ericsson @ 2009-08-17 8:22 UTC (permalink / raw)
To: Linus Torvalds
Cc: John Tapsell, Bryan Donlan, Pádraig Brady, Bug-coreutils,
Git Mailing List, Brandon Casey, Junio C Hamano, Nicolas Pitre,
Paul Kocher, Paul Kocher
In-Reply-To: <alpine.LFD.2.01.0908151336530.3162@localhost.localdomain>
Linus Torvalds wrote:
>
> On Sat, 15 Aug 2009, Linus Torvalds wrote:
>> That said, I don't know if the MPL is ok for X11. I've not looked at
>> compatibility issues with MPL. For git, we could just ignore the MPL,
>> since the GPLv2 was acceptable regardless of it.
>
> If MPL isn't ok for X11, then we'd need to make sure that even the
> silliest Mozilla crud has been rewritten. There really isn't much, but
> hey, the _history_ is based on the mozilla code, and who knows - the
> 'blk_SHA_CTX' struct has things like the fields in the same order as the
> Mozilla equivalent, for all those historical reasons.
>
> (Heh. Looking at that, I probably should move the 'size' field first,
> since that would have different alignment rules, and the struct would be
> more tightly packed that way, and initialize better).
>
> Afaik, none of the actual code remains (the mozilla SHA1 thing did the
> wrong thing for performance even for just the final bytes, and did those a
> byte at a time etc, so I rewrote even the trivial SHA1_Final parts).
>
> Of course, maybe the Mozilla people would be interested in taking my
> faster version, and say that the new-BSD license is ok, and make everybody
> happy. The only listed author for the Mozilla SHA1 is Paul Kocher. I added
> him to the Cc.
>
> Paul, for your information, we're talking about a faster rewritten "mostly
> portable" SHA1 routines that you can find at
>
> http://git.kernel.org/?p=git/git.git;a=tree;f=block-sha1;hb=pu
>
> (follow the "blob" pointers to see sha1.c and sha1.h). I don't know if
> you're active with Mozilla/Firefox or whether you even care, but you seem
> to be the logical choice of person to ask.
>
I contacted Paul in february this year to get permission to use the mozilla
sha1 code for libgit2. His reply then was:
"I'm not sure which version the diffs are relative to, so I haven't reviewed them.
It's fine to distribute under BSD, GPL, or LGPL, however."
I also got explicit permission to relicense it under GPLv2 with the gcc exception.
I added the mail-address I used to contact him to CC as well. Sorry if you get
this twice, Paul.
Naturally, I'd like to use the faster version for libgit2 as well. The people
who Linus listed as contributors earlier (Brandon Casy, Linus, Junio and Nicolas
Pitre) have already consented to relicense their git contributions for libgit2
use. If anyone would like to revoke that consent for this code, speak now please,
or I'll patch it into libgit2 as well.
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.
^ permalink raw reply
* Re: [msysGit] quick question about __stdcall at run-command.c mingw.c
From: Johannes Schindelin @ 2009-08-17 8:21 UTC (permalink / raw)
To: Johannes Sixt; +Cc: Pat Thoyts, Frank Li, git, msysGit
In-Reply-To: <4A890C32.6010507@viscovery.net>
Hi,
On Mon, 17 Aug 2009, Johannes Sixt wrote:
> Pat Thoyts schrieb:
> > 2009/8/17 Frank Li <lznuaa@gmail.com>:
> >> I am tring to clear VC build patch.
> >>
> >> I found __stdcall position break MSVC build.
> >>
> >> static __stdcall unsigned run_thread(void *data)
> >>
> >> MSVC require __stdcall should be between return type and function
> >> name. like static unsigned __stdcall run_thread(void *data)
> >>
> >> I think msys gcc should support MSVC format.
I think that it does.
But it is _your_ duty to check.
> >> Should I directly change to MSVC format or add _MSC_VER marcro like
> >>
> >> #if defined(__MINGW32__)
> >> static __stdcall unsigned run_thread(void *data)
> >> #elif defined(_MSC_VER) /*MSVC must put __stdcall between return value
> >> and function*/
> >> static unsigned __stdcall run_thread(void *data)
> >> #endif
Noooo! NO _MSC_VER crap in mingw.c. Really. I am able to repeat that as
often as you want me, but I'd prefer not to.
> > The win32 api prototype used for thread entry functions is declared as
> > a DWORD (WINAPI *LPTHREAD_START_ROUTINE)(LPVOID) type in the mingw
> > headers and WINAPI as #define WINAPI __stdcall. This is true for the
> > MSVC headers as well. So gcc and msvc are happy using the same
> > definition for such a function and just "static unsigned long WINAPI
> > run_thread(void *)" might well be sensible.
>
> Change the code to
>
> static unsigned __stdcall run_thread(void *data)
>
> The documentation explictly says: "The routine at start_address passed to
> _beginthreadex must use the __stdcall calling convention...". So __stdcall
> it is.
I could not agree more.
Ciao,
Dscho
^ permalink raw reply
* Re: sparse support in pu
From: Nguyen Thai Ngoc Duy @ 2009-08-17 8:17 UTC (permalink / raw)
To: skillzero; +Cc: git
In-Reply-To: <2729632a0908162309ma6e7d41kc3bafe4575120630@mail.gmail.com>
On Mon, Aug 17, 2009 at 1:09 PM, <skillzero@gmail.com> wrote:
> 1. Have people decided whether it should be on by default if you have
> a .git/info/sparse file? I'd definitely like it to be on by default.
> When I first tried it, I didn't realize I had to use --sparse to git
> checkout to get it to use the sparse rules. The same goes for a merge
> I did that happened to have a file in the excluded area (it included
> it because I didn't use --sparse to git merge).
I tend to make it enabled by default too. I have made it stricter to
trigger reading sparse in unpack_trees() -- only do it when
unpack_opts.update is TRUE. This should make it safer to be enabled by
default.
> 2. Is it not hooked up to git reset yet? I did a git checkout --sparse
> and things look liked I expected then I did a git reset --hard
> origin/master and it started checking out all the stuff previously
> excluded via .git/info/sparse. I tried --sparse, but it didn't know
> about that option.
Because sparse was disabled by default, and "git reset" did not enable
it. It'd be interesting to see what "git reset --hard" should do in
this case: will it apply .git/info/sparse or not, which brings us back
to the "default or not" question, hmm..
> 3. One thing that was confusing is that I needed a trailing slash on
> directories in .git/info/sparse to get them excluded. This seems
> different than .gitignore, which works for me without the trailing
> slash.
Hmm.. probably because Git feeds directories to .gitignore handling
functions. There is not much I can do, index does not have
directories. I don't know if it's worth generating "directories" from
index.
--
Duy
^ permalink raw reply
* Re: [PATCH 0/6 (v2)] Suggested for PU: revision caching system to significantly speed up packing/walking
From: Johannes Schindelin @ 2009-08-17 8:16 UTC (permalink / raw)
To: Nick Edelen
Cc: Sam Vilain, Junio C Hamano, Nicolas Pitre, Michael J Gruber,
Jeff King, Shawn O. Pearce, Andreas Ericsson, Christian Couder,
git@vger.kernel.org
In-Reply-To: <c77435a80908161756n88eb967wbc82058fc4975c04@mail.gmail.com>
Hi,
On Mon, 17 Aug 2009, Nick Edelen wrote:
> If I'm being confusing I can just upload them to v3, but I didn't want
> to bombard everyone with duplicates.
I think by now people _are_ confused, so it's probably better to "bomb"
everyone with a clean current version of your patch series.
Although I have to admit that I am scared away by the sheer size of the
individual patches (together with bad feelings from earlier discussions
that I want to avoid).
Ciao,
Dscho
^ permalink raw reply
* Re: "make quick-install-man" broke recently
From: Junio C Hamano @ 2009-08-17 8:02 UTC (permalink / raw)
To: Johannes Sixt
Cc: Jacob Helwig, Linus Torvalds, Kjetil Barvik, Randal L. Schwartz,
git@vger.kernel.org
In-Reply-To: <4A890E09.9010507@viscovery.net>
Johannes Sixt <j.sixt@viscovery.net> writes:
> Junio C Hamano schrieb:
>> check_path(): allow symlinked directories to checkout-index --prefix
>
>> +test_expect_success 'checkout-index -f twice with --prefix' '
>
> Please add SYMLINKS prerequisite before you publish this test case.
>
>> + mkdir -p tar/get &&
>> + ln -s tar/get there &&
>> + echo first &&
>> + git checkout-index -a -f --prefix=there/ &&
>> + echo second &&
>> + git checkout-index -a -f --prefix=there/
>> +'
Heh, I am not sure if the fix is the best approach to begin with yet ;-)
^ permalink raw reply
* Re: "make quick-install-man" broke recently
From: Johannes Sixt @ 2009-08-17 8:00 UTC (permalink / raw)
To: Junio C Hamano
Cc: Jacob Helwig, Linus Torvalds, Kjetil Barvik, Randal L. Schwartz,
git@vger.kernel.org
In-Reply-To: <7vhbw72ap3.fsf@alter.siamese.dyndns.org>
Junio C Hamano schrieb:
> check_path(): allow symlinked directories to checkout-index --prefix
> +test_expect_success 'checkout-index -f twice with --prefix' '
Please add SYMLINKS prerequisite before you publish this test case.
> + mkdir -p tar/get &&
> + ln -s tar/get there &&
> + echo first &&
> + git checkout-index -a -f --prefix=there/ &&
> + echo second &&
> + git checkout-index -a -f --prefix=there/
> +'
Thanks,
-- Hannes
^ permalink raw reply
* Re: [msysGit] quick question about __stdcall at run-command.c mingw.c
From: Johannes Sixt @ 2009-08-17 7:52 UTC (permalink / raw)
To: Pat Thoyts, Frank Li; +Cc: git, msysGit
In-Reply-To: <a5b261830908161703l2296448l698d07d01ae8a6d3@mail.gmail.com>
Pat Thoyts schrieb:
> 2009/8/17 Frank Li <lznuaa@gmail.com>:
>> I am tring to clear VC build patch.
>>
>> I found __stdcall position break MSVC build.
>>
>> static __stdcall unsigned run_thread(void *data)
>>
>> MSVC require __stdcall should be between return type and function name.
>> like
>> static unsigned __stdcall run_thread(void *data)
>>
>> I think msys gcc should support MSVC format.
>>
>> Should I directly change to MSVC format or add _MSC_VER marcro like
>>
>> #if defined(__MINGW32__)
>> static __stdcall unsigned run_thread(void *data)
>> #elif defined(_MSC_VER) /*MSVC must put __stdcall between return value
>> and function*/
>> static unsigned __stdcall run_thread(void *data)
>> #endif
>
> The win32 api prototype used for thread entry functions is declared as
> a DWORD (WINAPI *LPTHREAD_START_ROUTINE)(LPVOID) type in the mingw
> headers and WINAPI as #define WINAPI __stdcall. This is true for the
> MSVC headers as well. So gcc and msvc are happy using the same
> definition for such a function and just "static unsigned long WINAPI
> run_thread(void *)" might well be sensible.
Change the code to
static unsigned __stdcall run_thread(void *data)
The documentation explictly says: "The routine at start_address passed to
_beginthreadex must use the __stdcall calling convention...". So __stdcall
it is.
-- Hannes
^ permalink raw reply
* Re: How to stop sharing objects between repositories
From: Jeff King @ 2009-08-17 7:50 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Mike Galbraith, Johannes Schindelin, Jon Jensen, git
In-Reply-To: <7v1vna3nae.fsf@alter.siamese.dyndns.org>
On Mon, Aug 17, 2009 at 12:35:53AM -0700, Junio C Hamano wrote:
> > Did you miss the part where I asked "should we include instructions to
> > the user on how to fix this"?
>
> Actually, I didn't. It is very hard to lose data once you put it in git;
> "by following recovery insn the user can redo" is often trivially correct
> thanks to it.
>
> But it is not a very good option to cause the damage and then give
> recovery insn. The user might have ran out of quota, and even if he
> didn't, he wasted needless cycles for the unwanted sort of repacking.
OK, let's forget the warning, then. Hopefully between the note under
"clone -s" and the fact that most people should be using "git gc" these
days, it won't be a big issue.
-Peff
^ permalink raw reply
* Re: How to stop sharing objects between repositories
From: Junio C Hamano @ 2009-08-17 7:35 UTC (permalink / raw)
To: Jeff King; +Cc: Mike Galbraith, Johannes Schindelin, Jon Jensen, git
In-Reply-To: <20090817072559.GA9730@coredump.intra.peff.net>
Jeff King <peff@peff.net> writes:
>> But I can tell you that this "just warn" cannot be a good idea for a very
>> simple reason: breaking and then warning is useless---it is too late for
>> the user to do anything about it.
>
> Did you miss the part where I asked "should we include instructions to
> the user on how to fix this"?
Actually, I didn't. It is very hard to lose data once you put it in git;
"by following recovery insn the user can redo" is often trivially correct
thanks to it.
But it is not a very good option to cause the damage and then give
recovery insn. The user might have ran out of quota, and even if he
didn't, he wasted needless cycles for the unwanted sort of repacking.
^ permalink raw reply
* Re: How to stop sharing objects between repositories
From: Jeff King @ 2009-08-17 7:25 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Mike Galbraith, Johannes Schindelin, Jon Jensen, git
In-Reply-To: <7v63cm3ntl.fsf@alter.siamese.dyndns.org>
On Mon, Aug 17, 2009 at 12:24:22AM -0700, Junio C Hamano wrote:
> > I'm torn on whether this is actually a good idea.
>
> I would understand if you were torn if the proposed change were to refuse
> to run without -l in a repository with alternates when --force is not
> given, or something of that nature.
>
> But I can tell you that this "just warn" cannot be a good idea for a very
> simple reason: breaking and then warning is useless---it is too late for
> the user to do anything about it.
Did you miss the part where I asked "should we include instructions to
the user on how to fix this"?
-Peff
^ permalink raw reply
* Re: How to stop sharing objects between repositories
From: Junio C Hamano @ 2009-08-17 7:24 UTC (permalink / raw)
To: Jeff King; +Cc: Mike Galbraith, Johannes Schindelin, Jon Jensen, git
In-Reply-To: <20090817064801.GA31543@coredump.intra.peff.net>
Jeff King <peff@peff.net> writes:
> On Mon, Aug 17, 2009 at 04:21:22AM +0200, Mike Galbraith wrote:
>
>> > (1) Such a user does not necessarily know a casual "git repack -a" breaks
>> > the dependency, defeating the -s option s/he deliberately used in
>> > order to save disk space in the first place. Perhaps we can reword
>> > this further to kill two penguins with a single stone?
>>
>> Perhaps a runtime warning that you're about to break it? This user may
>> not even be the one who set the thing up, no?
>
> I'm not really sure what such a setup would look like....
> ...
> That being said, I can see there being setups where such a warning might
> be useful. However, we don't really know if the user _wants_ that
> effect, or if it is an accident.
> ...
> "here is how you break the dependency" advice will also get the warning.
>
> I'm torn on whether this is actually a good idea.
I would understand if you were torn if the proposed change were to refuse
to run without -l in a repository with alternates when --force is not
given, or something of that nature.
But I can tell you that this "just warn" cannot be a good idea for a very
simple reason: breaking and then warning is useless---it is too late for
the user to do anything about it.
^ permalink raw reply
* Re: Linus' sha1 is much faster!
From: George Spelvin @ 2009-08-17 7:23 UTC (permalink / raw)
To: bdonlan, johnflux, P; +Cc: art.08.09, git, linux, nico, torvalds
If it helps anyone resolve license issues, here's a from-FIPS-180-2
implementation that's placed in the public domain. That should be
compatible with any license.
It uses Linus's and Artur's performance ideas, and some of Linus' macro
ideas (in the rotate implementation), but tries to be textually different.
Is there anything recognizable that anyone cares to clam copyright to?
It's not quite 100% finished, as I haven't benchmarked it against Linus's
code yet, but it's functionally correct.
It's also clean with -W -Wall -Wextra.
TODO: Check if an initial copy to w[] is faster on i386 (less register
pressure).
/*
* Secure Hash Algorith SHA-1, as published in FIPS PUB 180-2.
*
* This implementation is in the public domain. Copyright abandoned.
* You may do anything you like with it, including evil things.
*
* This is a rewrite from scratch, based on Linus Torvalds' "block-sha1"
* from the git mailing list (August, 2009). Additional optimization
* ideas cribbed from
* - Artur Skawina (x86, particularly P4, and much benchmarking)
* - Nicilas Pitre (ARM)
*/
/* Cut here for sha1.h */
#include <stddef.h> /* For size_t */
#include <stdint.h> /* For uint32_t, uint64_t */
typedef struct SHA_context {
uint64_t len; /* May be shrunk to uint32_t */
uint32_t iv[5];
unsigned char buf[64]; /* Must be 32-bit aligned */
} SHA_CTX;
void SHA1_Init(struct SHA_context *c);
void SHA1_Update(struct SHA_context *c, void const *p, size_t n);
void SHA1_Final(unsigned char hash[20], struct SHA_context *c);
/* End of sha1.h */
#include <string.h> /* For memcpy() */
#include <arpa/inet.h> /* For ntohl() */
static void sha1_core(uint32_t iv[5], unsigned char const *p, size_t nblocks);
/* Machine specific hacks */
#if defined(__i386__) || defined(__x86_64__) || defined (__ppc__) || \
defined(__ppc64__) || defined(__powerpc__) || defined (__powerpc64__) || \
defined(__s390__) || defined(__s390x__)
/* Unaligned access is okay */
static inline uint32_t get_be32(unsigned char const *p)
{
return ntohl(*(uint32_t const *)p);
}
static inline void put_be32(unsigned char const *p, uint32_t v)
{
*(uint32_t *)p = htonl(v);
}
#else
/* Unaligned access is not okay; do conversion as byte fetches */
static inline uint32_t get_be32(unsigned char const *p)
{
return p[0] << 24 || p[1] << 16 | p[8] << 8 | p[3];
}
static inline void put_be32(unsigned char const *p, uint32_t v)
{
p[0] = v >> 24;
p[1] = v >> 16;
p[2] = v >> 8;
p[3] = v;
}
#endif
void SHA1_Init(struct SHA_context *c)
{
/* This is a prefix of the SHA_context structure */
static struct {
uint64_t len;
uint32_t iv[5];
} const iv = { 0,
{ 0x67452301, 0xEFCDAB89, 0x98BADCFE, 0x10325476, 0xC3D2E1F0 }
};
memcpy(c, &iv, sizeof iv);
}
void SHA1_Update(struct SHA_context *c, void const *p, size_t n)
{
size_t pos = c->len & 63; /* Offset into current input block */
c->len += n;
/* Initial partial block (if any) */
if (pos) {
size_t space = 63 - pos;
if (n < space)
goto end;
memcpy(c->buf + pos, p, space);
sha1_core(c->iv, c->buf, 1);
n -= space;
p = (char const *)p + space;
}
/* The large middle piece */
if (n >> 6) {
sha1_core(c->iv, p, n >> 6);
p = (char const *)p + (n & -(size_t)64);
n &= 63;
}
pos = 0;
end:
/* Final partial block (may be zero size) */
memcpy(c->buf + pos, p, n);
}
void SHA1_Final(unsigned char hash[20], struct SHA_context *c)
{
size_t pos = c->len & 63;
unsigned i;
/* Append a single 1 bit */
c->buf[pos++] = 0x80;
/* Append 0 bits until 64 bits remain in a block */
if (pos > 56) {
memset(c->buf + pos, 0, 64 - pos);
sha1_core(c->iv, c->buf, 1);
pos = 0;
}
memset(c->buf + pos, 0, 56 - pos);
/* Append total input length in bits */
((uint32_t *)c->buf)[14] = htonl((uint32_t)(c->len >> 29));
((uint32_t *)c->buf)[15] = htonl((uint32_t)c->len << 3);
/* Final hash round */
sha1_core(c->iv, c->buf, 1);
/* Copy hash result out */
for (i = 0; i < 5; i++)
put_be32(hash + 4*i, c->iv[i]);
}
/*
* Helper macros for sha1_core function. To avoid clutter, these macros
* are NOT fully parenthesized if it doesn't matter to the actual use.
*/
#if __GNUC__ && (defined(__i386__) || defined(__x86_64__))
/*
* GCC by itself only generates left rotates. Use right rotates if
* possible to be kinder to dinky implementations with iterative rotate
* instructions.
*/
#define ROT(op, x, k) \
({ uint32_t y; asm(op " %1,%0" : "=r" (y) : "I" (k), "0" (x)); y; })
#define ROL(x,k) ROT("roll", x, k)
#define ROR(x,k) ROT("rorl", x, k)
#else
/* Generic C equivalent */
#define ROT(x,l,r) ((x) << (l) | (x) >> (r))
#define ROL(x,k) ROT(x,k,32-(k))
#define ROR(x,k) ROT(x,32-(k),k)
#endif
/*
* An important temporary array in SHA-1 is the working array W[],
* which holds a scheduled key word per round. Only the last 16 words
* are relevant, so only use 16 words...
*/
#define W(i) w[(i) & 15]
/*
* If you have a small register set, it helps GCC to force stores to
* the w[] array to memory. Given 22 or more registers (e.g. PowerPC),
* GCC can map the entire w[] array to registers and this becomes
* counterproductive.
*
* The optimal kludge seems to differ between x86 and ARM. On the latter,
* forcing a full memory barrier is required to stop GCC from splitting
* live ranges for each round and generating a separate stack slot for
* each. (Which produces a huge stack frame and kills performance.)
*/
#if defined(__i386__) || defined(__x86_64__)
#define STORE(i, x) *(volatile uint32_t *)&W(i) = x
#elif __GNUC__ && defined(__arm__)
#define STORE(i, x) W(i) = x; __asm__("":::"memory")
#else
#define STORE(i, x) W(i) = x
#endif
/* The three round functions. F2 is also used as F4 */
#define F1(b,c,d) (((d ^ c) & b) ^ d) /* Bitwise b ? c : d */
#define F2(b,c,d) (d ^ c ^ b) /* Even parity */
#define F3(b,c,d) (d & c) + ((d ^ c) & b) /* Majority function */
/* The four round constants */
#define K1 0x5a827999 /* 2^30 * sqrt(2) */
#define K2 0x6ed9eba1 /* 2^30 * sqrt(3) */
#define K3 0x8f1bbcdc /* 2^30 * sqrt(5) */
#define K4 0xca62c1d6 /* 2^30 * sqrt(10) */
/* Rounds 0..15 fetch a word from the input */
#define FETCH(t,i) t = get_be32(p + 4*(i)); STORE(i,t)
/* Rounds 16..79 mix previous words to get a new one */
#define MIX(t,i) t = W(i) ^ W(i+2) ^ W(i+8) ^ W(i+13); t = ROL(t, 1)
/* Rounds 16..76 have to store back the result */
#define CALC(t,i) MIX(t,i); STORE(i,t)
/* The basic SHA-1 round */
#define SHA_ROUND(a,b,c,d,e,f,k,src,t,i) \
src(t,i); \
e += t + f(b,c,d) + k + ROL(a,5); \
b = ROR(b,2)
/* An aligned group of 5 rounds */
#define SHA_ROUND5(f,k,src,t,i) \
SHA_ROUND(a,b,c,d,e, f,k,src,t,i); \
SHA_ROUND(e,a,b,c,d, f,k,src,t,i+1); \
SHA_ROUND(d,e,a,b,c, f,k,src,t,i+2); \
SHA_ROUND(c,d,e,a,b, f,k,src,t,i+3); \
SHA_ROUND(b,c,d,e,a, f,k,src,t,i+4)
/*
* The core SHA-1 transform.
*
* iv[5] = Current SHA-1 hash state.
* p = Pointer to source data. Not necessarily aligned.
* nblocks = Number of 64-byte blocks at p. Guaranteed non-zero.
*/
static void
sha1_core(uint32_t iv[5], unsigned char const *p, size_t nblocks)
{
uint32_t e = iv[4], d = iv[3], c = iv[2], b = iv[1], a = iv[0];
uint32_t w[16];
do {
uint32_t t;
SHA_ROUND5(F1, K1, FETCH, t, 0);
SHA_ROUND5(F1, K1, FETCH, t, 5);
SHA_ROUND5(F1, K1, FETCH, t, 10);
SHA_ROUND(a,b,c,d,e, F1, K1, FETCH, t, 15);
SHA_ROUND(e,a,b,c,d, F1, K1, CALC, t, 16);
SHA_ROUND(d,e,a,b,c, F1, K1, CALC, t, 17);
SHA_ROUND(c,d,e,a,b, F1, K1, CALC, t, 18);
SHA_ROUND(b,c,d,e,a, F1, K1, CALC, t, 19);
SHA_ROUND5(F2, K2, CALC, t, 20);
SHA_ROUND5(F2, K2, CALC, t, 25);
SHA_ROUND5(F2, K2, CALC, t, 30);
SHA_ROUND5(F2, K2, CALC, t, 35);
SHA_ROUND5(F3, K3, CALC, t, 40);
SHA_ROUND5(F3, K3, CALC, t, 45);
SHA_ROUND5(F3, K3, CALC, t, 50);
SHA_ROUND5(F3, K3, CALC, t, 55);
SHA_ROUND5(F2, K4, CALC, t, 60);
SHA_ROUND5(F2, K4, CALC, t, 65);
SHA_ROUND5(F2, K4, CALC, t, 70);
SHA_ROUND(a,b,c,d,e, F2, K4, CALC, t, 75);
SHA_ROUND(e,a,b,c,d, F2, K4, CALC, t, 76);
/* Last 3 rounds don't need to store mixed value */
SHA_ROUND(d,e,a,b,c, F2, K4, MIX, t, 77);
SHA_ROUND(c,d,e,a,b, F2, K4, MIX, t, 78);
SHA_ROUND(b,c,d,e,a, F2, K4, MIX, t, 79);
iv[4] = e += iv[4];
iv[3] = d += iv[3];
iv[2] = c += iv[2];
iv[1] = b += iv[1];
iv[0] = a += iv[0];
} while (--nblocks);
}
/* Compile with -DUNITTEST for self-tests */
#if UNITTEST
#include <stdio.h>
/* Known answer test */
static void katest(char const *text, size_t len, unsigned char const hash[20])
{
SHA_CTX c;
unsigned char hash2[20];
int i;
SHA1_Init(&c);
SHA1_Update(&c, text, len);
SHA1_Final(hash2, &c);
for (i = 0; i < 20; i++)
if (hash[i] != hash2[i])
goto mismatch;
printf("%u-byte known answer test PASSED\n", len);
return;
mismatch:
printf("%u-byte known answer test FAILED:\n", len);
printf("Input: \"%.*s\"\n", len, text);
printf("Computed:");
for (i = 0; i < 20; i++)
printf(" %02x", hash2[i]);
printf("\nExpected:");
for (i = 0; i < 20; i++)
printf(" %02x", hash[i]);
putchar('\n');
}
int
main(void)
{
/* FIPS PUB 180.1 example A.1 */
static char const text1[3] = "abc";
static unsigned char const hash1[20] = {
0xa9, 0x99, 0x3e, 0x36, 0x47, 0x06, 0x81, 0x6a, 0xba, 0x3e,
0x25, 0x71, 0x78, 0x50, 0xc2, 0x6c, 0x9c, 0xd0, 0xd8, 0x9d };
/* FIPS PUB 180.1 example A.2 */
static char const text2[56] =
"abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq";
static unsigned char const hash2[20] = {
0x84, 0x98, 0x3e, 0x44, 0x1c, 0x3b, 0xd2, 0x6e, 0xba, 0xae,
0x4a, 0xa1, 0xf9, 0x51, 0x29, 0xe5, 0xe5, 0x46, 0x70, 0xf1 };
katest(text1, sizeof text1, hash1);
katest(text2, sizeof text2, hash2);
return 0;
}
#endif
^ permalink raw reply
* Re: How to stop sharing objects between repositories
From: Mike Galbraith @ 2009-08-17 7:12 UTC (permalink / raw)
To: Jeff King; +Cc: Junio C Hamano, Johannes Schindelin, Jon Jensen, git
In-Reply-To: <20090817064801.GA31543@coredump.intra.peff.net>
On Mon, 2009-08-17 at 02:48 -0400, Jeff King wrote:
> On Mon, Aug 17, 2009 at 04:21:22AM +0200, Mike Galbraith wrote:
>
> > > (1) Such a user does not necessarily know a casual "git repack -a" breaks
> > > the dependency, defeating the -s option s/he deliberately used in
> > > order to save disk space in the first place. Perhaps we can reword
> > > this further to kill two penguins with a single stone?
> >
> > Perhaps a runtime warning that you're about to break it? This user may
> > not even be the one who set the thing up, no?
>
> I'm not really sure what such a setup would look like. If it is a big
> hosting site like kernel.org or repo.or.cz, then probably it wouldn't
> matter much. The admins there should probably be running "git repack -l
> -d -A" periodically to consolidate the object stores (which can happen
> from this sort of repacking, or from people just pushing the same
> commits to their repos).
>
> That being said, I can see there being setups where such a warning might
> be useful. However, we don't really know if the user _wants_ that
> effect, or if it is an accident. So people following the recommnded
> "here is how you break the dependency" advice will also get the warning.
>
> I'm torn on whether this is actually a good idea.
Yeah. There are any number of ways to shoot oneself in the foot, and
while on the one hand idiot proofing can be nice if you were about to
screw up, "really really?" messages are most frequently annoying noise.
-Mike
^ permalink raw reply
* Re: "make quick-install-man" broke recently
From: Junio C Hamano @ 2009-08-17 6:53 UTC (permalink / raw)
To: Jacob Helwig
Cc: Linus Torvalds, Kjetil Barvik, Randal L. Schwartz,
git@vger.kernel.org
In-Reply-To: <8c9a060908162258ua50e34ah53c1ada9dbcd4aa0@mail.gmail.com>
Jacob Helwig <jacob.helwig@gmail.com> writes:
> I was able to reproduce this one one of the machines I have access to,
> and bisecting shows that it was introduced by b6986d8 (git-checkout: be
> careful about untracked symlinks).
Ahh..
-- >8 --
check_path(): allow symlinked directories to checkout-index --prefix
Merlyn noticed that Documentation/install-doc-quick.sh no longer correctly
removes old installed documents when the target directory has a leading
path that is a symlink. It turns out that "checkout-index --prefix" was
broken by recent b6986d8 (git-checkout: be careful about untracked
symlinks, 2009-07-29).
I suspect has_symlink_leading_path() could learn the third parameter
(prefix that is allowed to be symlinked directories) to allow us to retire
a similar function has_dirs_only_path().
Another avenue of fixing this I considered was to get rid of base_dir and
base_dir_len from "struct checkout", and instead make "git checkout-index"
when run with --prefix mkdir the leading path and chdir in there. It
might be the best longer term solution to this issue, as the base_dir
feature is used only by that rather obscure codepath as far as I know.
But at least this patch should fix this breakage.
SIgned-off-by: Junio C Hamano <gitster@pobox.com>
---
cache.h | 3 ---
entry.c | 15 ++++++++++++---
t/t2000-checkout-cache-clash.sh | 9 +++++++++
3 files changed, 21 insertions(+), 6 deletions(-)
diff --git a/cache.h b/cache.h
index 9222774..e6c7f33 100644
--- a/cache.h
+++ b/cache.h
@@ -468,9 +468,6 @@ extern int index_fd(unsigned char *sha1, int fd, struct stat *st, int write_obje
extern int index_path(unsigned char *sha1, const char *path, struct stat *st, int write_object);
extern void fill_stat_cache_info(struct cache_entry *ce, struct stat *st);
-/* "careful lstat()" */
-extern int check_path(const char *path, int len, struct stat *st);
-
#define REFRESH_REALLY 0x0001 /* ignore_valid */
#define REFRESH_UNMERGED 0x0002 /* allow unmerged */
#define REFRESH_QUIET 0x0004 /* be quiet about it */
diff --git a/entry.c b/entry.c
index f276cf3..6813f8a 100644
--- a/entry.c
+++ b/entry.c
@@ -179,9 +179,18 @@ static int write_entry(struct cache_entry *ce, char *path, const struct checkout
* This is like 'lstat()', except it refuses to follow symlinks
* in the path.
*/
-int check_path(const char *path, int len, struct stat *st)
+static int check_path(const char *path, int len, struct stat *st,
+ const struct checkout *co)
{
- if (has_symlink_leading_path(path, len)) {
+ if (co->base_dir_len) {
+ const char *slash = path + len;
+ while (path < slash && *slash != '/')
+ slash--;
+ if (!has_dirs_only_path(path, slash-path, co->base_dir_len)) {
+ errno = ENOENT;
+ return -1;
+ }
+ } else if (has_symlink_leading_path(path, len)) {
errno = ENOENT;
return -1;
}
@@ -201,7 +210,7 @@ int checkout_entry(struct cache_entry *ce, const struct checkout *state, char *t
strcpy(path + len, ce->name);
len += ce_namelen(ce);
- if (!check_path(path, len, &st)) {
+ if (!check_path(path, len, &st, state)) {
unsigned changed = ce_match_stat(ce, &st, CE_MATCH_IGNORE_VALID);
if (!changed)
return 0;
diff --git a/t/t2000-checkout-cache-clash.sh b/t/t2000-checkout-cache-clash.sh
index f7e1a73..3edade0 100755
--- a/t/t2000-checkout-cache-clash.sh
+++ b/t/t2000-checkout-cache-clash.sh
@@ -48,4 +48,13 @@ test_expect_success \
'git checkout-index conflicting paths.' \
'test -f path0 && test -d path1 && test -f path1/file1'
+test_expect_success 'checkout-index -f twice with --prefix' '
+ mkdir -p tar/get &&
+ ln -s tar/get there &&
+ echo first &&
+ git checkout-index -a -f --prefix=there/ &&
+ echo second &&
+ git checkout-index -a -f --prefix=there/
+'
+
test_done
^ permalink raw reply related
* Re: How to stop sharing objects between repositories
From: Jeff King @ 2009-08-17 6:48 UTC (permalink / raw)
To: Mike Galbraith; +Cc: Junio C Hamano, Johannes Schindelin, Jon Jensen, git
In-Reply-To: <1250475682.7155.16.camel@marge.simson.net>
On Mon, Aug 17, 2009 at 04:21:22AM +0200, Mike Galbraith wrote:
> > (1) Such a user does not necessarily know a casual "git repack -a" breaks
> > the dependency, defeating the -s option s/he deliberately used in
> > order to save disk space in the first place. Perhaps we can reword
> > this further to kill two penguins with a single stone?
>
> Perhaps a runtime warning that you're about to break it? This user may
> not even be the one who set the thing up, no?
I'm not really sure what such a setup would look like. If it is a big
hosting site like kernel.org or repo.or.cz, then probably it wouldn't
matter much. The admins there should probably be running "git repack -l
-d -A" periodically to consolidate the object stores (which can happen
from this sort of repacking, or from people just pushing the same
commits to their repos).
That being said, I can see there being setups where such a warning might
be useful. However, we don't really know if the user _wants_ that
effect, or if it is an accident. So people following the recommnded
"here is how you break the dependency" advice will also get the warning.
I'm torn on whether this is actually a good idea.
-- >8 --
Subject: [PATCH] repack: warn when "-l" is not used with alternates
Failing to use "-l" means that we will copy objects from the
source repository, nullifying the usefulness of "-s". We
don't want to make this an error, though, since "git repack
-a" is used to intentionally break the dependency.
Signed-off-by: Jeff King <peff@peff.net>
---
Is "test -s" portable? It's in POSIX, but I have some lingering doubt in
the back of my mind.
A user seeing such a warning can perhaps ^C to abort the pack. However,
should we also give instructions on how to undo the copying (which
should be "git repack -d -l -A")?
git-repack.sh | 9 +++++++++
1 files changed, 9 insertions(+), 0 deletions(-)
diff --git a/git-repack.sh b/git-repack.sh
index 1eb3bca..0bdc6e9 100755
--- a/git-repack.sh
+++ b/git-repack.sh
@@ -44,6 +44,15 @@ do
shift
done
+if test -z "$local" && test -s "$GIT_DIR/objects/info/alternates"; then
+cat >&2 <<'EOF'
+warning: this repository uses objects from other repositories via the
+warning: "alternates" mechanism; repacking without "-l" will cause objects
+warning: to be copied into this repository, wasting disk space.
+
+EOF
+fi
+
case "`git config --bool repack.usedeltabaseoffset || echo true`" in
true)
extra="$extra --delta-base-offset" ;;
--
1.6.4.283.ga2765.dirty
^ permalink raw reply related
* Re: How to stop sharing objects between repositories
From: Jeff King @ 2009-08-17 6:32 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Johannes Schindelin, Jon Jensen, git
In-Reply-To: <20090817061916.GA27530@coredump.intra.peff.net>
On Mon, Aug 17, 2009 at 02:19:16AM -0400, Jeff King wrote:
> 1. "git repack -a" is sufficient to break dependency, as it copies
> both packed and loose objects
>
> 2. "git repack" _may_ break the dependency, if there are no packs, as
> it copies only loose objects. It _may_ introduce inefficiency, but
> only if there are loose objects.
>
> 3. "git repack -l" always keeps the dependency and current efficiency.
>
> As an aside, making this list makes me realize there is no easy
> "keep the dependency and increase efficiency". In other words, pack
> everything that is not available otherwise, and then prune the
> remaining packs.
OK, I take it back. "git repack -d -l -A" will get rid of any packs that
are redundant with your alternate.
-Peff
^ permalink raw reply
* Re: How to stop sharing objects between repositories
From: Jeff King @ 2009-08-17 6:31 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Johannes Schindelin, Jon Jensen, git
In-Reply-To: <7vmy5z603d.fsf@alter.siamese.dyndns.org>
On Sun, Aug 16, 2009 at 12:16:22PM -0700, Junio C Hamano wrote:
> (2) IIRC, "git gc --auto" runs "repack -A". What is its effect with
> respect to this dependency between object stores? I suspect it would
> also break the dependency, but if so, is it a good thing? Perhaps
> should we change it to use a version that keeps the dependency
> instead?
No, it actually runs "repack -d -l -A", which behaves fine. I even
tested it to make sure.
BTW, the "gc.auto" setting is really annoying at low levels (I set
gc.auto to 1 for testing). For efficiency, it looks at only one
hashed object directory, and then assumes the other 255 contain roughly
the same number of objects. But you get bad sampling error when you have
fewer than 256 objects. I don't think it is worth caring about, though.
It doesn't seem very sane to set gc.auto to something so low.
-Peff
^ permalink raw reply
* Re: How to stop sharing objects between repositories
From: Jeff King @ 2009-08-17 6:19 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Johannes Schindelin, Jon Jensen, git
In-Reply-To: <7vmy5z603d.fsf@alter.siamese.dyndns.org>
On Sun, Aug 16, 2009 at 12:16:22PM -0700, Junio C Hamano wrote:
> After reading this, two points come to my mind. They may or may not be
> issues.
>
> (1) Such a user does not necessarily know a casual "git repack -a" breaks
> the dependency, defeating the -s option s/he deliberately used in
> order to save disk space in the first place. Perhaps we can reword
> this further to kill two penguins with a single stone?
>
> Note that the pack resulting from running `git repack -a` in the
> repository cloned with the `-s` option will include objects that
> are borrowed from the source repository. It essentially breaks
> the dependency created by cloning with the `-s` option by copying
> the objects from the source repository. To keep borrowing from
> the source repository to save disk space, do not use `repack -a`.
Good point, but I don't think this wording is quite right. You can also
cause such an inefficiency by simply running "git repack", if the source
has loose objects. In other words:
1. "git repack -a" is sufficient to break dependency, as it copies
both packed and loose objects
2. "git repack" _may_ break the dependency, if there are no packs, as
it copies only loose objects. It _may_ introduce inefficiency, but
only if there are loose objects.
3. "git repack -l" always keeps the dependency and current efficiency.
As an aside, making this list makes me realize there is no easy
"keep the dependency and increase efficiency". In other words, pack
everything that is not available otherwise, and then prune the
remaining packs.
Modified patch is below.
> We should suggest an alternative immediately after this sentence,
> e.g. "Instead, use `repack -l`" or something, but somebody should
> check if it is a valid/viable alternative.
It does work. From a user's perspective, I think "-l" would probably be
a more sane default. But I think it is off for historical reasons, and
these days we try to steer users towards "git gc", anyway, which does
use "-l" by default.
-- >8 --
Subject: [PATCH] docs: describe impact of repack on "clone -s"
The effects of repacking on a repository with alternates are
a bit subtle. The two main things users will want are:
1. Not to waste disk space by accidentally copying objects
which could be shared.
2. Copying all objects explicitly to break the dependency
on the source repo.
This patch describes both under the "clone -s"
documentation. It makes sense to put it there rather than in
git-repack.txt for both cases. For (1), we are warning the
user who is using "clone -s" about what _not_ to do, so we
need to get their attention when reading about "clone -s".
For (2), we are telling them how git-repack can be used to
accomplish a task, but until they know that git-repack is
the right tool, they have no reason to look at the repack
documentation.
Signed-off-by: Jeff King <peff@peff.net>
---
The extra deleted lines in the patch below are just cleaning up some
excess whitespace.
Documentation/git-clone.txt | 12 ++++++++++--
1 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index b14de6c..b25944f 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -72,8 +72,16 @@ These objects may be removed by normal git operations (such as 'git-commit')
which automatically call `git gc --auto`. (See linkgit:git-gc[1].)
If these objects are removed and were referenced by the cloned repository,
then the cloned repository will become corrupt.
-
-
++
+Note that running `git repack` without the `-l` option in a repository
+cloned with `-s` will copy objects from the source repository into a
+pack in the cloned repository, removing the disk space savings of `clone
+-s`. It is safe, however, to run `git gc`, which uses the `-l` option by
+default.
++
+If you want to break the dependency of a repository cloned with `-s` on
+its source repository, you can simply run `git repack -a` to copy all
+objects from the source repository into a pack in the cloned repository.
--reference <repository>::
If the reference repository is on the local machine
--
1.6.4.283.gec993
^ permalink raw reply related
* sparse support in pu
From: skillzero @ 2009-08-17 6:09 UTC (permalink / raw)
To: git
I had a couple comments and questions about the sparse stuff recently
merged into pu. First, I think it's an awesome feature. It's a nice
solution to my problems (1.5 GB tree where most people only need a
small subset, but with a lot of shared pieces).
1. Have people decided whether it should be on by default if you have
a .git/info/sparse file? I'd definitely like it to be on by default.
When I first tried it, I didn't realize I had to use --sparse to git
checkout to get it to use the sparse rules. The same goes for a merge
I did that happened to have a file in the excluded area (it included
it because I didn't use --sparse to git merge).
2. Is it not hooked up to git reset yet? I did a git checkout --sparse
and things look liked I expected then I did a git reset --hard
origin/master and it started checking out all the stuff previously
excluded via .git/info/sparse. I tried --sparse, but it didn't know
about that option.
3. One thing that was confusing is that I needed a trailing slash on
directories in .git/info/sparse to get them excluded. This seems
different than .gitignore, which works for me without the trailing
slash.
BTW...How to people normally try out stuff in pu? Do you just replace
your normal git stuff with the pu build? That's what I did because I
wasn't sure if I did /test/git if the other tools git invokes would
use the release version of git or the ones relative to itself (i.e.
the ones in /test/ for me). It seems like the only way to really test
stuff is to use it for your real work, but I also don't want to
corrupt our production repositories by using an experimental version
of git.
^ permalink raw reply
* Re: git http-push and MKCOL error (22/409)
From: Junio C Hamano @ 2009-08-17 6:08 UTC (permalink / raw)
To: Thomas Schlichter
Cc: Tay Ray Chuan, Junio C Hamano, willievu, Sean Davis, git
In-Reply-To: <200908170725.09592.thomas.schlichter@web.de>
Thomas Schlichter <thomas.schlichter@web.de> writes:
> Hmm. I hope you won't take this as an argument for not fixing it, because it
> is a clear regression! I tried git version 1.6.3.4, and it works flawlessly
> with exactly this server! Even during bisecting, some versions worked, some
> didn't (these after the mentioned commit...)
If it works (without changing anything else) with one version and with a
problematic patch it doesn't, it is a clear regression.
If you revert 5424bc5 (http*: add helper methods for fetching objects
(loose), 2009-06-06) from the tip of 'master', does it make the problem go
away?
The patch should look like something like the attached.
diff --git a/http-push.c b/http-push.c
index 00e83dc..d1ea702 100644
--- a/http-push.c
+++ b/http-push.c
@@ -115,10 +115,18 @@ struct transfer_request
struct remote_lock *lock;
struct curl_slist *headers;
struct buffer buffer;
+ char filename[PATH_MAX];
+ char tmpfile[PATH_MAX];
+ int local_fileno;
enum transfer_state state;
CURLcode curl_result;
char errorstr[CURL_ERROR_SIZE];
long http_code;
+ unsigned char real_sha1[20];
+ git_SHA_CTX c;
+ z_stream stream;
+ int zret;
+ int rename;
void *userData;
struct active_request_slot *slot;
struct transfer_request *next;
@@ -226,6 +234,15 @@ static struct curl_slist *get_dav_token_headers(struct remote_lock *lock, enum d
return dav_headers;
}
+static void append_remote_object_url(struct strbuf *buf, const char *url,
+ const char *hex,
+ int only_two_digit_prefix)
+{
+ strbuf_addf(buf, "%sobjects/%.*s/", url, 2, hex);
+ if (!only_two_digit_prefix)
+ strbuf_addf(buf, "%s", hex+2);
+}
+
static void finish_request(struct transfer_request *request);
static void release_request(struct transfer_request *request);
@@ -239,29 +256,169 @@ static void process_response(void *callback_data)
#ifdef USE_CURL_MULTI
+static char *get_remote_object_url(const char *url, const char *hex,
+ int only_two_digit_prefix)
+{
+ struct strbuf buf = STRBUF_INIT;
+ append_remote_object_url(&buf, url, hex, only_two_digit_prefix);
+ return strbuf_detach(&buf, NULL);
+}
+
+static size_t fwrite_sha1_file(void *ptr, size_t eltsize, size_t nmemb,
+ void *data)
+{
+ unsigned char expn[4096];
+ size_t size = eltsize * nmemb;
+ int posn = 0;
+ struct transfer_request *request = (struct transfer_request *)data;
+ do {
+ ssize_t retval = xwrite(request->local_fileno,
+ (char *) ptr + posn, size - posn);
+ if (retval < 0)
+ return posn;
+ posn += retval;
+ } while (posn < size);
+
+ request->stream.avail_in = size;
+ request->stream.next_in = ptr;
+ do {
+ request->stream.next_out = expn;
+ request->stream.avail_out = sizeof(expn);
+ request->zret = git_inflate(&request->stream, Z_SYNC_FLUSH);
+ git_SHA1_Update(&request->c, expn,
+ sizeof(expn) - request->stream.avail_out);
+ } while (request->stream.avail_in && request->zret == Z_OK);
+ data_received++;
+ return size;
+}
+
static void start_fetch_loose(struct transfer_request *request)
{
+ char *hex = sha1_to_hex(request->obj->sha1);
+ char *filename;
+ char prevfile[PATH_MAX];
+ char *url;
+ int prevlocal;
+ unsigned char prev_buf[PREV_BUF_SIZE];
+ ssize_t prev_read = 0;
+ long prev_posn = 0;
+ char range[RANGE_HEADER_SIZE];
+ struct curl_slist *range_header = NULL;
struct active_request_slot *slot;
- struct http_object_request *obj_req;
- obj_req = new_http_object_request(repo->url, request->obj->sha1);
- if (obj_req == NULL) {
+ filename = sha1_file_name(request->obj->sha1);
+ snprintf(request->filename, sizeof(request->filename), "%s", filename);
+ snprintf(request->tmpfile, sizeof(request->tmpfile),
+ "%s.temp", filename);
+
+ snprintf(prevfile, sizeof(prevfile), "%s.prev", request->filename);
+ unlink_or_warn(prevfile);
+ rename(request->tmpfile, prevfile);
+ unlink_or_warn(request->tmpfile);
+
+ if (request->local_fileno != -1)
+ error("fd leakage in start: %d", request->local_fileno);
+ request->local_fileno = open(request->tmpfile,
+ O_WRONLY | O_CREAT | O_EXCL, 0666);
+ /*
+ * This could have failed due to the "lazy directory creation";
+ * try to mkdir the last path component.
+ */
+ if (request->local_fileno < 0 && errno == ENOENT) {
+ char *dir = strrchr(request->tmpfile, '/');
+ if (dir) {
+ *dir = 0;
+ mkdir(request->tmpfile, 0777);
+ *dir = '/';
+ }
+ request->local_fileno = open(request->tmpfile,
+ O_WRONLY | O_CREAT | O_EXCL, 0666);
+ }
+
+ if (request->local_fileno < 0) {
request->state = ABORTED;
+ error("Couldn't create temporary file %s for %s: %s",
+ request->tmpfile, request->filename, strerror(errno));
return;
}
- slot = obj_req->slot;
+ memset(&request->stream, 0, sizeof(request->stream));
+
+ git_inflate_init(&request->stream);
+
+ git_SHA1_Init(&request->c);
+
+ url = get_remote_object_url(repo->url, hex, 0);
+ request->url = xstrdup(url);
+
+ /*
+ * If a previous temp file is present, process what was already
+ * fetched.
+ */
+ prevlocal = open(prevfile, O_RDONLY);
+ if (prevlocal != -1) {
+ do {
+ prev_read = xread(prevlocal, prev_buf, PREV_BUF_SIZE);
+ if (prev_read>0) {
+ if (fwrite_sha1_file(prev_buf,
+ 1,
+ prev_read,
+ request) == prev_read)
+ prev_posn += prev_read;
+ else
+ prev_read = -1;
+ }
+ } while (prev_read > 0);
+ close(prevlocal);
+ }
+ unlink_or_warn(prevfile);
+
+ /*
+ * Reset inflate/SHA1 if there was an error reading the previous temp
+ * file; also rewind to the beginning of the local file.
+ */
+ if (prev_read == -1) {
+ memset(&request->stream, 0, sizeof(request->stream));
+ git_inflate_init(&request->stream);
+ git_SHA1_Init(&request->c);
+ if (prev_posn>0) {
+ prev_posn = 0;
+ lseek(request->local_fileno, 0, SEEK_SET);
+ ftruncate(request->local_fileno, 0);
+ }
+ }
+
+ slot = get_active_slot();
slot->callback_func = process_response;
slot->callback_data = request;
request->slot = slot;
- request->userData = obj_req;
+
+ curl_easy_setopt(slot->curl, CURLOPT_FILE, request);
+ curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);
+ curl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, request->errorstr);
+ curl_easy_setopt(slot->curl, CURLOPT_URL, url);
+ curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);
+
+ /*
+ * If we have successfully processed data from a previous fetch
+ * attempt, only fetch the data we don't already have.
+ */
+ if (prev_posn>0) {
+ if (push_verbosely)
+ fprintf(stderr,
+ "Resuming fetch of object %s at byte %ld\n",
+ hex, prev_posn);
+ sprintf(range, "Range: bytes=%ld-", prev_posn);
+ range_header = curl_slist_append(range_header, range);
+ curl_easy_setopt(slot->curl,
+ CURLOPT_HTTPHEADER, range_header);
+ }
/* Try to get the request started, abort the request on error */
request->state = RUN_FETCH_LOOSE;
if (!start_active_slot(slot)) {
fprintf(stderr, "Unable to start GET request\n");
repo->can_update_info_refs = 0;
- release_http_object_request(obj_req);
release_request(request);
}
}
@@ -520,14 +677,16 @@ static void release_request(struct transfer_request *request)
entry->next = entry->next->next;
}
+ if (request->local_fileno != -1)
+ close(request->local_fileno);
free(request->url);
free(request);
}
static void finish_request(struct transfer_request *request)
{
+ struct stat st;
struct http_pack_request *preq;
- struct http_object_request *obj_req;
request->curl_result = request->slot->curl_result;
request->http_code = request->slot->http_code;
@@ -582,17 +741,39 @@ static void finish_request(struct transfer_request *request)
aborted = 1;
}
} else if (request->state == RUN_FETCH_LOOSE) {
- obj_req = (struct http_object_request *)request->userData;
-
- if (finish_http_object_request(obj_req) == 0)
- if (obj_req->rename == 0)
- request->obj->flags |= (LOCAL | REMOTE);
+ close(request->local_fileno);
+ request->local_fileno = -1;
+
+ if (request->curl_result != CURLE_OK &&
+ request->http_code != 416) {
+ if (stat(request->tmpfile, &st) == 0) {
+ if (st.st_size == 0)
+ unlink_or_warn(request->tmpfile);
+ }
+ } else {
+ if (request->http_code == 416)
+ warning("requested range invalid; we may already have all the data.");
+
+ git_inflate_end(&request->stream);
+ git_SHA1_Final(request->real_sha1, &request->c);
+ if (request->zret != Z_STREAM_END) {
+ unlink_or_warn(request->tmpfile);
+ } else if (hashcmp(request->obj->sha1, request->real_sha1)) {
+ unlink_or_warn(request->tmpfile);
+ } else {
+ request->rename =
+ move_temp_to_file(
+ request->tmpfile,
+ request->filename);
+ if (request->rename == 0)
+ request->obj->flags |= (LOCAL | REMOTE);
+ }
+ }
/* Try fetching packed if necessary */
- if (request->obj->flags & LOCAL) {
- release_http_object_request(obj_req);
+ if (request->obj->flags & LOCAL)
release_request(request);
- } else
+ else
start_fetch_packed(request);
} else if (request->state == RUN_FETCH_PACKED) {
@@ -664,6 +845,7 @@ static void add_fetch_request(struct object *obj)
request->url = NULL;
request->lock = NULL;
request->headers = NULL;
+ request->local_fileno = -1;
request->state = NEED_FETCH;
request->next = request_queue_head;
request_queue_head = request;
@@ -702,6 +884,7 @@ static int add_send_request(struct object *obj, struct remote_lock *lock)
request->url = NULL;
request->lock = lock;
request->headers = NULL;
+ request->local_fileno = -1;
request->state = NEED_PUSH;
request->next = request_queue_head;
request_queue_head = request;
diff --git a/http-walker.c b/http-walker.c
index 700bc13..8f7a975 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -3,6 +3,8 @@
#include "walker.h"
#include "http.h"
+#define PREV_BUF_SIZE 4096
+
struct alt_base
{
char *base;
@@ -23,8 +25,20 @@ struct object_request
struct walker *walker;
unsigned char sha1[20];
struct alt_base *repo;
+ char *url;
+ char filename[PATH_MAX];
+ char tmpfile[PATH_MAX];
+ int local;
enum object_request_state state;
- struct http_object_request *req;
+ CURLcode curl_result;
+ char errorstr[CURL_ERROR_SIZE];
+ long http_code;
+ unsigned char real_sha1[20];
+ git_SHA_CTX c;
+ z_stream stream;
+ int zret;
+ int rename;
+ struct active_request_slot *slot;
struct object_request *next;
};
@@ -45,6 +59,34 @@ struct walker_data {
static struct object_request *object_queue_head;
+static size_t fwrite_sha1_file(void *ptr, size_t eltsize, size_t nmemb,
+ void *data)
+{
+ unsigned char expn[4096];
+ size_t size = eltsize * nmemb;
+ int posn = 0;
+ struct object_request *obj_req = (struct object_request *)data;
+ do {
+ ssize_t retval = xwrite(obj_req->local,
+ (char *) ptr + posn, size - posn);
+ if (retval < 0)
+ return posn;
+ posn += retval;
+ } while (posn < size);
+
+ obj_req->stream.avail_in = size;
+ obj_req->stream.next_in = ptr;
+ do {
+ obj_req->stream.next_out = expn;
+ obj_req->stream.avail_out = sizeof(expn);
+ obj_req->zret = git_inflate(&obj_req->stream, Z_SYNC_FLUSH);
+ git_SHA1_Update(&obj_req->c, expn,
+ sizeof(expn) - obj_req->stream.avail_out);
+ } while (obj_req->stream.avail_in && obj_req->zret == Z_OK);
+ data_received++;
+ return size;
+}
+
static void fetch_alternates(struct walker *walker, const char *base);
static void process_object_response(void *callback_data);
@@ -52,35 +94,172 @@ static void process_object_response(void *callback_data);
static void start_object_request(struct walker *walker,
struct object_request *obj_req)
{
+ char *hex = sha1_to_hex(obj_req->sha1);
+ char prevfile[PATH_MAX];
+ char *url;
+ char *posn;
+ int prevlocal;
+ unsigned char prev_buf[PREV_BUF_SIZE];
+ ssize_t prev_read = 0;
+ long prev_posn = 0;
+ char range[RANGE_HEADER_SIZE];
+ struct curl_slist *range_header = NULL;
struct active_request_slot *slot;
- struct http_object_request *req;
- req = new_http_object_request(obj_req->repo->base, obj_req->sha1);
- if (req == NULL) {
+ snprintf(prevfile, sizeof(prevfile), "%s.prev", obj_req->filename);
+ unlink_or_warn(prevfile);
+ rename(obj_req->tmpfile, prevfile);
+ unlink_or_warn(obj_req->tmpfile);
+
+ if (obj_req->local != -1)
+ error("fd leakage in start: %d", obj_req->local);
+ obj_req->local = open(obj_req->tmpfile,
+ O_WRONLY | O_CREAT | O_EXCL, 0666);
+ /*
+ * This could have failed due to the "lazy directory creation";
+ * try to mkdir the last path component.
+ */
+ if (obj_req->local < 0 && errno == ENOENT) {
+ char *dir = strrchr(obj_req->tmpfile, '/');
+ if (dir) {
+ *dir = 0;
+ mkdir(obj_req->tmpfile, 0777);
+ *dir = '/';
+ }
+ obj_req->local = open(obj_req->tmpfile,
+ O_WRONLY | O_CREAT | O_EXCL, 0666);
+ }
+
+ if (obj_req->local < 0) {
obj_req->state = ABORTED;
+ error("Couldn't create temporary file %s for %s: %s",
+ obj_req->tmpfile, obj_req->filename, strerror(errno));
return;
}
- obj_req->req = req;
- slot = req->slot;
+ memset(&obj_req->stream, 0, sizeof(obj_req->stream));
+
+ git_inflate_init(&obj_req->stream);
+
+ git_SHA1_Init(&obj_req->c);
+
+ url = xmalloc(strlen(obj_req->repo->base) + 51);
+ obj_req->url = xmalloc(strlen(obj_req->repo->base) + 51);
+ strcpy(url, obj_req->repo->base);
+ posn = url + strlen(obj_req->repo->base);
+ strcpy(posn, "/objects/");
+ posn += 9;
+ memcpy(posn, hex, 2);
+ posn += 2;
+ *(posn++) = '/';
+ strcpy(posn, hex + 2);
+ strcpy(obj_req->url, url);
+
+ /*
+ * If a previous temp file is present, process what was already
+ * fetched.
+ */
+ prevlocal = open(prevfile, O_RDONLY);
+ if (prevlocal != -1) {
+ do {
+ prev_read = xread(prevlocal, prev_buf, PREV_BUF_SIZE);
+ if (prev_read>0) {
+ if (fwrite_sha1_file(prev_buf,
+ 1,
+ prev_read,
+ obj_req) == prev_read)
+ prev_posn += prev_read;
+ else
+ prev_read = -1;
+ }
+ } while (prev_read > 0);
+ close(prevlocal);
+ }
+ unlink_or_warn(prevfile);
+
+ /*
+ * Reset inflate/SHA1 if there was an error reading the previous temp
+ * file; also rewind to the beginning of the local file.
+ */
+ if (prev_read == -1) {
+ memset(&obj_req->stream, 0, sizeof(obj_req->stream));
+ git_inflate_init(&obj_req->stream);
+ git_SHA1_Init(&obj_req->c);
+ if (prev_posn>0) {
+ prev_posn = 0;
+ lseek(obj_req->local, 0, SEEK_SET);
+ ftruncate(obj_req->local, 0);
+ }
+ }
+
+ slot = get_active_slot();
slot->callback_func = process_object_response;
slot->callback_data = obj_req;
+ obj_req->slot = slot;
+
+ curl_easy_setopt(slot->curl, CURLOPT_FILE, obj_req);
+ curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);
+ curl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, obj_req->errorstr);
+ curl_easy_setopt(slot->curl, CURLOPT_URL, url);
+ curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);
+
+ /*
+ * If we have successfully processed data from a previous fetch
+ * attempt, only fetch the data we don't already have.
+ */
+ if (prev_posn>0) {
+ if (walker->get_verbosely)
+ fprintf(stderr,
+ "Resuming fetch of object %s at byte %ld\n",
+ hex, prev_posn);
+ sprintf(range, "Range: bytes=%ld-", prev_posn);
+ range_header = curl_slist_append(range_header, range);
+ curl_easy_setopt(slot->curl,
+ CURLOPT_HTTPHEADER, range_header);
+ }
/* Try to get the request started, abort the request on error */
obj_req->state = ACTIVE;
if (!start_active_slot(slot)) {
obj_req->state = ABORTED;
- release_http_object_request(req);
+ obj_req->slot = NULL;
+ close(obj_req->local);
+ obj_req->local = -1;
+ free(obj_req->url);
return;
}
}
static void finish_object_request(struct object_request *obj_req)
{
- if (finish_http_object_request(obj_req->req))
+ struct stat st;
+
+ close(obj_req->local);
+ obj_req->local = -1;
+
+ if (obj_req->http_code == 416) {
+ fprintf(stderr, "Warning: requested range invalid; we may already have all the data.\n");
+ } else if (obj_req->curl_result != CURLE_OK) {
+ if (stat(obj_req->tmpfile, &st) == 0)
+ if (st.st_size == 0)
+ unlink_or_warn(obj_req->tmpfile);
return;
+ }
+
+ git_inflate_end(&obj_req->stream);
+ git_SHA1_Final(obj_req->real_sha1, &obj_req->c);
+ if (obj_req->zret != Z_STREAM_END) {
+ unlink_or_warn(obj_req->tmpfile);
+ return;
+ }
+ if (hashcmp(obj_req->sha1, obj_req->real_sha1)) {
+ unlink_or_warn(obj_req->tmpfile);
+ return;
+ }
+ obj_req->rename =
+ move_temp_to_file(obj_req->tmpfile, obj_req->filename);
- if (obj_req->req->rename == 0)
+ if (obj_req->rename == 0)
walker_say(obj_req->walker, "got %s\n", sha1_to_hex(obj_req->sha1));
}
@@ -92,16 +271,19 @@ static void process_object_response(void *callback_data)
struct walker_data *data = walker->data;
struct alt_base *alt = data->alt;
- process_http_object_request(obj_req->req);
+ obj_req->curl_result = obj_req->slot->curl_result;
+ obj_req->http_code = obj_req->slot->http_code;
+ obj_req->slot = NULL;
obj_req->state = COMPLETE;
/* Use alternates if necessary */
- if (missing_target(obj_req->req)) {
+ if (missing_target(obj_req)) {
fetch_alternates(walker, alt->base);
if (obj_req->repo->next != NULL) {
obj_req->repo =
obj_req->repo->next;
- release_http_object_request(obj_req->req);
+ close(obj_req->local);
+ obj_req->local = -1;
start_object_request(walker, obj_req);
return;
}
@@ -114,8 +296,8 @@ static void release_object_request(struct object_request *obj_req)
{
struct object_request *entry = object_queue_head;
- if (obj_req->req !=NULL && obj_req->req->localfile != -1)
- error("fd leakage in release: %d", obj_req->req->localfile);
+ if (obj_req->local != -1)
+ error("fd leakage in release: %d", obj_req->local);
if (obj_req == object_queue_head) {
object_queue_head = obj_req->next;
} else {
@@ -125,6 +307,7 @@ static void release_object_request(struct object_request *obj_req)
entry->next = entry->next->next;
}
+ free(obj_req->url);
free(obj_req);
}
@@ -152,13 +335,19 @@ static void prefetch(struct walker *walker, unsigned char *sha1)
struct object_request *newreq;
struct object_request *tail;
struct walker_data *data = walker->data;
+ char *filename = sha1_file_name(sha1);
newreq = xmalloc(sizeof(*newreq));
newreq->walker = walker;
hashcpy(newreq->sha1, sha1);
newreq->repo = data->alt;
+ newreq->url = NULL;
+ newreq->local = -1;
newreq->state = WAITING;
- newreq->req = NULL;
+ snprintf(newreq->filename, sizeof(newreq->filename), "%s", filename);
+ snprintf(newreq->tmpfile, sizeof(newreq->tmpfile),
+ "%s.temp", filename);
+ newreq->slot = NULL;
newreq->next = NULL;
http_is_verbose = walker->get_verbosely;
@@ -449,6 +638,15 @@ abort:
static void abort_object_request(struct object_request *obj_req)
{
+ if (obj_req->local >= 0) {
+ close(obj_req->local);
+ obj_req->local = -1;
+ }
+ unlink_or_warn(obj_req->tmpfile);
+ if (obj_req->slot) {
+ release_active_slot(obj_req->slot);
+ obj_req->slot = NULL;
+ }
release_object_request(obj_req);
}
@@ -457,7 +655,6 @@ static int fetch_object(struct walker *walker, struct alt_base *repo, unsigned c
char *hex = sha1_to_hex(sha1);
int ret = 0;
struct object_request *obj_req = object_queue_head;
- struct http_object_request *req;
while (obj_req != NULL && hashcmp(obj_req->sha1, sha1))
obj_req = obj_req->next;
@@ -465,8 +662,6 @@ static int fetch_object(struct walker *walker, struct alt_base *repo, unsigned c
return error("Couldn't find request for %s in the queue", hex);
if (has_sha1_file(obj_req->sha1)) {
- if (obj_req->req != NULL)
- abort_http_object_request(obj_req->req);
abort_object_request(obj_req);
return 0;
}
@@ -478,42 +673,34 @@ static int fetch_object(struct walker *walker, struct alt_base *repo, unsigned c
start_object_request(walker, obj_req);
#endif
- /*
- * obj_req->req might change when fetching alternates in the callback
- * process_object_response; therefore, the "shortcut" variable, req,
- * is used only after we're done with slots.
- */
while (obj_req->state == ACTIVE)
- run_active_slot(obj_req->req->slot);
-
- req = obj_req->req;
+ run_active_slot(obj_req->slot);
- if (req->localfile != -1) {
- close(req->localfile);
- req->localfile = -1;
+ if (obj_req->local != -1) {
+ close(obj_req->local);
+ obj_req->local = -1;
}
if (obj_req->state == ABORTED) {
ret = error("Request for %s aborted", hex);
- } else if (req->curl_result != CURLE_OK &&
- req->http_code != 416) {
- if (missing_target(req))
+ } else if (obj_req->curl_result != CURLE_OK &&
+ obj_req->http_code != 416) {
+ if (missing_target(obj_req))
ret = -1; /* Be silent, it is probably in a pack. */
else
ret = error("%s (curl_result = %d, http_code = %ld, sha1 = %s)",
- req->errorstr, req->curl_result,
- req->http_code, hex);
- } else if (req->zret != Z_STREAM_END) {
+ obj_req->errorstr, obj_req->curl_result,
+ obj_req->http_code, hex);
+ } else if (obj_req->zret != Z_STREAM_END) {
walker->corrupt_object_found++;
- ret = error("File %s (%s) corrupt", hex, req->url);
- } else if (hashcmp(obj_req->sha1, req->real_sha1)) {
+ ret = error("File %s (%s) corrupt", hex, obj_req->url);
+ } else if (hashcmp(obj_req->sha1, obj_req->real_sha1)) {
ret = error("File %s has bad hash", hex);
- } else if (req->rename < 0) {
+ } else if (obj_req->rename < 0) {
ret = error("unable to write sha1 filename %s",
- req->filename);
+ obj_req->filename);
}
- release_http_object_request(req);
release_object_request(obj_req);
return ret;
}
diff --git a/http.c b/http.c
index 14d5357..ad7b6f8 100644
--- a/http.c
+++ b/http.c
@@ -12,10 +12,6 @@ static CURLM *curlm;
#ifndef NO_CURL_EASY_DUPHANDLE
static CURL *curl_default;
#endif
-
-#define PREV_BUF_SIZE 4096
-#define RANGE_HEADER_SIZE 30
-
char curl_errorstr[CURL_ERROR_SIZE];
static int curl_ssl_verify = -1;
@@ -45,7 +41,8 @@ static char *ssl_cert_password;
static int ssl_cert_password_required;
static struct curl_slist *pragma_header;
-static struct curl_slist *no_pragma_header;
+
+struct curl_slist *no_pragma_header;
static struct active_request_slot *active_queue_head;
@@ -715,23 +712,6 @@ static char *quote_ref_url(const char *base, const char *ref)
return strbuf_detach(&buf, NULL);
}
-void append_remote_object_url(struct strbuf *buf, const char *url,
- const char *hex,
- int only_two_digit_prefix)
-{
- strbuf_addf(buf, "%s/objects/%.*s/", url, 2, hex);
- if (!only_two_digit_prefix)
- strbuf_addf(buf, "%s", hex+2);
-}
-
-char *get_remote_object_url(const char *url, const char *hex,
- int only_two_digit_prefix)
-{
- struct strbuf buf = STRBUF_INIT;
- append_remote_object_url(&buf, url, hex, only_two_digit_prefix);
- return strbuf_detach(&buf, NULL);
-}
-
/* http_request() targets */
#define HTTP_REQUEST_STRBUF 0
#define HTTP_REQUEST_FILE 1
@@ -1061,231 +1041,3 @@ abort:
free(preq);
return NULL;
}
-
-/* Helpers for fetching objects (loose) */
-static size_t fwrite_sha1_file(void *ptr, size_t eltsize, size_t nmemb,
- void *data)
-{
- unsigned char expn[4096];
- size_t size = eltsize * nmemb;
- int posn = 0;
- struct http_object_request *freq =
- (struct http_object_request *)data;
- do {
- ssize_t retval = xwrite(freq->localfile,
- (char *) ptr + posn, size - posn);
- if (retval < 0)
- return posn;
- posn += retval;
- } while (posn < size);
-
- freq->stream.avail_in = size;
- freq->stream.next_in = ptr;
- do {
- freq->stream.next_out = expn;
- freq->stream.avail_out = sizeof(expn);
- freq->zret = git_inflate(&freq->stream, Z_SYNC_FLUSH);
- git_SHA1_Update(&freq->c, expn,
- sizeof(expn) - freq->stream.avail_out);
- } while (freq->stream.avail_in && freq->zret == Z_OK);
- data_received++;
- return size;
-}
-
-struct http_object_request *new_http_object_request(const char *base_url,
- unsigned char *sha1)
-{
- char *hex = sha1_to_hex(sha1);
- char *filename;
- char prevfile[PATH_MAX];
- int prevlocal;
- unsigned char prev_buf[PREV_BUF_SIZE];
- ssize_t prev_read = 0;
- long prev_posn = 0;
- char range[RANGE_HEADER_SIZE];
- struct curl_slist *range_header = NULL;
- struct http_object_request *freq;
-
- freq = xmalloc(sizeof(*freq));
- hashcpy(freq->sha1, sha1);
- freq->localfile = -1;
-
- filename = sha1_file_name(sha1);
- snprintf(freq->filename, sizeof(freq->filename), "%s", filename);
- snprintf(freq->tmpfile, sizeof(freq->tmpfile),
- "%s.temp", filename);
-
- snprintf(prevfile, sizeof(prevfile), "%s.prev", filename);
- unlink_or_warn(prevfile);
- rename(freq->tmpfile, prevfile);
- unlink_or_warn(freq->tmpfile);
-
- if (freq->localfile != -1)
- error("fd leakage in start: %d", freq->localfile);
- freq->localfile = open(freq->tmpfile,
- O_WRONLY | O_CREAT | O_EXCL, 0666);
- /*
- * This could have failed due to the "lazy directory creation";
- * try to mkdir the last path component.
- */
- if (freq->localfile < 0 && errno == ENOENT) {
- char *dir = strrchr(freq->tmpfile, '/');
- if (dir) {
- *dir = 0;
- mkdir(freq->tmpfile, 0777);
- *dir = '/';
- }
- freq->localfile = open(freq->tmpfile,
- O_WRONLY | O_CREAT | O_EXCL, 0666);
- }
-
- if (freq->localfile < 0) {
- error("Couldn't create temporary file %s for %s: %s",
- freq->tmpfile, freq->filename, strerror(errno));
- goto abort;
- }
-
- memset(&freq->stream, 0, sizeof(freq->stream));
-
- git_inflate_init(&freq->stream);
-
- git_SHA1_Init(&freq->c);
-
- freq->url = get_remote_object_url(base_url, hex, 0);
-
- /*
- * If a previous temp file is present, process what was already
- * fetched.
- */
- prevlocal = open(prevfile, O_RDONLY);
- if (prevlocal != -1) {
- do {
- prev_read = xread(prevlocal, prev_buf, PREV_BUF_SIZE);
- if (prev_read>0) {
- if (fwrite_sha1_file(prev_buf,
- 1,
- prev_read,
- freq) == prev_read) {
- prev_posn += prev_read;
- } else {
- prev_read = -1;
- }
- }
- } while (prev_read > 0);
- close(prevlocal);
- }
- unlink_or_warn(prevfile);
-
- /*
- * Reset inflate/SHA1 if there was an error reading the previous temp
- * file; also rewind to the beginning of the local file.
- */
- if (prev_read == -1) {
- memset(&freq->stream, 0, sizeof(freq->stream));
- git_inflate_init(&freq->stream);
- git_SHA1_Init(&freq->c);
- if (prev_posn>0) {
- prev_posn = 0;
- lseek(freq->localfile, 0, SEEK_SET);
- if (ftruncate(freq->localfile, 0) < 0) {
- error("Couldn't truncate temporary file %s for %s: %s",
- freq->tmpfile, freq->filename, strerror(errno));
- goto abort;
- }
- }
- }
-
- freq->slot = get_active_slot();
-
- curl_easy_setopt(freq->slot->curl, CURLOPT_FILE, freq);
- curl_easy_setopt(freq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);
- curl_easy_setopt(freq->slot->curl, CURLOPT_ERRORBUFFER, freq->errorstr);
- curl_easy_setopt(freq->slot->curl, CURLOPT_URL, freq->url);
- curl_easy_setopt(freq->slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);
-
- /*
- * If we have successfully processed data from a previous fetch
- * attempt, only fetch the data we don't already have.
- */
- if (prev_posn>0) {
- if (http_is_verbose)
- fprintf(stderr,
- "Resuming fetch of object %s at byte %ld\n",
- hex, prev_posn);
- sprintf(range, "Range: bytes=%ld-", prev_posn);
- range_header = curl_slist_append(range_header, range);
- curl_easy_setopt(freq->slot->curl,
- CURLOPT_HTTPHEADER, range_header);
- }
-
- return freq;
-
-abort:
- free(filename);
- free(freq->url);
- free(freq);
- return NULL;
-}
-
-void process_http_object_request(struct http_object_request *freq)
-{
- if (freq->slot == NULL)
- return;
- freq->curl_result = freq->slot->curl_result;
- freq->http_code = freq->slot->http_code;
- freq->slot = NULL;
-}
-
-int finish_http_object_request(struct http_object_request *freq)
-{
- struct stat st;
-
- close(freq->localfile);
- freq->localfile = -1;
-
- process_http_object_request(freq);
-
- if (freq->http_code == 416) {
- fprintf(stderr, "Warning: requested range invalid; we may already have all the data.\n");
- } else if (freq->curl_result != CURLE_OK) {
- if (stat(freq->tmpfile, &st) == 0)
- if (st.st_size == 0)
- unlink_or_warn(freq->tmpfile);
- return -1;
- }
-
- git_inflate_end(&freq->stream);
- git_SHA1_Final(freq->real_sha1, &freq->c);
- if (freq->zret != Z_STREAM_END) {
- unlink_or_warn(freq->tmpfile);
- return -1;
- }
- if (hashcmp(freq->sha1, freq->real_sha1)) {
- unlink_or_warn(freq->tmpfile);
- return -1;
- }
- freq->rename =
- move_temp_to_file(freq->tmpfile, freq->filename);
-
- return freq->rename;
-}
-
-void abort_http_object_request(struct http_object_request *freq)
-{
- unlink_or_warn(freq->tmpfile);
-
- release_http_object_request(freq);
-}
-
-void release_http_object_request(struct http_object_request *freq)
-{
- if (freq->localfile != -1) {
- close(freq->localfile);
- freq->localfile = -1;
- }
- if (freq->url != NULL) {
- free(freq->url);
- freq->url = NULL;
- }
- freq->slot = NULL;
-}
diff --git a/http.h b/http.h
index 4c4e99c..511c0c4 100644
--- a/http.h
+++ b/http.h
@@ -88,6 +88,10 @@ extern void add_fill_function(void *data, int (*fill)(void *));
extern void step_active_slots(void);
#endif
+extern struct curl_slist *no_pragma_header;
+
+#define RANGE_HEADER_SIZE 30
+
extern void http_init(struct remote *remote);
extern void http_cleanup(void);
@@ -110,13 +114,6 @@ static inline int missing__target(int code, int result)
#define missing_target(a) missing__target((a)->http_code, (a)->curl_result)
-/* Helpers for modifying and creating URLs */
-extern void append_remote_object_url(struct strbuf *buf, const char *url,
- const char *hex,
- int only_two_digit_prefix);
-extern char *get_remote_object_url(const char *url, const char *hex,
- int only_two_digit_prefix);
-
/* Options for http_request_*() */
#define HTTP_NO_CACHE 1
@@ -170,30 +167,4 @@ extern struct http_pack_request *new_http_pack_request(
extern int finish_http_pack_request(struct http_pack_request *preq);
extern void release_http_pack_request(struct http_pack_request *preq);
-/* Helpers for fetching object */
-struct http_object_request
-{
- char *url;
- char filename[PATH_MAX];
- char tmpfile[PATH_MAX];
- int localfile;
- CURLcode curl_result;
- char errorstr[CURL_ERROR_SIZE];
- long http_code;
- unsigned char sha1[20];
- unsigned char real_sha1[20];
- git_SHA_CTX c;
- z_stream stream;
- int zret;
- int rename;
- struct active_request_slot *slot;
-};
-
-extern struct http_object_request *new_http_object_request(
- const char *base_url, unsigned char *sha1);
-extern void process_http_object_request(struct http_object_request *freq);
-extern int finish_http_object_request(struct http_object_request *freq);
-extern void abort_http_object_request(struct http_object_request *freq);
-extern void release_http_object_request(struct http_object_request *freq);
-
#endif /* HTTP_H */
^ permalink raw reply related
* Re: "make quick-install-man" broke recently
From: Jacob Helwig @ 2009-08-17 5:58 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Randal L. Schwartz, git@vger.kernel.org
In-Reply-To: <7vd46v3tp1.fsf@alter.siamese.dyndns.org>
On Sun, Aug 16, 2009 at 22:17, Junio C Hamano<gitster@pobox.com> wrote:
> merlyn@stonehenge.com (Randal L. Schwartz) writes:
>
>> % rm -rf /opt/git/share/man
>> % make prefix=/opt/git quick-install-man
>> make -C Documentation quick-install-man
>> SUBDIR ../
>> make[2]: `GIT-VERSION-FILE' is up to date.
>> '/bin/sh' ./install-doc-quick.sh origin/man /opt/git/share/man
>> % make prefix=/opt/git quick-install-man
>> make -C Documentation quick-install-man
>> SUBDIR ../
>> make[2]: `GIT-VERSION-FILE' is up to date.
>> '/bin/sh' ./install-doc-quick.sh origin/man /opt/git/share/man
>> error: git checkout-index: unable to create file /opt/git/share/man/man1/git-add.1 (File exists)
>> error: git checkout-index: unable to create file /opt/git/share/man/man1/git-am.1 (File exists)
>> error: git checkout-index: unable to create file /opt/git/share/man/man1/git-annotate.1 (File exists)
>> error: git checkout-index: unable to create file /opt/git/share/man/man1/git-apply.1 (File exists)
>> [...]
>>
>> So it fails the second time. This is new behavior.
>
> Interesting, and this does not reproduce for me.
>
> I've tried "run twice back to back" like you did above, "run once, then
> 'find -type f -print | xargs touch' the target, and then run again", and also
> "run once, then 'find -type f -print | xargs chmod -w' the target, and then run again".
>
> None of these fail.
>
> One way I can think of to break it deliberately would be:
>
> make prefix=/var/tmp/gomikuzu quick-install-man
> find /var/tmp/gomikuzu -type d | xargs chmod -w
> make prefix=/var/tmp/gomikuzu quick-install-man
>
> But then the failure is different from what you showed above:
>
> error: unable to unlink old '/var/tmp/gomikuzu/share/...' (Permission denied)
>
> Puzzled.
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
I was able to reproduce this one one of the machines I have access to,
and bisecting shows that it was introduced by b6986d8 (git-checkout: be
careful about untracked symlinks). This makes sense, since home-dirs
are symlinks on the machine I was able to reproduce it on.
Hopefully this helps someone with a little stronger fu, to point them in
the right direction.
Randal, does your machine have a similar symlinked $HOME setup?
^ permalink raw reply
* Re: "make quick-install-man" broke recently
From: Randal L. Schwartz @ 2009-08-17 6:01 UTC (permalink / raw)
To: Jacob Helwig; +Cc: Junio C Hamano, git@vger.kernel.org
In-Reply-To: <8c9a060908162258ua50e34ah53c1ada9dbcd4aa0@mail.gmail.com>
>>>>> "Jacob" == Jacob Helwig <jacob.helwig@gmail.com> writes:
Jacob> Randal, does your machine have a similar symlinked $HOME setup?
/opt/git is through two symlinks, yes.
--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion
^ permalink raw reply
* Re: git http-push and MKCOL error (22/409)
From: Thomas Schlichter @ 2009-08-17 5:34 UTC (permalink / raw)
To: Tay Ray Chuan; +Cc: Junio C Hamano, Sean Davis, git
In-Reply-To: <200908170718.15423.thomas.schlichter@web.de>
Hi,
Am Montag 17 August 2009 07:18:15 schrieb Thomas Schlichter:
> Hi,
>
> Am Montag 17 August 2009 06:52:32 schrieb Tay Ray Chuan:
> > I assume you want to commit dummy.c?
>
> Yes, indeed...
>
> > But you didn't tell git to track
> > it in the first place (git add dummy.c).
>
> I think that was not neccessary, the parameter "-a" should do it for me...
OK, I tried it again, and the parameter "-a" was not sufficient. So I
obviously did run a "git add dummy.c". Sorry for missing that.
Now I do also have the error message for the first commit:
schlicht@netbook:~/dummy2$ git push origin master
Fetching remote heads...
refs
refs/heads
refs/tags
refs/heads
refs/tags
updating 'refs/heads/master'
from 0000000000000000000000000000000000000000
to 60157fdfdca4693f703c744fed95e1b31743bb4e
sending 3 objects
MKCOL e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 failed, aborting (22/409)
MKCOL 4aefccd66b8e2bfa58079910045c30bc792877dc failed, aborting (22/409)
MKCOL 60157fdfdca4693f703c744fed95e1b31743bb4e failed, aborting (22/409)
Updating remote server info
error: failed to push some refs to
'https://webdav.smartdrive.web.de/dummy2.git'
Kind regards,
Thomas Schlichter
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox