* Re: [RFC] Enable compilation by Makefile for the MSVC toolchain
From: Daniel Barkalow @ 2009-08-18 16:51 UTC (permalink / raw)
To: Marius Storm-Olsen
Cc: Johannes.Schindelin, msysgit, git, lznuaa, bonzini, kusmabite
In-Reply-To: <1250600335-8642-1-git-send-email-mstormo@gmail.com>
On Tue, 18 Aug 2009, Marius Storm-Olsen wrote:
> From: Marius Storm-Olsen <mstormo@gmail.com>
>
> By using GNU Make we can also compile with the MSVC toolchain.
> This is a rudementary patch, only meant as an RFC for now!!
>
> !! DO NOT COMMIT THIS UPSTREAM !!
> ---
> So, instead of rely on these vcproj files which *will* go stale, we can
> simply use the same Makefile system which everyone else is using. :)
> After all, we're just compiling with a different compiler. The end result
> will still rely on the *msysGit environment* to function, so we already
> require it. Thus, GNU Make is present, and we can use it.
>
> This implementation is a quick hack to make it compile (hence the RFC
> subject), so please don't even consider basing anything ontop of it ;)
>
> But, do point out all the do's and don'ts, and I'll try to polish it up
> to something which we can add to Frank's series..
>
>
> Makefile | 97 +++++++++++++++++++++++++++++++++++++++++++++++---------
> compat/msvc.h | 77 +++++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 158 insertions(+), 16 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index daf4296..2e14976 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -214,9 +214,13 @@ uname_V := $(shell sh -c 'uname -v 2>/dev/null || echo not')
>
> CFLAGS = -g -O2 -Wall
> LDFLAGS =
> +ARFLAGS = rcs\ # whitespace intentional
> ALL_CFLAGS = $(CFLAGS)
> ALL_LDFLAGS = $(LDFLAGS)
> STRIP ?= strip
> +COMPFLAG = -c
> +COBJFLAG = -o\ # whitespace intended
> +LOBJFLAG = -o\ # whitespace intended
I think it's nicer to write the significant whitespace with non-whitespace
text using something like:
empty=
space=$(empty) $(empty)
(...)
ARFLAGS = rcs$(space)
COBJFLAG = -o$(space)
On the other hand, I think it would be clearer to put the "rcs" in the
default version of $(AR), and have a $(AROBJFLAG) set to nothing there,
since the "rcs" isn't actually at all like the "-OUT:" with respect to
what it's doing there.
Possibly also to have two variables for the output of the toolchain
wrapper, one that is before the name of the file and one that's attached
to the name of the file.
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* Re: [msysGit] Re: [PATCH 02/11] Fix declare variable at mid of function
From: Matthieu Moy @ 2009-08-18 16:52 UTC (permalink / raw)
To: Frank Li; +Cc: git
In-Reply-To: <1976ea660908180911m7469ac20w48a28b90262d25f6@mail.gmail.com>
Frank Li <lznuaa@gmail.com> writes:
>> Okay, I will wait for Frank's updates (just fetched tgit.git and it still
>> contains the old branch), merge the early part and add the compiler flags.
>>
> Today, I just update 5 patch according review feedback.
> Do I need send it again?
Yes, this is the use here. Preferably edit the message to replace
[PATCH] with [PATCH v2] or so.
> How do I know if patch has been applied main line?
Answering the question with another: shouldn't we add a section like
this to SubmittingPatches ?
>From 3ee45ab5992fd084c130460f07454061ce3cf057 Mon Sep 17 00:00:00 2001
From: Matthieu Moy <Matthieu.Moy@imag.fr>
Date: Tue, 18 Aug 2009 18:48:47 +0200
Subject: [PATCH] SubmittingPatches: draft section to know patches status
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
Documentation/SubmittingPatches | 14 ++++++++++++++
1 files changed, 14 insertions(+), 0 deletions(-)
diff --git a/Documentation/SubmittingPatches b/Documentation/SubmittingPatches
index 76fc84d..c686f86 100644
--- a/Documentation/SubmittingPatches
+++ b/Documentation/SubmittingPatches
@@ -280,6 +280,20 @@ people play with it without having to pick up and apply the patch to
their trees themselves.
------------------------------------------------
+Know the status of your patch after submission
+
+* You can use Git itself to find out when your patch is merged in
+ master. 'git pull --rebase' will automatically skip already-applied
+ patches, and will let you know. This works only if you rebase on top
+ of the branch in which your patch has been merged (i.e. it will not
+ tell you if your patch is merged in pu if you rebase on top of
+ master).
+
+* Read the git mailing list, the maintainer regularly posts messages
+ entitled "What's cooking in git.git" and "What's in git.git" giving
+ the status of various proposed changes.
+
+------------------------------------------------
MUA specific hints
Some of patches I receive or pick up from the list share common
--
1.6.4.313.g38b9
--
Matthieu
^ permalink raw reply related
* Re: [PATCH] block-sha1: Windows declares ntohl() in winsock2.h
From: Sebastian Schuberth @ 2009-08-18 16:59 UTC (permalink / raw)
To: Nicolas Pitre
Cc: Junio C Hamano, Artur Skawina, Johannes Sixt, msysGit,
Linus Torvalds, Git Mailing List
In-Reply-To: <alpine.LFD.2.00.0908181240400.6044@xanadu.home>
On Tue, Aug 18, 2009 at 18:43, Nicolas Pitre<nico@cam.org> wrote:
>> Well... Given that git already uses ntohl/htonl quite extensively in
>> its core already, I'd suggest making this more globally available
>> instead.
>
> What about something like this?
I like the idea of making bswap available more globally, but I'm not
sure if it's worth to introduce a new file for only that purpose.
Isn't there already a central header for such things?
Moreover, including compat/bswap.h would only give you ntohl()/htonl()
on one platform. For consistency, I'd expect to get those for any
platform if I include compat/bswap.h, but maybe I'm not aware of some
Git source code rules.
Finally, there's a typo in your comment saying "sinple" instead of "simple".
--
Sebastian Schuberth
^ permalink raw reply
* Re: Git User's Survey 2009 partial summary, part 2 - from first 10
From: Jakub Narebski @ 2009-08-18 17:00 UTC (permalink / raw)
To: Nicolas Sebrecht; +Cc: git
In-Reply-To: <20090818155452.GA12938@vidovic>
On Tue, 18 Aug 2009, Nicolas Sebrecht wrote:
> The 17/08/09, Jakub Narebski wrote:
>
> > Git User's Survey 2009 partial summary, part 2 - git difficulty,
> > proficiency, uses, install, OS, editors.
>
> Thanks for these partial summaries.
You are welcome. Next summary either around 1 September (3/4 duration),
or after passing number of responses in previous survey; it also depends
when I would have time.
> > 3) Have you found Git easy to learn?
> > 4) Have you found Git easy to use?
> > (Choice - Single answer)
> >
> > ================================================
> > Answer | to learn [%] | to use [%]
> > ------------------------------------------------
> > Very easy | 4% | 9%
> > Easy | 20% | 36%
> > Reasonably easy | 55% | 45%
> > Hard | 19% | 8%
> > Very hard | 2% | 1%
> > ------------------------------------------------
> > Total respondents | 2942 | 2959
> > ================================================
>
> <...>
>
> > What's interesting is comparing (percentage) results for questions
> > 3. and 4.; how hard is git to learn versus how hard is to use. It
> > seems like Git is reasonably easy to learn, and reasonably easy to
> > easy to use. So it looks like Git just have somewhat steep learning
> > curve, and the difficulty to learn pays in being more powerful to
> > use.
>
> I believe it would be interesting to know who (from the question 6.)
> think what later. We may expect that people of the grade 4 and 5 ("can
> offer advice" and "know it very well") underestimate the difficulty to
> learn Git.
Well, you can do some analysis yourself, using just created filters at
http://www.survs.com/shareResults?survey=2PIMZGU0&rndm=678J66QRA2
For example:
================================================|
| proficiency with Git |
To learn |-----------------------------|
| any*[%] | 1-3 [%] | 4-5 [%] |
------------------------------------------------|
Very easy | 4% | 3% | 5% |
Easy | 20% | 19% | 22% |
Reasonably easy | 55% | 55% | 56% |
Hard | 19% | 20% | 16% |
Very hard | 2% | 3% | 1% |
------------------------------------------------|
Total respondents | 3005 | 1786 | 1212 |
================================================|
Footnote:
^^^^^^^^^
[*] This means no filter, which includes people who didn't answered
question about proficiency, but answered this one.
> Also (and as you said), this "Git's users" survey won't have answers
> from unsatisfied users who left Git. We can't rate the number of users
> who left Git because they found it too much hard to learn.
Here also you have to be careful about the fact that "correlation does
not imply causation" (or, if you prefer Latin version, "cum hoc ergo
propter hoc"). The causation might be both ways: gurus don't find
git hard to learn, or don't remember that it was hard to learn; people
who find git hard to learn do not become git gurus.
--
Jakub Narebski
http://git.or.cz/gitwiki/GitSurvey2009
^ permalink raw reply
* Re: [PATCH] block-sha1: Windows declares ntohl() in winsock2.h
From: Junio C Hamano @ 2009-08-18 17:05 UTC (permalink / raw)
To: Sebastian Schuberth
Cc: Nicolas Pitre, Junio C Hamano, Artur Skawina, Johannes Sixt,
msysGit, Linus Torvalds, Git Mailing List
In-Reply-To: <bdca99240908180959h69f37671k4d526fbf4814e8d1@mail.gmail.com>
Sebastian Schuberth <sschuberth@gmail.com> writes:
> I like the idea of making bswap available more globally, but I'm not
> sure if it's worth to introduce a new file for only that purpose.
> Isn't there already a central header for such things?
>
> Moreover, including compat/bswap.h would only give you ntohl()/htonl()
> on one platform.
We do not include compat/ directly from the source; git-compat-util.h is
supposed to be the first thing included (as some platforms have peculiar
requirements on the order in which system header files are included, and
one of the reasons git-compat-util.h is there). Hence by including it,
you get ntohl/htonl everywhere.
To reduce confusion, you may want to rename compat/bswap.h to something
like compat/ntohl-htonl-fix.h ;-)
^ permalink raw reply
* Re: [RFC] Enable compilation by Makefile for the MSVC toolchain
From: Pau Garcia i Quiles @ 2009-08-18 17:06 UTC (permalink / raw)
To: Johan 't Hart
Cc: Marius Storm-Olsen, Johannes.Schindelin, msysgit, git, lznuaa,
bonzini, kusmabite
In-Reply-To: <4A8AD561.1020303@gmail.com>
On Tue, Aug 18, 2009 at 6:22 PM, Johan 't Hart<johanthart@gmail.com> wrote:
> Marius Storm-Olsen schreef:
>>
>> From: Marius Storm-Olsen <mstormo@gmail.com>
>>
>> By using GNU Make we can also compile with the MSVC toolchain.
>> This is a rudementary patch, only meant as an RFC for now!!
>>
>
> Would this mean that only the MSVC toolchain is used to build git in batch?
> Or does GNU Make create a .vcproj file like CMake?
That patch uses GNU Make but invoking the MSVC compiler, linker, etc
instead of the GNU compiler, linker, etc. It'd be the same if you
wanted to use the Intel compiler, the Sun compiler, etc.
> Because that ofcource is
> the whole purpose of using CMake. One can use the Visual Studio IDE to hack
> on git.
CMake is also useful to create installers/tarballs (see CPack), run
tests (see CTest; it's NOT a unit test library but a tests launcher on
steroids) and submit the results of compilation, tests and
installation to a dashboard where you can easily see what (if
anything) went wrong (see CDash, formerly Dart).
--
Pau Garcia i Quiles
http://www.elpauer.org
(Due to my workload, I may need 10 days to answer)
^ permalink raw reply
* Re: [RFC] Enable compilation by Makefile for the MSVC toolchain
From: Pau Garcia i Quiles @ 2009-08-18 17:07 UTC (permalink / raw)
To: Johannes Schindelin
Cc: Marius Storm-Olsen, msysgit, git, lznuaa, bonzini, kusmabite
In-Reply-To: <alpine.DEB.1.00.0908181605370.4680@intel-tinevez-2-302>
On Tue, Aug 18, 2009 at 4:11 PM, Johannes
Schindelin<Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
> On Tue, 18 Aug 2009, Marius Storm-Olsen wrote:
>
>> So, instead of rely on these vcproj files which *will* go stale, we can
>> simply use the same Makefile system which everyone else is using. :)
>> After all, we're just compiling with a different compiler. The end
>> result will still rely on the *msysGit environment* to function, so we
>> already require it. Thus, GNU Make is present, and we can use it.
>
> We can also use sed or perl to generate/modify the .vcproj files, or run
> CMake (once Pau got it to build), and package the stuff using zip (once I
> got that to build).
Re: package the stuff using ZIP, it's trivial to do with CPack (part
of the CMake chain). Just tell me what you want to get and I'll
implement it.
--
Pau Garcia i Quiles
http://www.elpauer.org
(Due to my workload, I may need 10 days to answer)
^ permalink raw reply
* Re: [RFC] Enable compilation by Makefile for the MSVC toolchain
From: Marius Storm-Olsen @ 2009-08-18 17:36 UTC (permalink / raw)
To: Junio C Hamano
Cc: Johannes.Schindelin, msysgit, git, lznuaa, bonzini, kusmabite
In-Reply-To: <7vtz05dq0p.fsf@alter.siamese.dyndns.org>
Junio C Hamano said the following on 18.08.2009 18:51:
> Marius Storm-Olsen <mstormo@gmail.com> writes:
>
> Marius Storm-Olsen <mstormo@gmail.com> writes:
>
>> +ARFLAGS = rcs\ # whitespace intentional
>> +COMPFLAG = -c
>> +COBJFLAG = -o\ # whitespace intended
>> +LOBJFLAG = -o\ # whitespace intended
>> ...
>> +ifneq (,$(findstring Microsoft Visual Studio, $(INCLUDE)))
>> + CC = cl
>> + COBJFLAG = -Fo
>> + LOBJFLAG = -OUT:
>> + CFLAGS =
>> git.o: git.c common-cmds.h GIT-CFLAGS
>> ...
>> - $(ALL_CFLAGS) -c $(filter %.c,$^)
>> + $(ALL_CFLAGS) $(COMPFLAG) $(COBJFLAG)git.o $(filter %.c,$^)
>
> Since use of make implies use of shell, this makes me wonder if it would
> make sense to go one step further by giving msvc users a thin shell
> wrapper mcvc-cc that turns bog-standard cc command line into whatever cl
> uses.
That, of course, is also a feasible option :)
However, there are still things you wouldn't want to pass on to this
script, for example, we don't pass on all the CFLAGS to the linker on
Windows.
The "whitespace intended" in the patch are quite nasty though..
--
.marius
^ permalink raw reply
* Re: [RFC] Enable compilation by Makefile for the MSVC toolchain
From: Marius Storm-Olsen @ 2009-08-18 17:38 UTC (permalink / raw)
To: Daniel Barkalow
Cc: Johannes.Schindelin, msysgit, git, lznuaa, bonzini, kusmabite
In-Reply-To: <alpine.LNX.2.00.0908181226160.7195@iabervon.org>
Daniel Barkalow said the following on 18.08.2009 18:51:
> On Tue, 18 Aug 2009, Marius Storm-Olsen wrote:
>
>> From: Marius Storm-Olsen <mstormo@gmail.com>
>>
>> By using GNU Make we can also compile with the MSVC toolchain.
>> This is a rudementary patch, only meant as an RFC for now!!
>>
>> !! DO NOT COMMIT THIS UPSTREAM !!
>> ---
>> So, instead of rely on these vcproj files which *will* go stale, we can
>> simply use the same Makefile system which everyone else is using. :)
>> After all, we're just compiling with a different compiler. The end result
>> will still rely on the *msysGit environment* to function, so we already
>> require it. Thus, GNU Make is present, and we can use it.
>>
>> This implementation is a quick hack to make it compile (hence the RFC
>> subject), so please don't even consider basing anything ontop of it ;)
>>
>> But, do point out all the do's and don'ts, and I'll try to polish it up
>> to something which we can add to Frank's series..
>>
>>
>> Makefile | 97 +++++++++++++++++++++++++++++++++++++++++++++++---------
>> compat/msvc.h | 77 +++++++++++++++++++++++++++++++++++++++++++++
>> 2 files changed, 158 insertions(+), 16 deletions(-)
>>
>> diff --git a/Makefile b/Makefile
>> index daf4296..2e14976 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -214,9 +214,13 @@ uname_V := $(shell sh -c 'uname -v 2>/dev/null || echo not')
>>
>> CFLAGS = -g -O2 -Wall
>> LDFLAGS =
>> +ARFLAGS = rcs\ # whitespace intentional
>> ALL_CFLAGS = $(CFLAGS)
>> ALL_LDFLAGS = $(LDFLAGS)
>> STRIP ?= strip
>> +COMPFLAG = -c
>> +COBJFLAG = -o\ # whitespace intended
>> +LOBJFLAG = -o\ # whitespace intended
>
> I think it's nicer to write the significant whitespace with non-whitespace
> text using something like:
>
> empty=
> space=$(empty) $(empty)
>
> (...)
>
> ARFLAGS = rcs$(space)
>
> COBJFLAG = -o$(space)
>
> On the other hand, I think it would be clearer to put the "rcs" in the
> default version of $(AR), and have a $(AROBJFLAG) set to nothing there,
> since the "rcs" isn't actually at all like the "-OUT:" with respect to
> what it's doing there.
>
> Possibly also to have two variables for the output of the toolchain
> wrapper, one that is before the name of the file and one that's attached
> to the name of the file.
I agree on all accounts. If we don't go for a wrapper script, as Junio
suggested, then I'll definitely do it like you suggested.
--
.marius
^ permalink raw reply
* Re: [RFC] Enable compilation by Makefile for the MSVC toolchain
From: Johan 't Hart @ 2009-08-18 17:41 UTC (permalink / raw)
To: Junio C Hamano
Cc: Marius Storm-Olsen, Johannes.Schindelin, msysgit, git, lznuaa,
bonzini, kusmabite
In-Reply-To: <7vtz05dq0p.fsf@alter.siamese.dyndns.org>
Junio C Hamano schreef:
> Since use of make implies use of shell, this makes me wonder if it would
> make sense to go one step further by giving msvc users a thin shell
> wrapper mcvc-cc that turns bog-standard cc command line into whatever cl
> uses.
Just using the msvc toolchain for building git misses the whole purpose
of what VC is used for. MSVC would be used because of the IDE, not for
the compiler IMO. msysgit already has a nice toolchain, using gcc, to
build git. I think there is no reason to use MSVC for that purpose.
The purpose of CMake was to build an .vcproj file to be used by the IDE,
not for a batch process that builds git with the VC toolchain. Users
could than hack git with the IDE from MSVC. And ofcource git should then
also be compilable by MSVC's cl.exe, because the IDE is not capable of
using another toolchain then its own...
^ permalink raw reply
* Re: [RFC PATCH] stash: accept options also when subcommand 'save' is omitted
From: Jeff King @ 2009-08-18 17:45 UTC (permalink / raw)
To: Matthieu Moy; +Cc: git, Johannes Schindelin
In-Reply-To: <vpqws51l1hb.fsf@bauges.imag.fr>
On Tue, Aug 18, 2009 at 03:01:52PM +0200, Matthieu Moy wrote:
> Hmm, googling a bit, I just noticed that there's already something in
> pu:
> ea41cfc4f (Make 'git stash -k' a short form for 'git stash save --keep-index')
> which also does the trick, while adding a -k alias for --keep-index.
>
> [...]
>
> Mine has at least two advantages:
>
> * It won't require changing the code again when new options are added
> to 'git stash save'.
>
> * It works with 'git stash -k -q' for example, while the other
> proposal checks that $# == 1, which won't work if there are more
> than one option.
I think yours is nicer, especially as we have just added the
'-p|--patch' option, as well. With what is there now, you can do "git
stash -p", but not "git stash -p -k".
> But I may have missed its drawbacks ;-)
The only I can think of is that bogus input will provoke 'save'. So
something like:
git stash --apply
will invoke "git stash save --apply", which doesn't even complain. It
just tries to make a stash with message --apply. Now of course this
input is obviously bogus, but probably the right thing to do is
complain.
OTOH, I think it is the fault of "git stash save --apply" for not doing
the complaining, so your patch really isn't making it worse. Probably it
should barf on anything unrecognized starting with a '-', and allow '--'
to separate the message from the rest of the options (in the rare case
that you want a message starting with '-').
-Peff
^ permalink raw reply
* Re: [RFC] Enable compilation by Makefile for the MSVC toolchain
From: Paolo Bonzini @ 2009-08-18 17:44 UTC (permalink / raw)
To: Pau Garcia i Quiles
Cc: Johan 't Hart, Marius Storm-Olsen, Johannes.Schindelin,
msysgit, git, lznuaa, kusmabite
In-Reply-To: <3af572ac0908181006n2e00b561x374007bff2e2d31f@mail.gmail.com>
> CMake is also useful to create installers/tarballs (see CPack), run
> tests (see CTest; it's NOT a unit test library but a tests launcher on
> steroids) and submit the results of compilation, tests and
> installation to a dashboard where you can easily see what (if
> anything) went wrong (see CDash, formerly Dart).
If I know the git hackers, CMake will _never_ be used except _possibly_
for generating a .vcproj file.
Paolo
^ permalink raw reply
* Re: [RFC] Enable compilation by Makefile for the MSVC toolchain
From: Marius Storm-Olsen @ 2009-08-18 17:55 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: msysgit, git, lznuaa, bonzini, kusmabite
In-Reply-To: <alpine.DEB.1.00.0908181605370.4680@intel-tinevez-2-302>
Johannes Schindelin said the following on 18.08.2009 16:11:
> On Tue, 18 Aug 2009, Marius Storm-Olsen wrote:
>> So, instead of rely on these vcproj files which *will* go stale, we
>> can simply use the same Makefile system which everyone else is
>> using. :) After all, we're just compiling with a different
>> compiler. The end result will still rely on the *msysGit
>> environment* to function, so we already require it. Thus, GNU Make
>> is present, and we can use it.
>
> We can also use sed or perl to generate/modify the .vcproj files, or
> run CMake (once Pau got it to build), and package the stuff using zip
> (once I got that to build).
Really? That would be some script being able to parse the Makefile, and
create something reasonable as a vcproj script :) Keeping all the
options in sync, conditional files/libs, all the various end executables
(a separate .vcproj for each of them, and a solution file (.sln)to tie
them all together into a .. "solution", a complete product blah blah
blah) etc.
Just using the (GNU) Makefile as the 'default' way of compiling with
MSVC seems to be the most reasonable to me. Then we could just have a
contrib/vc/msvc2008.vcproj which someone can update when they feel like
it. The default way should be hard to get out-of-sync, IMO.
>> diff --git a/Makefile b/Makefile
>> +ifneq (,$(findstring Microsoft Visual Studio, $(INCLUDE)))
...
>> +else
>> ifneq (,$(findstring MINGW,$(uname_S)))
>> pathsep = ;
>> NO_PREAD = YesPlease
>
> This means that gcc is never used when Visual C++ is available? Hmm.
Yes it does :)
First, "This implementation is a quick hack to make it compile", so it
was all about just making it compile, not being 100% correct. There are
still some warnings and options which shouldn't be passed on to cl, so
it's by no means complete. Second, having MSVC include paths in INCLUDE
when you're compiling with MinGW would be plain wrong. I could use LIB
instead, which would be even wronger ;)
I was actually going to replace it with a USE_MSVC variable check
instead though, so you'd rather do 'make USE_MSVC=1'
>> diff --git a/compat/msvc.h b/compat/msvc.h
>> index 6071565..a9d5f7f 100644
>> --- a/compat/msvc.h
>> +++ b/compat/msvc.h
>> @@ -10,50 +10,120 @@
>>
>> /*Configuration*/
>>
>> +#ifndef NO_PREAD
>> #define NO_PREAD
>> +#endif
>
> Why? You now have the stuff in two places. If you want to keep them in
> compat/msvc.h to be able to generate .vcproj files, I'd rather not have
> them duplicated in the Makefile.
The changes to the msvc.h where added just keep Frank's vcproj project
working. I would have just nuked all that stuff in the msvc.h since the
Makefile would just provide it for me anyways..
--
.marius
^ permalink raw reply
* Re: Continue git clone after interruption
From: Nicolas Pitre @ 2009-08-18 17:56 UTC (permalink / raw)
To: Tomasz Kontusz; +Cc: git
In-Reply-To: <1250578735.2885.40.camel@cf-48>
On Tue, 18 Aug 2009, Tomasz Kontusz wrote:
> Ok, so it looks like it's not implementable without some kind of cache
> server-side, so the server would know what the pack it was sending
> looked like.
> But here's my idea: make server send objects in different order (the
> newest commit + whatever it points to first, then next one,then
> another...). Then it would be possible to look at what we got, tell
> server we have nothing, and want [the newest commit that was not
> complete]. I know the reason why it is sorted the way it is, but I think
> that the way data is stored after clone is clients problem, so the
> client should reorganize packs the way it wants.
That won't buy you much. You should realize that a pack is made of:
1) Commit objects. Yes they're all put together at the front of the pack,
but they roughly are the equivalent of:
git log --pretty=raw | gzip | wc -c
For the Linux repo as of now that is around 32 MB.
2) Tree andblob objects. Those are the bulk of the content for the top
commit. The top commit is usually not delta compressed because we
want fast access to the top commit, and that is used as the base for
further delta compression for older commits. So the very first
commit is whole at the front of the pack right after the commit
objects. you can estimate the size of this data with:
git archive --format=tar HEAD | gzip | wc -c
On the same Linux repo this is currently 75 MB.
3) Delta objects. Those are making the rest of the pack, plus a couple
tree/blob objects that were not found in the top commit and are
different enough from any object in that top commit not to be
represented as deltas. Still, the majority of objects for all the
remaining commits are delta objects.
So... if we reorder objects, all that we can do is to spread commit
objects around so that the objects referenced by one commit are all seen
before another commit object is included. That would cut on that
initial 32 MB.
However you still have to get that 75 MB in order to at least be able to
look at _one_ commit. So you've only reduced your critical download
size from 107 MB to 75 MB. This is some improvement, of course, but not
worth the bother IMHO. If we're to have restartable clone, it has to
work for any size.
And that's where the real problem is. I don't think having servers to
cache pack results for every fetch requests is sensible as that would be
an immediate DoS attack vector.
And because the object order in a pack is not defined by the protocol,
we cannot expect the server to necessarily always provide the same
object order either. For example, it is already undefined in which
order you'll receive objects as threaded delta search is non
deterministic and two identical fetch requests may end up with slightly
different packing. Or load balancing may redirect your fetch requests
to different git servers which might have different versions of zlib, or
even git itself, affecting the object packing order and/or size.
Now... What _could_ be done, though, is some extension to the
git-archive command. One thing that is well and strictly defined in git
is the file path sort order. So given a commit SHA1, you should always
get the same files in the same order from git-archive. For an initial
clone, git could attempt fetching the top commit using the remote
git-archive service and locally reconstruct that top commit that way.
if the transfer is interrupted in the middle, then the remote
git-archive could be told how to resume the transfer by telling it how
many files and how many bytes in the current file to skip. This way the
server doesn't need to perform any sort of caching and remains
stateless.
You then end up with a pretty shallow repository. The clone process
could then fall back to the traditional native git transfer protocol to
deepen the history of that shallow repository. And then that special
packing sort order to distribute commit objects would make sense since
each commit would then have a fairly small set of new objects, and most
of them would be deltas anyway, making the data size per commit really
small and any interrupted transfer much less of an issue.
Nicolas
^ permalink raw reply
* Re: [PATCH] block-sha1: Windows declares ntohl() in winsock2.h
From: Nicolas Pitre @ 2009-08-18 18:01 UTC (permalink / raw)
To: Junio C Hamano
Cc: Sebastian Schuberth, Artur Skawina, Johannes Sixt, msysGit,
Linus Torvalds, Git Mailing List
In-Reply-To: <7v1vn9f4mz.fsf@alter.siamese.dyndns.org>
On Tue, 18 Aug 2009, Junio C Hamano wrote:
> Nicolas Pitre <nico@cam.org> writes:
>
> > On Tue, 18 Aug 2009, Nicolas Pitre wrote:
> >
> >> Well... Given that git already uses ntohl/htonl quite extensively in
> >> its core already, I'd suggest making this more globally available
> >> instead.
> >
> > What about something like this?
>
> For git's own use, I would be much happier with this change.
>
> But given that there are some people wanting to snarf block-sha1/*.[ch]
> out to use them standalone, I have a slight hesitation against introducing
> the dependency to git-compat-util.h, making it unclear to them that all
> this file wants from outside are ntohl, htonl and memcpy.
Should we really care to keep our code suboptimal just to make it
readily reusable by other projects? That seems a bit backward to me.
We might simply add a comment telling people what git-compat-util.h is
actually for, namely memcpy(), ntohl() and htonl(). That should be
trivial for them to use the appropriate substitute.
Nicolas
^ permalink raw reply
* Re: [RFC] Enable compilation by Makefile for the MSVC toolchain
From: Marius Storm-Olsen @ 2009-08-18 18:06 UTC (permalink / raw)
To: Johan 't Hart
Cc: Junio C Hamano, Johannes.Schindelin, msysgit, git, lznuaa,
bonzini, kusmabite
In-Reply-To: <4A8AE7C5.7050600@gmail.com>
Johan 't Hart said the following on 18.08.2009 19:41:
> Junio C Hamano schreef:
>> Since use of make implies use of shell, this makes me wonder if it
>> would make sense to go one step further by giving msvc users a thin
>> shell wrapper mcvc-cc that turns bog-standard cc command line into
>> whatever cl uses.
>
> Just using the msvc toolchain for building git misses the whole
> purpose of what VC is used for. MSVC would be used because of the
> IDE, not for the compiler IMO.
Bull. MSVC produces superior code on Windows compared to MinGW, I'm
afraid. Add the /LTGC (Link Time Code Generation), and MSVC generates
very good cross compile-unit optimized code. (I know gcc has the option,
but it's not as good, by far)
Coupled now with built-in static code analysis, these are only two
reasons why *I* would want to build it directly from the command line
without worrying that my .vcproj is out-of-sync with the main development.
You can still debug with the MSVC debugger if you'd like, and the MSVC
IDE allows you to wrap Makefile project too, so you can *still* use the
IDE..
--
.marius
^ permalink raw reply
* Re: [PATCH] block-sha1: Windows declares ntohl() in winsock2.h
From: Nicolas Pitre @ 2009-08-18 18:10 UTC (permalink / raw)
To: Sebastian Schuberth
Cc: Junio C Hamano, Artur Skawina, Johannes Sixt, msysGit,
Linus Torvalds, Git Mailing List
In-Reply-To: <bdca99240908180959h69f37671k4d526fbf4814e8d1@mail.gmail.com>
[-- Attachment #1: Type: TEXT/PLAIN, Size: 1542 bytes --]
On Tue, 18 Aug 2009, Sebastian Schuberth wrote:
> On Tue, Aug 18, 2009 at 18:43, Nicolas Pitre<nico@cam.org> wrote:
>
> >> Well... Given that git already uses ntohl/htonl quite extensively in
> >> its core already, I'd suggest making this more globally available
> >> instead.
> >
> > What about something like this?
>
> I like the idea of making bswap available more globally, but I'm not
> sure if it's worth to introduce a new file for only that purpose.
> Isn't there already a central header for such things?
That central header is already quite crowded. A bit of isolation might
not hurt.
Furthermore, other platforms might wish to add their own (re)definitions
for those byte swap operations, so it has the potential to grow. I for
example have a better implementation for ARM than what is provided by
glibc. (Yeah yeah, maybe glibc should be fixed instead, but that
reasoning goes for all those other libraries too).
> Moreover, including compat/bswap.h would only give you ntohl()/htonl()
> on one platform. For consistency, I'd expect to get those for any
> platform if I include compat/bswap.h, but maybe I'm not aware of some
> Git source code rules.
You get it by default for all platforms already by including
git-compat-util.h. The compat/bswap.h is not meant to be included by
random c files. If compat/bswap.h happens to contain a better version
for your architecture then it'll override the default one.
> Finally, there's a typo in your comment saying "sinple" instead of "simple".
Thanks
Nicolas
^ permalink raw reply
* Re: [PATCH] block-sha1: Windows declares ntohl() in winsock2.h
From: Nicolas Pitre @ 2009-08-18 18:16 UTC (permalink / raw)
To: Junio C Hamano
Cc: Sebastian Schuberth, Artur Skawina, Johannes Sixt, msysGit,
Linus Torvalds, Git Mailing List
In-Reply-To: <7vpratdpc8.fsf@alter.siamese.dyndns.org>
On Tue, 18 Aug 2009, Junio C Hamano wrote:
> To reduce confusion, you may want to rename compat/bswap.h to something
> like compat/ntohl-htonl-fix.h ;-)
Bah. If you wish, you can edit the patch directly for this, unless you
really prefer me to repost. Maybe we might want to add a 8-byte
versions of those as well eventually, which is why I chose a more
generic name.
Nicolas
^ permalink raw reply
* Re: [RFC] Enable compilation by Makefile for the MSVC toolchain
From: Daniel Barkalow @ 2009-08-18 18:42 UTC (permalink / raw)
To: Marius Storm-Olsen
Cc: Johannes Schindelin, msysgit, git, lznuaa, bonzini, kusmabite
In-Reply-To: <4A8AEAF5.6070205@gmail.com>
On Tue, 18 Aug 2009, Marius Storm-Olsen wrote:
> Johannes Schindelin said the following on 18.08.2009 16:11:
> > On Tue, 18 Aug 2009, Marius Storm-Olsen wrote:
> > > So, instead of rely on these vcproj files which *will* go stale, we
> > > can simply use the same Makefile system which everyone else is
> > > using. :) After all, we're just compiling with a different
> > > compiler. The end result will still rely on the *msysGit
> > > environment* to function, so we already require it. Thus, GNU Make
> > > is present, and we can use it.
> >
> > We can also use sed or perl to generate/modify the .vcproj files, or
> > run CMake (once Pau got it to build), and package the stuff using zip
> > (once I got that to build).
>
> Really? That would be some script being able to parse the Makefile, and create
> something reasonable as a vcproj script :) Keeping all the options in sync,
> conditional files/libs, all the various end executables (a separate .vcproj
> for each of them, and a solution file (.sln)to tie them all together into a ..
> "solution", a complete product blah blah blah) etc.
I think it wouldn't be impossible to split the Makefile into an
easy-to-parse part and an irrelevant-to-vcproj part. Certainly you don't
need GNU Make to read a file:
SCRIPT_SH += git-am.sh
SCRIPT_SH += git-bisect.sh
SCRIPT_PERL += git-add--interactive.perl
PROGRAM_NAMES += git-fast-import
BUILTINS_IN_OBJS += add
(etc)
That is, we can probably describe the project sufficiently with a lot of
VAR += value
lines, using only constant values and variables we specify, and then the
Makefile declares them empty and does an
include UserServicableParts
and puts together the variables it needs. And things that aren't GNU Make
could also process this file without enormous difficulty, since it's
essentially a .ini file or java resource file with a stray + on each line.
Of course, people interested in the vcproj thing would have to update
whatever makes VC do the right thing when new *rules* are introduced, but
that's a lot less common than new *files* being introduced, and also more
obvious (in the sense that the included file is setting variables that the
builder doesn't know what to do with).
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* Re: Continue git clone after interruption
From: Jakub Narebski @ 2009-08-18 18:45 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Tomasz Kontusz, git
In-Reply-To: <alpine.LFD.2.00.0908181246470.6044@xanadu.home>
Nicolas Pitre <nico@cam.org> writes:
> On Tue, 18 Aug 2009, Tomasz Kontusz wrote:
>
> > Ok, so it looks like it's not implementable without some kind of cache
> > server-side, so the server would know what the pack it was sending
> > looked like.
> > But here's my idea: make server send objects in different order (the
> > newest commit + whatever it points to first, then next one,then
> > another...). Then it would be possible to look at what we got, tell
> > server we have nothing, and want [the newest commit that was not
> > complete]. I know the reason why it is sorted the way it is, but I think
> > that the way data is stored after clone is clients problem, so the
> > client should reorganize packs the way it wants.
>
> That won't buy you much. You should realize that a pack is made of:
>
> 1) Commit objects. Yes they're all put together at the front of the pack,
> but they roughly are the equivalent of:
>
> git log --pretty=raw | gzip | wc -c
>
> For the Linux repo as of now that is around 32 MB.
For my clone of Git repository this gives 3.8 MB
> 2) Tree and blob objects. Those are the bulk of the content for the top
> commit. The top commit is usually not delta compressed because we
> want fast access to the top commit, and that is used as the base for
> further delta compression for older commits. So the very first
> commit is whole at the front of the pack right after the commit
> objects. you can estimate the size of this data with:
>
> git archive --format=tar HEAD | gzip | wc -c
>
> On the same Linux repo this is currently 75 MB.
On the same Git repository this gives 2.5 MB
>
> 3) Delta objects. Those are making the rest of the pack, plus a couple
> tree/blob objects that were not found in the top commit and are
> different enough from any object in that top commit not to be
> represented as deltas. Still, the majority of objects for all the
> remaining commits are delta objects.
You forgot that delta chains are bound by pack.depth limit, which
defaults to 50. You would have then additional full objects.
The single packfile for this (just gc'ed) Git repository is 37 MB.
Much more than 3.8 MB + 2.5 MB = 6.3 MB.
[cut]
There is another way which we can go to implement resumable clone.
Let's git first try to clone whole repository (single pack; BTW what
happens if this pack is larger than file size limit for given
filesystem?). If it fails, client ask first for first half of of
repository (half as in bisect, but it is server that has to calculate
it). If it downloads, it will ask server for the rest of repository.
If it fails, it would reduce size in half again, and ask about 1/4 of
repository in packfile first.
The only extension required is for server to support additional
capability, which enable for client to ask for appropriate 1/2^n part
of repository (approximately), or 1/2^n between have and want.
--
Jakub Narebski
Poland
ShadeHawk on #git
^ permalink raw reply
* Re: [PATCH] block-sha1: Windows declares ntohl() in winsock2.h
From: Junio C Hamano @ 2009-08-18 19:00 UTC (permalink / raw)
To: Nicolas Pitre
Cc: Sebastian Schuberth, Artur Skawina, Johannes Sixt, msysGit,
Linus Torvalds, Git Mailing List
In-Reply-To: <alpine.LFD.2.00.0908181357330.6044@xanadu.home>
Nicolas Pitre <nico@cam.org> writes:
> On Tue, 18 Aug 2009, Junio C Hamano wrote:
>
>> For git's own use, I would be much happier with this change.
>>
>> But given that there are some people wanting to snarf block-sha1/*.[ch]
>> out to use them standalone, I have a slight hesitation against introducing
>> the dependency to git-compat-util.h, making it unclear to them that all
>> this file wants from outside are ntohl, htonl and memcpy.
>
> Should we really care to keep our code suboptimal just to make it
> readily reusable by other projects? That seems a bit backward to me.
You are right; and I should give a bit more credit to their intelligence.
The source (block-sha1/sha1.c) is short enough that they can figure this
out for themselves even without any additional comments.
Another issue, especially with your "openssl sha1 removal" patch, is if we
can assume gcc everywhere. As far as I can tell, block-sha1/sha1.c will
be the first unconditional use of inline asm or statement expression on
i386/amd64. Are folks on Solaris and other platforms Ok with this?
^ permalink raw reply
* Re: [PATCH] block-sha1: Windows declares ntohl() in winsock2.h
From: Nicolas Pitre @ 2009-08-18 19:22 UTC (permalink / raw)
To: Junio C Hamano
Cc: Sebastian Schuberth, Artur Skawina, Johannes Sixt, msysGit,
Linus Torvalds, Git Mailing List
In-Reply-To: <7vk511dk11.fsf@alter.siamese.dyndns.org>
On Tue, 18 Aug 2009, Junio C Hamano wrote:
> Nicolas Pitre <nico@cam.org> writes:
>
> > On Tue, 18 Aug 2009, Junio C Hamano wrote:
> >
> >> For git's own use, I would be much happier with this change.
> >>
> >> But given that there are some people wanting to snarf block-sha1/*.[ch]
> >> out to use them standalone, I have a slight hesitation against introducing
> >> the dependency to git-compat-util.h, making it unclear to them that all
> >> this file wants from outside are ntohl, htonl and memcpy.
> >
> > Should we really care to keep our code suboptimal just to make it
> > readily reusable by other projects? That seems a bit backward to me.
>
> You are right; and I should give a bit more credit to their intelligence.
> The source (block-sha1/sha1.c) is short enough that they can figure this
> out for themselves even without any additional comments.
Well, I gave in and added a comment to the patch anyway, with more
improvements in the case of constant values. Patch follows.
> Another issue, especially with your "openssl sha1 removal" patch, is if we
> can assume gcc everywhere. As far as I can tell, block-sha1/sha1.c will
> be the first unconditional use of inline asm or statement expression on
> i386/amd64. Are folks on Solaris and other platforms Ok with this?
I guess we can guard the first with
ifdef(__GNUC__) which should help
people with MSVC. That should take care of x86 at least.
Nicolas
^ permalink raw reply
* Re: [PATCH][resend] git-svn: Respect GIT_SSH setting
From: Junio C Hamano @ 2009-08-18 19:25 UTC (permalink / raw)
To: Karthik R; +Cc: git
In-Reply-To: <4A89EC07.2010402@fastmail.fm>
Karthik R <karthikr@fastmail.fm> writes:
>> Two questions.
>>
>> - What if a user has SVN_SSH exported _and_ wants to use a different one
>> from the one s/he uses for git? Naturally such a user would set both
>> environment variables and differently, but this seems to override the
>> value in SVN_SSH;
>>
> Do you mean user wants to use a different one with "git svn
> ... svn+ssh://" (than the one with "git clone ssh://") ?
Yes.
> In this case
> - defining SVN_SSH, but not GIT_SSH will still work (with this patch,
> GIT_SSH overrides)
Which means if you need to use GIT_SSH to specify one and SVN_SSH to
specify another, you have trouble. IOW, you cannot use anything but
whatever the default is for native git access over ssh:// protocol.
> - but SVN_SSH needs to have \\s.
>
> So unless the user already knew of this quirk, we'll only see
> unescaped \s - so it *does* make sense to escape the \s (if the user
> knew, then too many escaped \s still work).
>
>> - Can a user have SVN_SSH exported, on MSWin32 or msys, and use svn
>> outside git? If so, what does the value of SVN_SSH look like? Does it
>> typically have necessary doubling of backslashes already?
>>
> With subversion for Windows, these \\s are not needed (but doesn't
> cause any break). The doubling is something to do with the bash (in
> msys) I think.
>
Ok, so does that mean the logic should look more like the one you quoted
below without saying yes/no/anything? The points are:
(1) do not muck with SVN_SSH if already given by the user.
(2) when and only when we reuse value from GIT_SSH for SVN_SSH, double
the backslashes.
>> What I am getting at is, if the patch should look something like this
>> instead:
>>
>> if (! exists $ENV{SVN_SSH}) {
>> if (exists $ENV{GIT_SSH}) {
>> $ENV{SVN_SSH} = $ENV{GIT_SSH};
>> if ($^O eq 'MSWin32' || $^O eq 'msys') {
>> $ENV{SVN_SSH} =~ s/\\/\\\\/g;
>> }
>> }
>> }
>>
>>
^ permalink raw reply
* [PATCH] make sure byte swapping is optimal for git
From: Nicolas Pitre @ 2009-08-18 19:26 UTC (permalink / raw)
To: Junio C Hamano
Cc: Sebastian Schuberth, Artur Skawina, Johannes Sixt, msysGit,
Linus Torvalds, Git Mailing List
In-Reply-To: <alpine.LFD.2.00.0908181516510.6044@xanadu.home>
We rely on ntohl() and htonl() to perform byte swapping in many places.
However, some platforms have libraries providing really poor
implementations of those which might cause significant performance
issues, especially with the block-sha1 code.
Signed-off-by: Nicolas Pitre <nico@cam.org>
---
On Tue, 18 Aug 2009, Nicolas Pitre wrote:
> Well, I gave in and added a comment to the patch anyway, with more
> improvements in the case of constant values. Patch follows.
diff --git a/block-sha1/sha1.c b/block-sha1/sha1.c
index 464cb25..d31f2e3 100644
--- a/block-sha1/sha1.c
+++ b/block-sha1/sha1.c
@@ -4,8 +4,8 @@
* and to avoid unnecessary copies into the context array.
*/
-#include <string.h>
-#include <arpa/inet.h>
+/* this is only to get definitions for memcpy(), ntohl() and htonl() */
+#include "../git-compat-util.h"
#include "sha1.h"
diff --git a/compat/bswap.h b/compat/bswap.h
new file mode 100644
index 0000000..7246a12
--- /dev/null
+++ b/compat/bswap.h
@@ -0,0 +1,36 @@
+/*
+ * Let's make sure we always have a sane definition for ntohl()/htonl().
+ * Some libraries define those as a function call, just to perform byte
+ * shifting, bringing significant overhead to what should be a simple
+ * operation.
+ */
+
+/*
+ * Default version that the compiler ought to optimize properly with
+ * constant values.
+ */
+static inline unsigned int default_swab32(unsigned int val)
+{
+ return (((val & 0xff000000) >> 24) |
+ ((val & 0x00ff0000) >> 8) |
+ ((val & 0x0000ff00) << 8) |
+ ((val & 0x000000ff) << 24));
+}
+
+#if defined(__GNUC__) && (defined(__i386__) || defined(__x86_64__))
+
+#define bswap32(x) ({ \
+ unsigned int __res; \
+ if (__builtin_constant_p(x)) { \
+ __res = default_swab32(x); \
+ } else { \
+ __asm__("bswap %0" : "=r" (__res) : "0" (x)); \
+ } \
+ __res; })
+
+#undef ntohl
+#undef htonl
+#define ntohl(x) bswap32(x)
+#define htonl(x) bswap32(x)
+
+#endif
diff --git a/git-compat-util.h b/git-compat-util.h
index 9f941e4..000859e 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -176,6 +176,8 @@ extern char *gitbasename(char *);
#endif
#endif
+#include "compat/bswap.h"
+
/* General helper functions */
extern void usage(const char *err) NORETURN;
extern void die(const char *err, ...) NORETURN __attribute__((format (printf, 1, 2)));
^ permalink raw reply related
* [PATCH] block-sha1: guard gcc extensions with __GNUC__
From: Nicolas Pitre @ 2009-08-18 19:37 UTC (permalink / raw)
To: Junio C Hamano
Cc: Sebastian Schuberth, Artur Skawina, Johannes Sixt, msysGit,
Linus Torvalds, Git Mailing List
In-Reply-To: <alpine.LFD.2.00.0908181516510.6044@xanadu.home>
With this, the code should now be portable to any C compiler.
Signed-off-by: Nicolas Pitre <nico@cam.org>
---
On Tue, 18 Aug 2009, Nicolas Pitre wrote:
> On Tue, 18 Aug 2009, Junio C Hamano wrote:
>
> > Another issue, especially with your "openssl sha1 removal" patch, is if we
> > can assume gcc everywhere. As far as I can tell, block-sha1/sha1.c will
> > be the first unconditional use of inline asm or statement expression on
> > i386/amd64. Are folks on Solaris and other platforms Ok with this?
>
> I guess we can guard the first with ifdef(__GNUC__) which should help
> people with MSVC. That should take care of x86 at least.
Here it is.
diff --git a/block-sha1/sha1.c b/block-sha1/sha1.c
index d31f2e3..92d9121 100644
--- a/block-sha1/sha1.c
+++ b/block-sha1/sha1.c
@@ -9,7 +9,7 @@
#include "sha1.h"
-#if defined(__i386__) || defined(__x86_64__)
+#if defined(__GNUC__) && (defined(__i386__) || defined(__x86_64__))
/*
* Force usage of rol or ror by selecting the one with the smaller constant.
@@ -54,7 +54,7 @@
#if defined(__i386__) || defined(__x86_64__)
#define setW(x, val) (*(volatile unsigned int *)&W(x) = (val))
-#elif defined(__arm__)
+#elif defined(__GNUC__) && defined(__arm__)
#define setW(x, val) do { W(x) = (val); __asm__("":::"memory"); } while (0)
#else
#define setW(x, val) (W(x) = (val))
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox