* Re: how to determine oldest supported version of git
From: Neal Kreitzinger @ 2012-02-03 20:19 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Neal Kreitzinger, git
In-Reply-To: <7v8vkktt6y.fsf@alter.siamese.dyndns.org>
On 2/2/2012 10:52 PM, Junio C Hamano wrote:
> As other people mentioned, if you are on a (probably paid) support plan
> from a(n enterprise) distro, asking them would be the best way, and if you
> are running Git supplied as part of a distro, the distro would dictate the
> version it supplies to you, so asking here would not help very much.
We compile our git from you (and your cohort). (Our paid distro does
not keep up.) Your replies have been *very* helpful.
thanks!
v/r,
neal
^ permalink raw reply
* Re: [PATCH, RFC] Fix build problems related to profile-directed optimization
From: Junio C Hamano @ 2012-02-03 19:58 UTC (permalink / raw)
To: Theodore Tso; +Cc: git, Andi Kleen, Clemens Buchacher
In-Reply-To: <BEE56B27-1C86-4C26-A584-3FF179B7A3D6@mit.edu>
Theodore Tso <tytso@MIT.EDU> writes:
> On Feb 3, 2012, at 1:00 AM, Junio C Hamano wrote:
>
>>
>> * I am happy that this version handles this well:
>>
>> $ make PROFILE=BUILD install
>>
>> even though you did not advertise as such in INSTALL ;-).
>
> I can mention it, although it will mean adding more verbiage about
> profile-directed optimization into the INSTALL.
Oh, sorry, I didn't mean it that way. Please read it as: "Something that
is a natural thing for people to expect after reading what is in INSTALL
works correctly. Yay! Thanks."
^ permalink raw reply
* Re: How best to handle multiple-authorship commits in GIT?
From: Valerie Aurora @ 2012-02-03 19:49 UTC (permalink / raw)
To: David Howells; +Cc: git@vger.kernel.org, Rusty Russell
In-Reply-To: <4681.1328276820@redhat.com>
On Fri, Feb 3, 2012 at 5:47 AM, David Howells <dhowells@redhat.com> wrote:
> Valerie Aurora <valerie.aurora@gmail.com> wrote:
>
>> And for a complete (meaningful) rewrite such as David has done, he
>> changes the commit authorship and adds a Signed-off-by for the
>> original author.
>
> Val[*] hasn't signed off all her patches, and indeed I've merged together some
> patches that she has signed off and some she hasn't. I can't simply add
> Signed-off-by her without her permission. However, if she's willing for me to
> add such lines, then I can do so.
Cc me on the next posting and I'll review and add my Signed-off-by if
it's missing anywhere it should be? Or send them privately, whichever
you prefer.
-VAL
^ permalink raw reply
* Re: [ANNOUNCE] Git 1.7.9
From: Junio C Hamano @ 2012-02-03 19:41 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <201202031352.10279.jnareb@gmail.com>
Jakub Narebski <jnareb@gmail.com> writes:
>> > RPM build errors:
>> > Installed (but unpackaged) file(s) found:
>> > /usr/share/locale/is/LC_MESSAGES/git.mo
>> >
>>
>> I think it should be simply ignored at least for now. I stopped touching
>> the rpm spec since August last year (the only reason I was running rpmbuild
>> was to install them on k.org), so I didn't notice.
>
> So for the time being something like that would be an acceptable fix?
Except that the patch probably wants to go to git.spec.in, removing that
installation target would be a good local workaround for now.
I said "local workaround", meaning that I am not all that interested in
applying such a patch myself. Somebody who wants to do RPM needs to
decide what subpackage(s) it should go before we start accepting more po/
files, and at that time the workaround needs to be reverted.
And hopefully that should happen soonish ;-)
^ permalink raw reply
* Re: Problem pushing to a Novell share
From: Junio C Hamano @ 2012-02-03 19:33 UTC (permalink / raw)
To: git; +Cc: Rüdiger Kessel
In-Reply-To: <CAJ4nRM2+u9OcLgJBN7+_rv1OiXETGNhozAfvYtj=hVNTfCMxhQ@mail.gmail.com>
[jc: just forwarding to the list --- the msysgit list might have been a
better place to forward to, but I know msysgit folks read this list too,
so...]
Rüdiger Kessel <ruediger.kessel@gmail.com> writes:
I have confimed that it is not a mingw problem. Direct calls to
CreateFile() in kernel32.dll show the same behaviour.
On a Novel share I get Error 80 "File exists" if a directory does not
exist, while I get Error 3 "Path not found" on a local drive
2012/2/3 Rüdiger Kessel <ruediger.kessel@gmail.com>:
Well, it is a Novell bug, but what can we do?
I studied it in detail and open() sets errno=EEXIST if the directory
is missing on a Novell share causing git_mkstemp_mode looping 16k
times and never creates the dir.
The correct errno would be ENOENT, but that not what the systems is sending.
So it is either using stat on the directory before trying to create
the tmp-file or GIT cannot be used with Novell shares. What is it
going to be?
Protocolling the mkdir error handling I saw that git creates quite a
number of new directories this way. Therefore my pessimistic approach
might be too costly at the end.
^ permalink raw reply
* Re: [PATCH, RFC] Fix build problems related to profile-directed optimization
From: Andi Kleen @ 2012-02-03 18:39 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Ted Ts'o, git, Clemens Buchacher
In-Reply-To: <7vr4ycsbga.fsf@alter.siamese.dyndns.org>
Thanks everyone for improving this.
I should add that any improvements will depend on your compiler version.
I would also expect better numbers when combined with LTO in gcc 4.7,
but haven't tried so far.
BTW it would be really nice to figure out a subset of the test suite
that runs faster and gives similar speedup like the full one.
-Andi
--
ak@linux.intel.com -- Speaking for myself only
^ permalink raw reply
* Re: How best to handle multiple-authorship commits in GIT?
From: Junio C Hamano @ 2012-02-03 18:27 UTC (permalink / raw)
To: David Howells; +Cc: Valerie Aurora, git@vger.kernel.org, Rusty Russell
In-Reply-To: <4681.1328276820@redhat.com>
David Howells <dhowells@redhat.com> writes:
> Valerie Aurora <valerie.aurora@gmail.com> wrote:
>
>> And for a complete (meaningful) rewrite such as David has done, he
>> changes the commit authorship and adds a Signed-off-by for the
>> original author.
>
> Val[*] hasn't signed off all her patches, and indeed I've merged together some
> patches that she has signed off and some she hasn't. I can't simply add
> Signed-off-by her without her permission. However, if she's willing for me to
> add such lines, then I can do so.
>
>> Signed-off-by: Some Upstream Author
>> Signed-off-by: Maintainer or Merger (rewrote error handling)
>
> And if the changes are more than can be put in what's left of the line? I
> would've thought it would make more sense to do something like:
>
> Signed-off-by: Valerie Aurora <valerie.aurora@gmail.com> (Original author)
> Signed-off-by: David Howells <dhowells@redhat.com> (Further development)
>
> David
That all sounds sensible.
I personally think the "recognition" factor Valerie alluded to in one of
her earlier message is a real and important issue, but I do not think
adding arbitrary number of "author" headers to the commit object would
help very much to solve it, for various reasons:
* While we made it easy to run "git shortlog -s -n --since=3.months" and
congratulate himself with "I now am the third most active person!" for
anybody, Git itself does not ship an equally easy way to analyze other
kinds of contributions to your project. I am merely a bystander, but
if I recall correctly, there were discussions on how to recognize
contributions by bug-reporters and testers using the history stored in
Git on the kernel list. The types of contribution you would want to
recognize however would be different from project to project. For that
kind of analysis, you would be better off doing something like what
lwn.net does, mining the text from the message part of the log.
* Even if we limit the issue to "who wrote X" (replace X with the name of
any piece of software), taking "author" field as anything more than an
approximation would be asking for a trouble. Not all patches are of
equal impact and importance.
* You would also have to think about how you would present "git shortlog"
output if you updated Git to record more than one "author" field in the
commit header. If Valerie wrote 27 patches by herself, 33 patches
together with you sitting next to each other, 17 patches with somebody
else, how would the entries for her, you and the third person look
like? Or would combinations of "Valerie & David", "Valerie & the
third person", etc. have separate entries in the output?
In short, I would say that you should take the name recorded in the
"author" field nothing more than the primary contact for a particular
commit to be used in case others have question on it later.
^ permalink raw reply
* Re: [PATCH, RFC] Fix build problems related to profile-directed optimization
From: Theodore Tso @ 2012-02-03 18:19 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Theodore Tso, git, Andi Kleen, Clemens Buchacher
In-Reply-To: <7vr4ycsbga.fsf@alter.siamese.dyndns.org>
On Feb 3, 2012, at 1:00 AM, Junio C Hamano wrote:
>
> * I am happy that this version handles this well:
>
> $ make PROFILE=BUILD install
>
> even though you did not advertise as such in INSTALL ;-).
I can mention it, although it will mean adding more verbiage about profile-directed optimization into the INSTALL.
My assumption was that people who did this would usually be installing into --prefix=/usr as root, but there certainly will be anal people like myself who want to install profile-optimized binaries into ~/bin. :-)
> * However, I think "clean" target should remove *.gcda unconditionally.
>
> $ make PROFILE=BUILD install ; make clean ; git clean -n -x | grep gcda
Will fix.
> * Running "make PROFILE=BUILD install" immediately after another one,
> without "make clean" in between, resulted in full rebuild and test
> before the second "install", which somewhat surprised me. I however do
> not think this is a big show-stopper problem.
Hmm… that surprises me too. If
make PROFILE=BUILD all
make PROFILE=BUILD install
works correctly, I don't understand why a second "make PROFILE=BUILD install" issued after the above sequence would result in complete rebuild and test pass, unless something in the "make install" rules is modifying the build tree as a side-effect of the install pass, which I'd argue is a bug. I'll take a look at it.
-- Ted
^ permalink raw reply
* Re: [RFC/PATCH 0/2] Commits with ancient timestamps
From: Junio C Hamano @ 2012-02-03 18:01 UTC (permalink / raw)
To: Thomas Rast; +Cc: git, Han-Wen Nienhuys
In-Reply-To: <87mx90yz5y.fsf@thomas.inf.ethz.ch>
Thomas Rast <trast@inf.ethz.ch> writes:
> Doing this just makes me wonder how important exactly the 1970-1975
> range is. Is there a notable software history from that era that can be
> recovered?
That is not really a valid question. People who wrote private stuff in
that era deserve to be users of Git, too.
> (Your [1/2] does not seem to parse negative offsets from the unix epoch,
> so anything before 1970 is still out.)
Yes, pre-epoch timestamps are also useful for projects like US
Constitution.
http://thread.gmane.org/gmane.comp.version-control.git/152433/focus=152725
For that, we would need to use and pass around time_t (or intmax_t if we
follow the reason why originally Linus chose to avoid time_t) throughout
the codebase. If you actually write commit objects that record pre-epoch
timestamps, however, they will become unreadable by the current versions
of Git (as they would not understand such a negative raw timestamp).
In any case, that is a goal for a much longer term.
But even after such a change happens, you still need a way for Git to
replay a raw timestamp stored in commit objects without regressing the
parse_date() interface too much. These two patches show one way to do so
with minimum disruption.
As an added bonus, with the second patch, the way to spell a raw timestamp
happens to become compatible with how GNU date accepts one, i.e.
$ date -d @1000000000
even though we do not have to encourage the use of this notation by humans,
tools and script writers may find it useful.
^ permalink raw reply
* Re: [PATCH/RFCv2 (version B)] gitweb: Allow UTF-8 encoded CGI query parameters and path_info
From: Michał Kiedrowicz @ 2012-02-03 17:45 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <201202031344.55750.jnareb@gmail.com>
Jakub Narebski <jnareb@gmail.com> wrote:
> Gitweb tries hard to properly process UTF-8 data, by marking output
> from git commands and contents of files as UTF-8 with to_utf8()
> subroutine. This ensures that gitweb would print correctly UTF-8
> e.g. in 'log' and 'commit' views.
>
> Unfortunately it misses another source of potentially Unicode input,
> namely query parameters. The result is that one cannot search for a
> string containing characters outside US-ASCII. For example searching
> for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
> WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
> bytes in UTF-8 and percent-encoded as %C5%81) result in the following
> incorrect data in search field
>
> Michał Kiedrowicz
>
> This is caused by CGI by default treating '0xc5 0x82' bytes as two
> characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
> query parameter is not processed explicitly as UTF-8 encoded string.
>
> The solution used here follows "Using Unicode in a Perl CGI script"
> article on http://www.lemoda.net/cgi/perl-unicode/index.html:
>
> use CGI;
> use Encode 'decode_utf8;
> my $value = params('input');
> $value = decode_utf8($value);
>
> Decoding UTF-8 is done when filling %input_params hash and $path_info
> variable; the former required to move from explicit $cgi->param(<label>)
> to $input_params{<name>} in a few places, which is a good idea anyway.
>
> Another required change was to add -override=>1 parameter to
> $cgi->textfield() invocation (in search form). Otherwise CGI would
> use values from query string if it is present, filling value from
> $cgi->param... without decode_utf8(). As we are using value of
> appropriate parameter anyway, -override=>1 doesn't change the
> situation but makes gitweb fill search field correctly.
>
> Alternate solution would be to simply use the '-utf8' pragma (via
> "use CGI '-utf8';"), but according to CGI.pm documentation it may
> cause problems with POST requests containing binary files... and
> it requires CGI 3.31 (I think), released with perl v5.8.9.
>
> Noticed-by: Michał Kiedrowicz <michal.kiedrowicz@gmail.com>
> Signed-off-by: Jakub Narębski <jnareb@gmail.com>
> ---
> On Fri, 3 Feb 2012, Michal Kiedrowicz wrote:
> > Jakub Narebski <jnareb@gmail.com> wrote:
>
> > > Is it what you mean by "this doesn't work for me", i.e. working
> > > search, garbage in search field?
> >
> > I mean "garbage in search field". Search works even without the patch
> > (at least on Debian with git-1.7.7.3, perl-5.10.1 and CGI-3.43; I
> > don't have my notebook nearby at the moment to check).
> [...]
>
> > > Damn. If we use $cgi->textfield(-name => "s", -value => $searchtext)
> > > like in gitweb, CGI.pm would read $cgi->param("s") by itself -
> > > without decoding.
> >
> > Makes sense. When I tried calling to_utf8() in the line that defines
> > textfield (this was my first approach to this problem), it haven't
> > changed anything.
>
> Yes, and it doesn't makes sense in gitweb case - we use value of
> $cgi->param("s") as default value of text field anyway, but in
> Unicode-aware way.
>
> > > To skip this we need to pass -force=>1 or
> > > -override=>1 (i.e. further changes to gitweb).
>
> This patch does this.
>
> Does it make work for you?
>
Yes, it works for me. Search form properly displays "ł". Thanks!
^ permalink raw reply
* Re: Alternates corruption issue
From: Junio C Hamano @ 2012-02-03 17:38 UTC (permalink / raw)
To: Jeff King
Cc: Jonathan Nieder, Richard Purdie, GIT Mailing-list, Hart, Darren,
Ashfield, Bruce
In-Reply-To: <20120203120215.GA31441@sigill.intra.peff.net>
Jeff King <peff@peff.net> writes:
> However, with the ordering change, there is a technically a regression
> in one case: a random file "foo" next to a repo "foo.git". Saying "git
> ls-remote foo" used to prefer "foo.git", and will now select the file
> "foo" only to fail.
Yeah, very true X-<.
> Thanks for noticing. I saw this issue when I was writing the original
> version of the patch, and meant to revisit it and at least document it
> in the commit message, but I ended up forgetting.
No, thanks for working on this.
^ permalink raw reply
* Re: Push from an SSH Terminal
From: Neal Groothuis @ 2012-02-03 17:10 UTC (permalink / raw)
To: Feanil Patel; +Cc: git
In-Reply-To: <CAG94OYxbOYCjd5qNBh8EF2gyezHWMqX1-R2MYgk8gkFYcrMjuQ@mail.gmail.com>
> On Fri, Feb 3, 2012 at 11:21 AM, Neal Groothuis <ngroot@lo-cal.org> wrote:
>>> The
>>> repository is password protected so if I'm physically at Comp B, I get
>>> a gui prompt for my username and password. However Comp A does not
>>> have X Forwarding setup to Comp B so I can't get the gui interface for
>>> the username and password when I try to do the push. Â Is there an
>>> alternative way to provide my credentials when doing a git push that
>>> does not require a gui?
>>
>> What protocol are you using to access the repository on Comp C?
>>
> I'm pulling and pushing over HTTP from Comp C.
Check to see if the GIT_ASKPASS and/or SSH_ASKPASS environment variables
are set, and if the core.askpass config variable is set. If any of these
are set, unset them. Git should fall back to a simple password prompt.
- Neal
^ permalink raw reply
* Re: Git performance results on a large repository
From: Joshua Redstone @ 2012-02-03 17:00 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason; +Cc: git@vger.kernel.org
In-Reply-To: <CACBZZX4BsFZxB6A-Hg-k37FBavgTV8SDiQTK_sVh9Mb9iskiEw@mail.gmail.com>
Hi Ævar,
Thanks for the comments. I've included a bunch more info on the test repo
below. It is based on a growth model of two of our current repositories
(I.e., it's not a perforce import). We already have some of the easily
separable projects in separate repositories, like HPHP. If we could
split our largest repos into multiple ones, that would help the scaling
issue. However, the code in those repos is rather interdependent and we
believe it'd hurt more than help to split it up, at least for the
medium-term future. We derive a fair amount of benefit from the code
sharing and keeping things together in a single repo, so it's not clear
when it'd make sense to get more aggressive splitting things up.
Some more information on the test repository: The working directory is
9.5 GB, the median file size is 2 KB. The average depth of a directory
(counting the number of '/'s) is 3.6 levels and the average depth of a
file is 4.6. More detailed histograms of the repository composition is
below:
------------------------
Histogram of depth of every directory in the repo (dirs=`find . -type d` ;
(for dir in $dirs; do t=${dir//[^\/]/}; echo ${#t} ; done) |
~/tmp/histo.py)
* The .git directory itself has only 161 files, so although included,
doesn't affect the numbers significantly)
[0.0 - 1.3): 271
[1.3 - 2.6): 9966
[2.6 - 3.9): 56595
[3.9 - 5.2): 230239
[5.2 - 6.5): 67394
[6.5 - 7.8): 22868
[7.8 - 9.1): 6568
[9.1 - 10.4): 420
[10.4 - 11.7): 45
[11.7 - 13.0]: 21
n=394387 mean=4.671830, median=5.000000, stddev=1.272658
Histogram of depth of every file in the repo (files=`git ls-files` ; (for
file in $files; do t=${file//[^\/]/}; echo ${#t} ; done) | ~/tmp/histo.py)
* 'git ls-files' does not prefix entries with ./, like the 'find' command
above, does, hence why the average appears to be the same as the directory
stats
[0.0 - 1.3]: 1274
[1.3 - 2.6]: 35353
[2.6 - 3.9]: 196747
[3.9 - 5.2]: 786647
[5.2 - 6.5]: 225913
[6.5 - 7.8]: 77667
[7.8 - 9.1]: 22130
[9.1 - 10.4]: 1599
[10.4 - 11.7]: 164
[11.7 - 13.0]: 118
n=1347612 mean=4.655750, median=5.000000, stddev=1.278399
Histogram of file sizes (for first 50k files - this command takes a
while): files=`git ls-files` ; (for file in $files; do stat -c%s $file ;
done) | ~/tmp/histo.py
[ 0.0 - 4.7): 0
[ 4.7 - 22.5): 2
[ 22.5 - 106.8): 0
[ 106.8 - 506.8): 0
[ 506.8 - 2404.7): 31142
[ 2404.7 - 11409.9): 17837
[ 11409.9 - 54137.1): 942
[ 54137.1 - 256866.9): 53
[ 256866.9 - 1218769.7): 18
[ 1218769.7 - 5782760.0]: 5
n=49999 mean=3590.953239, median=1772.000000, stddev=42835.330259
Cheers,
Josh
On 2/3/12 9:56 AM, "Ævar Arnfjörð Bjarmason" <avarab@gmail.com> wrote:
>On Fri, Feb 3, 2012 at 15:20, Joshua Redstone <joshua.redstone@fb.com>
>wrote:
>
>> We (Facebook) have been investigating source control systems to meet our
>> growing needs. We already use git fairly widely, but have noticed it
>> getting slower as we grow, and we want to make sure we have a good story
>> going forward. We're debating how to proceed and would like to solicit
>> people's thoughts.
>
>Where I work we also have a relatively large Git repository. Around
>30k files, a couple of hundred thousand commits, clone size around
>half a GB.
>
>You haven't supplied background info on this but it really seems to me
>like your testcase is converting something like a humongous Perforce
>repository directly to Git.
>
>While you /can/ do this it's not a good idea, you should split up
>repositories at the boundaries code or data doesn't directly cross
>over, e.g. there's no reason why you need HipHop PHP in the same
>repository as Cassandra or the Facebook chat system, is there?
>
>While Git could better with large repositories (in particular applying
>commits in interactive rebase seems to be to slow down on bigger
>repositories) there's only so much you can do about stat-ing 1.3
>million files.
>
>A structure that would make more sense would be to split up that giant
>repository into a lot of other repositories, most of them probably
>have no direct dependencies on other components, but even those that
>do can sometimes just use some other repository as a submodule.
>
>Even if you have the requirement that you'd like to roll out
>*everything* at a certain point in time you can still solve that with
>a super-repository that has all the other ones as submodules, and
>creates a tag for every rollout or something like that.
^ permalink raw reply
* Re: Push from an SSH Terminal
From: Feanil Patel @ 2012-02-03 16:40 UTC (permalink / raw)
To: Neal Groothuis; +Cc: git
In-Reply-To: <21607.38.96.167.131.1328286083.squirrel@mail.lo-cal.org>
On Fri, Feb 3, 2012 at 11:21 AM, Neal Groothuis <ngroot@lo-cal.org> wrote:
>> The
>> repository is password protected so if I'm physically at Comp B, I get
>> a gui prompt for my username and password. However Comp A does not
>> have X Forwarding setup to Comp B so I can't get the gui interface for
>> the username and password when I try to do the push. Is there an
>> alternative way to provide my credentials when doing a git push that
>> does not require a gui?
>
> What protocol are you using to access the repository on Comp C?
>
> - Neal
>
I'm pulling and pushing over HTTP from Comp C.
^ permalink raw reply
* Re: Push from an SSH Terminal
From: Neal Groothuis @ 2012-02-03 16:21 UTC (permalink / raw)
To: Feanil Patel; +Cc: git
In-Reply-To: <CAG94OYxX5foffvaFLQv7=wXguGC2TLgccdDFrC+ERzv_gXZ=ug@mail.gmail.com>
> The
> repository is password protected so if I'm physically at Comp B, I get
> a gui prompt for my username and password. However Comp A does not
> have X Forwarding setup to Comp B so I can't get the gui interface for
> the username and password when I try to do the push. Is there an
> alternative way to provide my credentials when doing a git push that
> does not require a gui?
What protocol are you using to access the repository on Comp C?
- Neal
^ permalink raw reply
* Push from an SSH Terminal
From: Feanil Patel @ 2012-02-03 15:50 UTC (permalink / raw)
To: git
Hi Everyone,
I tried looking for an answer to my problem online without much luck,
perhaps you can help me. I'm SSHed from my laptop(Comp A) over to a
computer(Comp B) that has my git repo on it. I made some changes and
comitted them. Now I want to push them to my other server(Comp C). The
repository is password protected so if I'm physically at Comp B, I get
a gui prompt for my username and password. However Comp A does not
have X Forwarding setup to Comp B so I can't get the gui interface for
the username and password when I try to do the push. Is there an
alternative way to provide my credentials when doing a git push that
does not require a gui?
--
Feanil Patel
^ permalink raw reply
* Re: Git performance results on a large repository
From: Ævar Arnfjörð Bjarmason @ 2012-02-03 14:56 UTC (permalink / raw)
To: Joshua Redstone; +Cc: git@vger.kernel.org
In-Reply-To: <CB5074CF.3AD7A%joshua.redstone@fb.com>
On Fri, Feb 3, 2012 at 15:20, Joshua Redstone <joshua.redstone@fb.com> wrote:
> We (Facebook) have been investigating source control systems to meet our
> growing needs. We already use git fairly widely, but have noticed it
> getting slower as we grow, and we want to make sure we have a good story
> going forward. We're debating how to proceed and would like to solicit
> people's thoughts.
Where I work we also have a relatively large Git repository. Around
30k files, a couple of hundred thousand commits, clone size around
half a GB.
You haven't supplied background info on this but it really seems to me
like your testcase is converting something like a humongous Perforce
repository directly to Git.
While you /can/ do this it's not a good idea, you should split up
repositories at the boundaries code or data doesn't directly cross
over, e.g. there's no reason why you need HipHop PHP in the same
repository as Cassandra or the Facebook chat system, is there?
While Git could better with large repositories (in particular applying
commits in interactive rebase seems to be to slow down on bigger
repositories) there's only so much you can do about stat-ing 1.3
million files.
A structure that would make more sense would be to split up that giant
repository into a lot of other repositories, most of them probably
have no direct dependencies on other components, but even those that
do can sometimes just use some other repository as a submodule.
Even if you have the requirement that you'd like to roll out
*everything* at a certain point in time you can still solve that with
a super-repository that has all the other ones as submodules, and
creates a tag for every rollout or something like that.
^ permalink raw reply
* Re: [RFC/PATCH 0/2] Commits with ancient timestamps
From: Han-Wen Nienhuys @ 2012-02-03 14:53 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <1328218903-5681-1-git-send-email-gitster@pobox.com>
On Thu, Feb 2, 2012 at 7:41 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Result of conversion of ancient history from other SCMs, and output from
> other third-party tools, can record timestamps that predates inception of
> Git. They can cause "git am", "git rebase" and "git commit --amend" to
> misbehave, because the raw git timestamp e.g.
>
> author <a.u.thor@example.com> 1328214896 -0800
>
> are read from the commit object and passed to parse_date() machinery,
As a bit of context: we have some internal tools at Google that create
administrative commits that should have no timestamp. I am using "0"
and "1" as a deterministic timestamps in these cases (ie. the start of
the epoch). While this works well in general, there are some git
subcommands that barf on this, causing user-unhappiness.
This patch will hopefully resolve these breakages.
--
Han-Wen Nienhuys
Google Engineering Belo Horizonte
hanwen@google.com
^ permalink raw reply
* Re: Alternates corruption issue
From: Richard Purdie @ 2012-02-03 14:40 UTC (permalink / raw)
To: Jeff King
Cc: Jonathan Nieder, Junio C Hamano, GIT Mailing-list, Hart, Darren,
Ashfield, Bruce
In-Reply-To: <20120202215913.GA26727@sigill.intra.peff.net>
On Thu, 2012-02-02 at 16:59 -0500, Jeff King wrote:
> On Tue, Jan 31, 2012 at 04:22:58PM -0600, Jonathan Nieder wrote:
>
> > Anyway, thanks for explaining. Hopefully I can get to this soon and
> > factor out a common function for get_repo_path and enter_repo to call
> > so playing with the ordering becomes a little less scary. ;-)
>
> So here's what I think we should apply to fix the particular issue that
> Richard mentioned at the start of this thread.
>
> Besides tweaking the ordering, the main contribution is a set of tests
> that actually check some of these ambiguous cases (especially checking
> the fact that both code paths behave identically!). I didn't factor the
> logic into a common function, but doing so should be a little safer on
> top of these tests, if you're still interested.
I didn't have much to add to the discussion yesterday but this solution
looks good to me and should resolve the problems I was seeing.
Thanks!
Richard
^ permalink raw reply
* Re: Breakage in master?
From: Joel C. Salomon @ 2012-02-03 14:22 UTC (permalink / raw)
To: Erik Faye-Lund
Cc: Jeff King, Git Mailing List, msysGit,
Ævar Arnfjörð
In-Reply-To: <CABPQNSbQTF1UiDuOZkX-KrTQ7oFyVvx6FxZ85c9uCF2FFUtTSg@mail.gmail.com>
On 02/03/2012 09:05 AM, Erik Faye-Lund wrote:
> On Fri, Feb 3, 2012 at 2:55 PM, Joel C. Salomon <joelcsalomon@gmail.com> wrote:
>> That doesn't sound right; MinGW defaults to linking to a fairly old
>> version of the Windows C library (MSVCRT.dll from Visual Studio 6,
>> IIRC).
>
> I'm not entirely sure what you are arguing. On MinGW, calling
> vsnprintf vs _vsnprintf leads to different implementations on MinGW.
> This is documented in the release-notes:
> http://sourceforge.net/project/shownotes.php?release_id=24832
Never mind; I stand corrected. It's been some time since I actively
followed MinGW development.
--Joel
^ permalink raw reply
* Git performance results on a large repository
From: Joshua Redstone @ 2012-02-03 14:20 UTC (permalink / raw)
To: git@vger.kernel.org
Hi Git folks,
We (Facebook) have been investigating source control systems to meet our
growing needs. We already use git fairly widely, but have noticed it
getting slower as we grow, and we want to make sure we have a good story
going forward. We're debating how to proceed and would like to solicit
people's thoughts.
To better understand git scalability, I've built up a large, synthetic
repository and measured a few git operations on it. I summarize the
results here.
The test repo has 4 million commits, linear history and about 1.3 million
files. The size of the .git directory is about 15GB, and has been
repacked with 'git repack -a -d -f --max-pack-size=10g --depth=100
--window=250'. This repack took about 2 days on a beefy machine (I.e.,
lots of ram and flash). The size of the index file is 191 MB. I can share
the script that generated it if people are interested - It basically picks
2-5 files, modifies a line or two and adds a few lines at the end
consisting of random dictionary words, occasionally creates a new file,
commits all the modifications and repeats.
I timed a few common operations with both a warm OS file cache and a cold
cache. i.e., I did a 'echo 3 | tee /proc/sys/vm/drop_caches' and then did
the operation in question a few times (first timing is the cold timing,
the next few are the warm timings). The following results are on a server
with average hard drive (I.e., not flash) and > 10GB of ram.
'git status' : 39 minutes cold, and 24 seconds warm.
'git blame': 44 minutes cold, 11 minutes warm.
'git add' (appending a few chars to the end of a file and adding it): 7
seconds cold and 5 seconds warm.
'git commit -m "foo bar3" --no-verify --untracked-files=no --quiet
--no-status': 41 minutes cold, 20 seconds warm. I also hacked a version
of git to remove the three or four places where 'git commit' stats every
file in the repo, and this dropped the times to 30 minutes cold and 8
seconds warm.
The git performance we observed here is too slow for our needs. So the
question becomes, if we want to keep using git going forward, what's the
best way to improve performance. It seems clear we'll probably need some
specialized servers (e.g., to perform git-blame quickly) and maybe
specialized file system integration to detect what files have changed in a
working tree.
One way to get there is to do some deep code modifications to git
internals, to, for example, create some abstractions and interfaces that
allow plugging in the specialized servers. Another way is to leave git
internals as they are and develop a layer of wrapper scripts around all
the git commands that do the necessary interfacing. The wrapper scripts
seem perhaps easier in the short-term, but may lead to increasing
divergence from how git behaves natively and also a layer of complexity.
Thoughts?
Cheers,
Josh
^ permalink raw reply
* Re: Breakage in master?
From: Erik Faye-Lund @ 2012-02-03 14:05 UTC (permalink / raw)
To: Joel C. Salomon
Cc: Jeff King, Git Mailing List, msysGit,
Ævar Arnfjörð
In-Reply-To: <4F2BE759.4000902@gmail.com>
On Fri, Feb 3, 2012 at 2:55 PM, Joel C. Salomon <joelcsalomon@gmail.com> wrote:
> On 02/03/2012 07:28 AM, Erik Faye-Lund wrote:
>> On Thu, Feb 2, 2012 at 6:46 PM, Jeff King <peff@peff.net> wrote:
>>> vsnprintf should generally never be returning -1 (it should return the
>>> number of characters that would have been written). Since you're on
>>> Windows, I assume you're using the replacement version in
>>> compat/snprintf.c.
>>
>> No. SNPRINTF_RETURNS_BOGUS is only set for the MSVC target, not for
>> the MinGW target. I'm assuming that means MinGW-runtime has a sane
>> vsnprintf implementation.
>
> That doesn't sound right; MinGW defaults to linking to a fairly old
> version of the Windows C library (MSVCRT.dll from Visual Studio 6,
> IIRC). According to <http://mingw.org/wiki/C99> there exists libmingwex
> with some functions (especially those from <stdio.h>) replaced for
> Standard compatibility, but it's "far from complete". (Is msysGit using
> it anyway?)
I'm not entirely sure what you are arguing. On MinGW, calling
vsnprintf vs _vsnprintf leads to different implementations on MinGW.
This is documented in the release-notes:
http://sourceforge.net/project/shownotes.php?release_id=24832
"As in previous releases, the MinGW implementations of snprintf() and
vsnprintf() are the default for these two functions, with the MSVCRT
alternatives being called as _snprintf() and _vsnprintf()."
I don't see how this is contradicted by your argument of a third,
C99-ish implementation. I'm pretty sure the "far from complete"-part
is about the C99-features anyway.
^ permalink raw reply
* Re: Breakage in master?
From: Joel C. Salomon @ 2012-02-03 13:55 UTC (permalink / raw)
To: kusmabite
Cc: Jeff King, Git Mailing List, msysGit,
Ævar Arnfjörð
In-Reply-To: <CABPQNSZfKCTsuusPpHa2djEOeGVN9z5s_Fr+S3EaHiv7Q4Re9w@mail.gmail.com>
On 02/03/2012 07:28 AM, Erik Faye-Lund wrote:
> On Thu, Feb 2, 2012 at 6:46 PM, Jeff King <peff@peff.net> wrote:
>> vsnprintf should generally never be returning -1 (it should return the
>> number of characters that would have been written). Since you're on
>> Windows, I assume you're using the replacement version in
>> compat/snprintf.c.
>
> No. SNPRINTF_RETURNS_BOGUS is only set for the MSVC target, not for
> the MinGW target. I'm assuming that means MinGW-runtime has a sane
> vsnprintf implementation.
That doesn't sound right; MinGW defaults to linking to a fairly old
version of the Windows C library (MSVCRT.dll from Visual Studio 6,
IIRC). According to <http://mingw.org/wiki/C99> there exists libmingwex
with some functions (especially those from <stdio.h>) replaced for
Standard compatibility, but it's "far from complete". (Is msysGit using
it anyway?)
--Joel
^ permalink raw reply
* Re: How best to handle multiple-authorship commits in GIT?
From: David Howells @ 2012-02-03 13:47 UTC (permalink / raw)
To: Valerie Aurora; +Cc: dhowells, git@vger.kernel.org, Rusty Russell
In-Reply-To: <CAD-XujkVK=tOtmVS90U0KAutFZ55jxsHMKuuMppXOi-H6ZY=RQ@mail.gmail.com>
Valerie Aurora <valerie.aurora@gmail.com> wrote:
> And for a complete (meaningful) rewrite such as David has done, he
> changes the commit authorship and adds a Signed-off-by for the
> original author.
Val[*] hasn't signed off all her patches, and indeed I've merged together some
patches that she has signed off and some she hasn't. I can't simply add
Signed-off-by her without her permission. However, if she's willing for me to
add such lines, then I can do so.
> Signed-off-by: Some Upstream Author
> Signed-off-by: Maintainer or Merger (rewrote error handling)
And if the changes are more than can be put in what's left of the line? I
would've thought it would make more sense to do something like:
Signed-off-by: Valerie Aurora <valerie.aurora@gmail.com> (Original author)
Signed-off-by: David Howells <dhowells@redhat.com> (Further development)
David
[*] Apologies for talking about/to you in the third person, Val.
^ permalink raw reply
* Re: [PATCH] t0300-credentials: Word around a solaris /bin/sh bug
From: Ben Walton @ 2012-02-03 13:45 UTC (permalink / raw)
To: Jeff King; +Cc: Junio C Hamano, git
In-Reply-To: <20120203120657.GB31441@sigill.intra.peff.net>
Excerpts from Jeff King's message of Fri Feb 03 07:06:57 -0500 2012:
> When I have hard-coded "#!/bin/sh", my thinking is usually "this is
> less cumbersome to type and to read, and this script-let is so small
> that even Solaris will get it right".
This is why I opted to stick with /bin/sh and just avoid the damage.
Overall, using a sane shell is a better option...It is harder to read
though.
Thanks
-Ben
--
Ben Walton
Systems Programmer - CHASS
University of Toronto
C:416.407.5610 | W:416.978.4302
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox