* Installing git-svn on Linux without root
From: Andrew Keller @ 2012-02-04 2:10 UTC (permalink / raw)
To: Git List
I am attempting to install git, including the ability to access subversion repositories on a Linux machine. I do not have root access on the machine, so I prepended my PATH with a folder in my home directory.
Installing Git worked just fine, but when I try to clone a subversion repository, I get:
$ git svn clone file:///svn --prefix=svn/ --no-metadata --trunk=dba/trunk --branches=dba/branches --tags=dba/tags dba
Initialized empty Git repository in /home/kelleran/Documents/togit/converted/dba/.git/
Can't locate SVN/Core.pm in @INC (@INC contains: /homedirs/kelleran/local/lib/perl5/site_perl/5.8.8 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/5.8.8 .) at /homedirs/kelleran/local/libexec/git-core/git-svn line 41.
Google suggested that the above error could be due to missing perl bindings. So, I installed swig, and followed the instructions for installing the perl bindings: http://svn.apache.org/repos/asf/subversion/trunk/subversion/bindings/swig/INSTALL (I used the alternate build steps, since I had to set the prefix).
Unfortunately, I still get exactly the same error. So, I looked to see whether or not the missing library was installed:
$ find ~/local -iname Core.pm
/homedirs/kelleran/local/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/SVN/Core.pm
So, the module does exist, but not in a location included by @INC. This sounds like a simple misconfiguration during the installation on my part, but after reading the manuals and searching the web, I was unable to find a parameter that gets git to be able to see the perl bindings.
I'm guessing I need either more sleep or a fresh point of view. Any thoughts?
The machine is running Linux 2.6.32 64-bit, and has perl 5.8.8. Of the software I installed, I am using:
libpcre 8.21
swig 2.8.4
neon 0.29.6
apr 1.3.x
apr-util 1.3.x
subversion 1.7.2
git 1.7.9
Thanks,
Andrew Keller
^ permalink raw reply
* Re: Git performance results on a large repository
From: Evgeny Sazhin @ 2012-02-04 1:25 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Joshua Redstone, git@vger.kernel.org
In-Reply-To: <CACBZZX4BsFZxB6A-Hg-k37FBavgTV8SDiQTK_sVh9Mb9iskiEw@mail.gmail.com>
On Feb 3, 2012, at 9:56 AM, Ævar Arnfjörð Bjarmason wrote:
> On Fri, Feb 3, 2012 at 15:20, Joshua Redstone <joshua.redstone@fb.com> wrote:
>
>> We (Facebook) have been investigating source control systems to meet our
>> growing needs. We already use git fairly widely, but have noticed it
>> getting slower as we grow, and we want to make sure we have a good story
>> going forward. We're debating how to proceed and would like to solicit
>> people's thoughts.
>
> Where I work we also have a relatively large Git repository. Around
> 30k files, a couple of hundred thousand commits, clone size around
> half a GB.
>
> You haven't supplied background info on this but it really seems to me
> like your testcase is converting something like a humongous Perforce
> repository directly to Git.
>
> While you /can/ do this it's not a good idea, you should split up
> repositories at the boundaries code or data doesn't directly cross
> over, e.g. there's no reason why you need HipHop PHP in the same
> repository as Cassandra or the Facebook chat system, is there?
>
> While Git could better with large repositories (in particular applying
> commits in interactive rebase seems to be to slow down on bigger
> repositories) there's only so much you can do about stat-ing 1.3
> million files.
>
> A structure that would make more sense would be to split up that giant
> repository into a lot of other repositories, most of them probably
> have no direct dependencies on other components, but even those that
> do can sometimes just use some other repository as a submodule.
>
> Even if you have the requirement that you'd like to roll out
> *everything* at a certain point in time you can still solve that with
> a super-repository that has all the other ones as submodules, and
> creates a tag for every rollout or something like that.
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
I concur. I'm working in the company with many years of development history with several huge CVS repos and we are slowly but surely migrating the codebase from CVS to Git.
Split the things up. This will allow you to reorganize things better and there is IMHO no downsides.
As for rollout - i think this job should be given to build/release system that will have an ability to gather necessary code from different repos and tag it properly.
just my 2 cents
Thanks,
Eugene
^ permalink raw reply
* [RFC/PATCH] verify-tag: check sig of all tags to given object
From: Tom Grennan @ 2012-02-04 1:25 UTC (permalink / raw)
To: git; +Cc: jasampler
If the command argument is a non-tag object, scan and verify all tags to
the given object; for example:
john$ git tag -s -m "I approve" john-README master:README
...
john$ git tag -s -m "I recommend" john-HEAD HEAD
...
john$ git push <url> tag john-README
john$ git push <url> tag john-HEAD
jane$ git fetch --tags <url>
jane$ git tag -s -m "I also approve" jane-README master:README
...
jane$ git push <url> tag jane-README
jeff$ git fetch --tags <url>
jeff$ git verify-tag master:README
tag john-README: OK
tag jane-README: OK
jeff$ git verify-tag HEAD
tag john-HEAD: OK
Signed-off-by: Tom Grennan <tom.grennan@ericsson.com>
---
Documentation/git-verify-tag.txt | 6 +++-
builtin/verify-tag.c | 53 +++++++++++++++++++++++++++++++++++---
2 files changed, 53 insertions(+), 6 deletions(-)
diff --git a/Documentation/git-verify-tag.txt b/Documentation/git-verify-tag.txt
index 5ff76e8..ce47f95 100644
--- a/Documentation/git-verify-tag.txt
+++ b/Documentation/git-verify-tag.txt
@@ -8,7 +8,7 @@ git-verify-tag - Check the GPG signature of tags
SYNOPSIS
--------
[verse]
-'git verify-tag' <tag>...
+'git verify-tag' <object>...
DESCRIPTION
-----------
@@ -20,8 +20,10 @@ OPTIONS
--verbose::
Print the contents of the tag object before validating it.
-<tag>...::
+<object>...::
SHA1 identifiers of git tag objects.
+ For non-tag objects, scan and verify all tags to the given
+ object.
GIT
---
diff --git a/builtin/verify-tag.c b/builtin/verify-tag.c
index 28c2174..df9e93c 100644
--- a/builtin/verify-tag.c
+++ b/builtin/verify-tag.c
@@ -7,6 +7,7 @@
*/
#include "cache.h"
#include "builtin.h"
+#include "refs.h"
#include "tag.h"
#include "run-command.h"
#include <signal.h>
@@ -14,7 +15,7 @@
#include "gpg-interface.h"
static const char * const verify_tag_usage[] = {
- "git verify-tag [-v|--verbose] <tag>...",
+ "git verify-tag [-v|--verbose] <object>...",
NULL
};
@@ -32,6 +33,46 @@ static int run_gpg_verify(const char *buf, unsigned long size, int verbose)
return verify_signed_buffer(buf, len, buf + len, size - len, NULL);
}
+struct obj_filter {
+ const unsigned char *sha1;
+ int verbose;
+ struct strbuf sb;
+};
+
+static int verify_tag_of_obj(const char *refname, const unsigned char *sha1,
+ int flag, void *cb_data)
+{
+ struct obj_filter *obj = cb_data;
+ enum object_type type;
+ unsigned long size;
+ int len, ret;
+ char *buf = NULL;
+ unsigned char tagged_sha1[20];
+
+ if ((type = sha1_object_info(sha1, NULL), type == OBJ_TAG) \
+ && (buf = read_sha1_file(sha1, &type, &size), buf) \
+ && !memcmp("object ", buf, 7) \
+ && !get_sha1_hex(buf + 7, tagged_sha1) \
+ && buf[47] == '\n' \
+ && !memcmp(obj->sha1, tagged_sha1, 20) \
+ && (len = parse_signature(buf, size), len != size)) {
+ strbuf_reset(&obj->sb);
+ ret = verify_signed_buffer(buf, len, buf + len, size - len,
+ &obj->sb);
+ if (obj->verbose) {
+ write_in_full(1, buf, len);
+ write_in_full(1, obj->sb.buf, obj->sb.len);
+ } else if (ret) {
+ printf("tag %s: FAILED\n", refname);
+ write_in_full(1, obj->sb.buf, obj->sb.len);
+ } else
+ printf("tag %s: OK\n", refname);
+ }
+ if (buf)
+ free(buf);
+ return 0;
+}
+
static int verify_tag(const char *name, int verbose)
{
enum object_type type;
@@ -44,9 +85,13 @@ static int verify_tag(const char *name, int verbose)
return error("tag '%s' not found.", name);
type = sha1_object_info(sha1, NULL);
- if (type != OBJ_TAG)
- return error("%s: cannot verify a non-tag object of type %s.",
- name, typename(type));
+ if (type != OBJ_TAG) {
+ struct obj_filter obj = { sha1, verbose };
+ strbuf_init(&obj.sb, 4096);
+ for_each_tag_ref(verify_tag_of_obj, (void *) &obj);
+ strbuf_release(&obj.sb);
+ return 0;
+ }
buf = read_sha1_file(sha1, &type, &size);
if (!buf)
--
1.7.9.dirty
^ permalink raw reply related
* Re: Git performance results on a large repository
From: Zeki Mokhtarzada @ 2012-02-04 0:01 UTC (permalink / raw)
To: git
In-Reply-To: <CB5074CF.3AD7A%joshua.redstone@fb.com>
> The test repo has 4 million commits, linear history and about 1.3 million
> files. The size of the .git directory is about 15GB, and has been
> repacked with 'git repack -a -d -f --max-pack-size=10g --depth=100
> --window=250'. This repack took about 2 days on a beefy machine (I.e.,
> lots of ram and flash). The size of the index file is 191 MB. I can share
Are you willing to give up all or part of your history in your working
repository? I've heard of larger projects starting from scratch (i.e. copy all
of your files into a brand new repo.) You can keep your old repo around for
archival purposes. Also, how much of your repo is code, versus static assets.
You could move all of your static assets (images, css, maybe some js?) into
another repo, and then merge the two repo's together at build time if you
absolutely need them deployed together.
Here are a couple strategies for doing a partial truncate:
http://stackoverflow.com/questions/4515580/how-do-i-remove-the-old-history-from-a-git-repository
http://bogdan.org.ua/2011/03/28/how-to-truncate-git-history-sample-script-included.html
-Zeki
^ permalink raw reply
* Re: [1.7.9] usage regression when merging annotated tag objects
From: Junio C Hamano @ 2012-02-03 23:54 UTC (permalink / raw)
To: Bart Trojanowski; +Cc: git
In-Reply-To: <CADeLxZTsq1M5oEb1u5Oqfxq3dYXL6E_uN9bXaTqaOZiA0fgdJQ@mail.gmail.com>
Some examples you gave were irrelevant, so I'd give an updated version.
Here are the facts of the day, without judging if the behaviour is good or
bad.
1) If you are at Linux v3.2.2 and do not have any development on top,
$ git merge v3.2.3
historically would have fast-forwarded. Git v1.7.9 would now create a
merge commit, authored by you (who is unlikely to be Linus).
2) You do not want such a merge, so try to work it around by this:
$ git merge --ff-only v3.2.3
fatal: Not possible to fast-forward, aborting.
which is refused because merging a tag object requires a new merge
commit.
Here are my assessments.
1. I do not think the first one is a real issue. 99% of the people who are
merely following along the upstream will never say "git merge v3.2.3".
They will instead say "git pull" and this _will_ fast-forward. No
merging of tag objects involved. Also when they want to check out that
specific version, they won't be using "git merge". It will be "git
checkout v3.2.3". So I do not think this is an issue for the case
where it used to result in a fast-forward.
1.5 A variant of the first one is when you have forked and are trying to
synchronize with the latest stable. In that case, you _do_ want a merge
to happen. It is possible that you may not want to get the "mergetag"
header in the resulting merge commit, and "git merge v3.2.3^0" is a new
way to do so.
Strictly speaking, this _is_ a usage regression caused by the new
meaning "git merge" gained in v1.7.9. Recording the tag in the a merge
commit, however, is the whole point of "git merge v3.2.3" that is given
a tag; this behaviour is not going to to change.
2. This is somewhat problematic. "git merge --ff-only v2.6.29" to people
who merely follow Linus has always been possible, and I would expect it
to be the case.
But again, the reason they said --ff-only in the first place is because
they feared that they might have some unexpected commits in their
history, and asked "git merge" to error out if the command has to
create a merge to let them know. So at that point, they could be
trained to run "git merge --ff-only v3.2.3^0" instead, *given enough
clue*.
The problem is that we are not giving enough clue. We just say "Not
possible to fast-forward" without explaining why.
We could solve this in one of two ways. We could tell them to merge
v3.2.3^0 instead. Or we could just go ahead and do that for them
automatically ourselves. I am inclined to say that we should unwrap
the tag given from the command line when --ff-only was given, i.e. we
do the latter.
^ permalink raw reply
* Re: Git performance results on a large repository
From: Chris Lee @ 2012-02-03 23:35 UTC (permalink / raw)
To: Joshua Redstone; +Cc: git@vger.kernel.org
In-Reply-To: <CB5074CF.3AD7A%joshua.redstone@fb.com>
On Fri, Feb 3, 2012 at 6:20 AM, Joshua Redstone <joshua.redstone@fb.com> wrote:
> [snip]
>
> The git performance we observed here is too slow for our needs. So the
> question becomes, if we want to keep using git going forward, what's the
> best way to improve performance. It seems clear we'll probably need some
> specialized servers (e.g., to perform git-blame quickly) and maybe
> specialized file system integration to detect what files have changed in a
> working tree.
Have you considered upgrading all of engineering to SSDs? 200+GB SSDs
are under $400USD nowadays.
-clee
^ permalink raw reply
* Re: [PATCH v4 13/13] tag: add --column
From: Junio C Hamano @ 2012-02-03 23:30 UTC (permalink / raw)
To: Nguyễn Thái Ngọc Duy; +Cc: git
In-Reply-To: <1328276078-27955-14-git-send-email-pclouds@gmail.com>
Nguyễn Thái Ngọc Duy <pclouds@gmail.com> writes:
> @@ -421,6 +428,8 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
> };
>
> git_config(git_tag_config, NULL);
> + if (!colopts)
> + colopts = git_colopts;
>
> memset(&opt, 0, sizeof(opt));
>
> @@ -441,9 +450,19 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
>
> if (list + delete + verify > 1)
> usage_with_options(git_tag_usage, options);
> - if (list)
> - return list_tags(argv, lines == -1 ? 0 : lines,
> - with_commit);
> + if (list) {
> + int ret;
> + if (lines == -1) {
> + struct column_options copts;
> + memset(&copts, 0, sizeof(copts));
> + copts.padding = 2;
> + run_column_filter(colopts, &copts);
> + }
> + ret = list_tags(argv, lines == -1 ? 0 : lines, with_commit);
> + if (lines == -1)
> + stop_column_filter();
> + return ret;
> + }
> if (lines != -1)
> die(_("-n option is only allowed with -l."));
> if (with_commit)
The patch is surprisingly small, which is a good sign. The same comment
to the silent suppression "branch -v --column" applies here to "tag -n".
^ permalink raw reply
* Re: [PATCH] t0300-credentials: Word around a solaris /bin/sh bug
From: Jeff King @ 2012-02-03 23:27 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Ben Walton, git
In-Reply-To: <7vipjnmt8a.fsf@alter.siamese.dyndns.org>
On Fri, Feb 03, 2012 at 02:45:25PM -0800, Junio C Hamano wrote:
> Let's not over-engineer this and stick to the simple-stupid-sufficient.
Fair enough.
> Something like this?
> [...]
> +# Prepare a script to be used in the test
> +write_script () {
> + {
> + echo "#!${2-"$SHELL_PATH"}"
> + cat
> + } >"$1" &&
> + chmod +x "$1"
> +}
Looks good to me (it probably doesn't matter, but you may want to
connect the echo and cat via &&).
-Peff
^ permalink raw reply
* Re: [PATCH v4 11/13] status: add --column
From: Junio C Hamano @ 2012-02-03 23:19 UTC (permalink / raw)
To: Nguyễn Thái Ngọc Duy; +Cc: git
In-Reply-To: <1328276078-27955-12-git-send-email-pclouds@gmail.com>
Nguyễn Thái Ngọc Duy <pclouds@gmail.com> writes:
> @@ -1251,7 +1260,7 @@ int cmd_status(int argc, const char **argv, const char *prefix)
> case STATUS_FORMAT_LONG:
> s.verbose = verbose;
> s.ignore_submodule_arg = ignore_submodule_arg;
> - wt_status_print(&s);
> + wt_status_print(&s, colopts);
> break;
> }
Do you really need to pass colopts around as a separate parameter all the
way through the callchain?
Why isn't it a new member of wt_status that sits next to existing
use_color, verbose, etc. that define _how_ the status is shown?
^ permalink raw reply
* Re: [PATCH v4 02/13] column: add API to print items in columns
From: Junio C Hamano @ 2012-02-03 23:16 UTC (permalink / raw)
To: Nguyễn Thái Ngọc Duy; +Cc: git
In-Reply-To: <1328276078-27955-3-git-send-email-pclouds@gmail.com>
Nguyễn Thái Ngọc Duy <pclouds@gmail.com> writes:
> +#define MODE(mode) ((mode) & COL_MODE)
> + ...
> +void print_columns(const struct string_list *list, int mode,
> + struct column_options *opts)
> +{
> + const char *indent = "", *nl = "\n";
> + int padding = 1, width = term_columns();
> +
> + if (!list->nr)
> + return;
> + if (opts) {
> + if (opts->indent)
> + indent = opts->indent;
> + if (opts->nl)
> + nl = opts->nl;
> + if (opts->width)
> + width = opts->width;
> + padding = opts->padding;
> + }
> + if (width <= 1 || !(mode & COL_ENABLED)) {
Unless there is a compelling reason not to, make a flag word used as
collection of bitfields an unsigned, i.e. not "int mode".
^ permalink raw reply
* Re: [PATCH v4 10/13] branch: add --column
From: Junio C Hamano @ 2012-02-03 23:11 UTC (permalink / raw)
To: Nguyễn Thái Ngọc Duy; +Cc: git
In-Reply-To: <1328276078-27955-11-git-send-email-pclouds@gmail.com>
Nguyễn Thái Ngọc Duy <pclouds@gmail.com> writes:
> @@ -474,7 +482,7 @@ static void print_ref_item(struct ref_item *item, int maxwidth, int verbose,
> else if (verbose)
> /* " f7c0c00 [ahead 58, behind 197] vcs-svn: drop obj_pool.h" */
> add_verbose_info(&out, item, verbose, abbrev);
> - printf("%s\n", out.buf);
> + print_cell(&output, colopts, out.buf);
> strbuf_release(&name);
> strbuf_release(&out);
> }
Hmm, disabling column output when verbose is in effect without telling the
user what we are doing needs to be fixed, but because of that, at least
this codepath won't try to stuff potentially long strings in a columnar
form.
I am not sure about the utility of columnar output for "git branch" in the
short form. You no longer can just scan the leftmost column to scan for
'*' to see the current branch.
^ permalink raw reply
* [1.7.9] usage regression when merging annotated tag objects
From: Bart Trojanowski @ 2012-02-03 23:08 UTC (permalink / raw)
To: git
I recently started using git 1.7.9. Earlier today GregKH released a stable
kernel update and I tried my tried and true procedure using 'git merge
--ff-only v3.2.3'. I was a bit surprised with the results.
There are two tags I am toying with...
69bade0 is v3.2.3
3499d64 is v3.2.2
And here is where we start...
$ git describe
v3.2.2
$ git merge-base v3.2.2 v3.2.3 | xargs git describe
v3.2.2
(it is thus eligible for a fast-forward)
Finally, the strangeness...
$ git merge --ff-only v3.2.3
fatal: Not possible to fast-forward, aborting.
$ git merge --ff-only v3.2.3~
Updating 3499d64..7b171c5
Fast-forward
...
$ git merge --ff-only v3.2.3
fatal: Not possible to fast-forward, aborting.
After talking to Junio, he pointed out that "merging tag objects gained new
meanings in 1.7.9".
I am not sure if my confusion will be shared by others and if --ff-only
needs a clarification. Perhaps --ff-only should just continue to work as
it did before, and fast-forward from tag to tag.
Cheers,
-Bart
^ permalink raw reply
* Re: Git performance results on a large repository
From: Matt Graham @ 2012-02-03 23:05 UTC (permalink / raw)
To: Joshua Redstone
Cc: Ævar Arnfjörð Bjarmason, git@vger.kernel.org
In-Reply-To: <CB5179E9.3B751%joshua.redstone@fb.com>
Hi Josh,
On Fri, Feb 3, 2012 at 17:00, Joshua Redstone <joshua.redstone@fb.com> wrote:
> Thanks for the comments. I've included a bunch more info on the test repo
> below. It is based on a growth model of two of our current repositories
> (I.e., it's not a perforce import). We already have some of the easily
> separable projects in separate repositories, like HPHP. If we could
> split our largest repos into multiple ones, that would help the scaling
> issue. However, the code in those repos is rather interdependent and we
> believe it'd hurt more than help to split it up, at least for the
> medium-term future. We derive a fair amount of benefit from the code
> sharing and keeping things together in a single repo, so it's not clear
> when it'd make sense to get more aggressive splitting things up.
>
> Some more information on the test repository: The working directory is
> 9.5 GB, the median file size is 2 KB. The average depth of a directory
> (counting the number of '/'s) is 3.6 levels and the average depth of a
> file is 4.6. More detailed histograms of the repository composition is
> below:
Do you have a histogram of the types of files in the repo?
And as suggested earlier, is svn working for you now because it allows
sparse checkout? I imagine the stats for svn on the full repo would
be comparable or worse to what you measured with git?
^ permalink raw reply
* Re: [PATCH v4 09/13] help: reuse print_columns() for help -a
From: Junio C Hamano @ 2012-02-03 23:05 UTC (permalink / raw)
To: Nguyễn Thái Ngọc Duy; +Cc: git
In-Reply-To: <1328276078-27955-10-git-send-email-pclouds@gmail.com>
Nguyễn Thái Ngọc Duy <pclouds@gmail.com> writes:
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
> help.c | 47 +++++++++++++----------------------------------
> 1 files changed, 13 insertions(+), 34 deletions(-)
Nice.
^ permalink raw reply
* Re: [PATCH v4 08/13] column: add column.ui for default column output settings
From: Junio C Hamano @ 2012-02-03 23:04 UTC (permalink / raw)
To: Nguyễn Thái Ngọc Duy; +Cc: git
In-Reply-To: <1328276078-27955-9-git-send-email-pclouds@gmail.com>
Nguyễn Thái Ngọc Duy <pclouds@gmail.com> writes:
> diff --git a/column.h b/column.h
> index 1912cb0..afdafc4 100644
> --- a/column.h
> +++ b/column.h
> @@ -17,6 +17,8 @@ struct column_options {
> const char *nl;
> };
>
> +extern int git_colopts;
For a global state variable, I'd prefer to see it spelled out, e.g.
git_column_opts or even git_column_options. It's not like you would be
referring to this variable from everywhere---you would use it only from
fallback codepaths after parse_options() returns, or something, no?
^ permalink raw reply
* Re: [PATCH v4 03/13] parseopt: make OPT_INTEGER support hexadecimal as well
From: Junio C Hamano @ 2012-02-03 22:59 UTC (permalink / raw)
To: Nguyễn Thái Ngọc Duy; +Cc: git
In-Reply-To: <1328276078-27955-4-git-send-email-pclouds@gmail.com>
Nguyễn Thái Ngọc Duy <pclouds@gmail.com> writes:
> - *(int *)opt->value = strtol(arg, (char **)&s, 10);
> + if (!prefixcmp(arg, "0x") || !prefixcmp(arg, "0X"))
> + *(int *)opt->value = strtol(arg + 2, (char **)&s, 16);
> + else
> + *(int *)opt->value = strtol(arg, (char **)&s, 10);
Can't you just do "strtol(arg, (char **)&s, 0)" instead?
^ permalink raw reply
* Re: Git performance results on a large repository
From: Sam Vilain @ 2012-02-03 22:57 UTC (permalink / raw)
To: Joshua Redstone
Cc: Ævar Arnfjörð Bjarmason, git@vger.kernel.org
In-Reply-To: <4F2C6276.1070100@vilain.net>
On 2/3/12 2:40 PM, Sam Vilain wrote:
> As the git object storage model is write–only and content–addressed,
> it should git this kind of scaling well.
^^^
Could have sworn I typed 'suit' there. My fingers have auto–correct ;-)
Sam
^ permalink raw reply
* Re: [PATCH v4 02/13] column: add API to print items in columns
From: Junio C Hamano @ 2012-02-03 22:55 UTC (permalink / raw)
To: Nguyễn Thái Ngọc Duy; +Cc: git
In-Reply-To: <1328276078-27955-3-git-send-email-pclouds@gmail.com>
Nguyễn Thái Ngọc Duy <pclouds@gmail.com> writes:
> +struct string_list_item *add_cell_to_list(struct string_list *list,
> + int mode,
> + const char *string)
> +{
> + if (mode & COL_ENABLED)
> + return string_list_append(list, string);
> + return NULL;
> +}
> +
> +void print_cell(struct string_list *list, int mode, const char *string)
> +{
> + if (!add_cell_to_list(list, mode, string))
> + printf("%s\n", string);
> +}
I find these two functions showing an extremely bad taste in the code
design. "If we for some reason fail to add the string to the list, we
fallback to print it straight away" is the natural reading of this code,
but that is absolutly not the impression the author of this piece of
crap^Wcode wanted to give to its readers.
Am I being too ascestically perfectionist?
^ permalink raw reply
* Re: Git performance results on a large repository
From: Sam Vilain @ 2012-02-03 22:40 UTC (permalink / raw)
To: Joshua Redstone
Cc: Ævar Arnfjörð Bjarmason, git@vger.kernel.org
In-Reply-To: <CB5179E9.3B751%joshua.redstone@fb.com>
Joshua,
You have an interesting use case.
If I were you I'd consider investigating the git fast-import protocol.
It has become bi–directional, and is essentially socket access to a git
repository with read and transactional update capability. With a few
more commands implemented, it may even be capable of providing all
functionality required for command–line git use.
It is already possible that the ".git" directory can be a file: this
case is used for submodules in git 1.7.8 and higher. For this use case,
there would be an extra field to the ".git" file which is created. It
would indicate the hostname (and port) to connect its internal
'fast-import' stream to. 'clone' would consist of creating this file,
and then getting the server to stream the objects from its pack to the
client.
With the hard–working part of git on the other end of a network service,
you could back it by a re–implementation of git which is written to be
distributed in Hadoop. There are at least two similar implementations
of git that are like this: one for cassandra which was written by github
as a research project, and Google's implementation on top of their
BigTable/GFS/whatever. As the git object storage model is write–only
and content–addressed, it should git this kind of scaling well.
There have also been designs at various times for sparse check–outs; ie
check–outs where you don't check out the root of the repository but a
sub–tree.
With both of these features, clients could easily check out a small part
of the repository very quickly. This is probably the only case which
SVN still does better than git at, which is a particular blocker for use
cases like repositories with large binaries in them and for projects
such as the one you have (another one with a similar problem was KDE,
where their projects moved around the repository a lot, and refactoring
touched many projects simultaneously at times).
It's a large undertaking, alright.
Sam,
just another git community propeller–head.
On 2/3/12 9:00 AM, Joshua Redstone wrote:
> Hi Ævar,
>
>
> Thanks for the comments. I've included a bunch more info on the test repo
> below. It is based on a growth model of two of our current repositories
> (I.e., it's not a perforce import). We already have some of the easily
> separable projects in separate repositories, like HPHP. If we could
> split our largest repos into multiple ones, that would help the scaling
> issue. However, the code in those repos is rather interdependent and we
> believe it'd hurt more than help to split it up, at least for the
> medium-term future. We derive a fair amount of benefit from the code
> sharing and keeping things together in a single repo, so it's not clear
> when it'd make sense to get more aggressive splitting things up.
>
> Some more information on the test repository: The working directory is
> 9.5 GB, the median file size is 2 KB. The average depth of a directory
> (counting the number of '/'s) is 3.6 levels and the average depth of a
> file is 4.6. More detailed histograms of the repository composition is
> below:
>
> ------------------------
>
> Histogram of depth of every directory in the repo (dirs=`find . -type d` ;
> (for dir in $dirs; do t=${dir//[^\/]/}; echo ${#t} ; done) |
> ~/tmp/histo.py)
> * The .git directory itself has only 161 files, so although included,
> doesn't affect the numbers significantly)
>
> [0.0 - 1.3): 271
> [1.3 - 2.6): 9966
> [2.6 - 3.9): 56595
> [3.9 - 5.2): 230239
> [5.2 - 6.5): 67394
> [6.5 - 7.8): 22868
> [7.8 - 9.1): 6568
> [9.1 - 10.4): 420
> [10.4 - 11.7): 45
> [11.7 - 13.0]: 21
> n=394387 mean=4.671830, median=5.000000, stddev=1.272658
>
>
> Histogram of depth of every file in the repo (files=`git ls-files` ; (for
> file in $files; do t=${file//[^\/]/}; echo ${#t} ; done) | ~/tmp/histo.py)
> * 'git ls-files' does not prefix entries with ./, like the 'find' command
> above, does, hence why the average appears to be the same as the directory
> stats
>
> [0.0 - 1.3]: 1274
> [1.3 - 2.6]: 35353
> [2.6 - 3.9]: 196747
> [3.9 - 5.2]: 786647
> [5.2 - 6.5]: 225913
> [6.5 - 7.8]: 77667
> [7.8 - 9.1]: 22130
> [9.1 - 10.4]: 1599
> [10.4 - 11.7]: 164
> [11.7 - 13.0]: 118
> n=1347612 mean=4.655750, median=5.000000, stddev=1.278399
>
>
> Histogram of file sizes (for first 50k files - this command takes a
> while): files=`git ls-files` ; (for file in $files; do stat -c%s $file ;
> done) | ~/tmp/histo.py
>
> [ 0.0 - 4.7): 0
> [ 4.7 - 22.5): 2
> [ 22.5 - 106.8): 0
> [ 106.8 - 506.8): 0
> [ 506.8 - 2404.7): 31142
> [ 2404.7 - 11409.9): 17837
> [ 11409.9 - 54137.1): 942
> [ 54137.1 - 256866.9): 53
> [ 256866.9 - 1218769.7): 18
> [ 1218769.7 - 5782760.0]: 5
> n=49999 mean=3590.953239, median=1772.000000, stddev=42835.330259
>
> Cheers,
> Josh
>
>
>
>
>
>
> On 2/3/12 9:56 AM, "Ævar Arnfjörð Bjarmason"<avarab@gmail.com> wrote:
>
>> On Fri, Feb 3, 2012 at 15:20, Joshua Redstone<joshua.redstone@fb.com>
>> wrote:
>>
>>> We (Facebook) have been investigating source control systems to meet our
>>> growing needs. We already use git fairly widely, but have noticed it
>>> getting slower as we grow, and we want to make sure we have a good story
>>> going forward. We're debating how to proceed and would like to solicit
>>> people's thoughts.
>>
>> Where I work we also have a relatively large Git repository. Around
>> 30k files, a couple of hundred thousand commits, clone size around
>> half a GB.
>>
>> You haven't supplied background info on this but it really seems to me
>> like your testcase is converting something like a humongous Perforce
>> repository directly to Git.
>>
>> While you /can/ do this it's not a good idea, you should split up
>> repositories at the boundaries code or data doesn't directly cross
>> over, e.g. there's no reason why you need HipHop PHP in the same
>> repository as Cassandra or the Facebook chat system, is there?
>>
>> While Git could better with large repositories (in particular applying
>> commits in interactive rebase seems to be to slow down on bigger
>> repositories) there's only so much you can do about stat-ing 1.3
>> million files.
>>
>> A structure that would make more sense would be to split up that giant
>> repository into a lot of other repositories, most of them probably
>> have no direct dependencies on other components, but even those that
>> do can sometimes just use some other repository as a submodule.
>>
>> Even if you have the requirement that you'd like to roll out
>> *everything* at a certain point in time you can still solve that with
>> a super-repository that has all the other ones as submodules, and
>> creates a tag for every rollout or something like that.
>
> N�����r��y���b�X��ǧv�^�){.n�+����ا�\x17��ܨ}���Ơz�&j:+v���\a����zZ+��+zf���h���~����i���z�\x1e�w���?����&�)ߢ^[fl===
^ permalink raw reply
* Re: [PATCH] t0300-credentials: Word around a solaris /bin/sh bug
From: Junio C Hamano @ 2012-02-03 22:45 UTC (permalink / raw)
To: Jeff King; +Cc: Ben Walton, git
In-Reply-To: <20120203215507.GB3472@sigill.intra.peff.net>
Jeff King <peff@peff.net> writes:
>> 2) echo "#!$2" ;;
>> *) BUG ;;
>> esac >"$1" &&
>> cat >>"$1" &&
>> chmod +x "$1"
>> }
>>
>
> Nice. I was going to suggest a wrapper like "write_sh_script" so you
> didn't have to spell out $SHELL_PATH, but I think the auto-detection
> makes sense (and falling back to shell makes even more sense, as that
> covers 99% of the cases anyway).
Let's not over-engineer this and stick to the simple-stupid-sufficient.
Something like this?
t/test-lib.sh | 9 +++++++++
1 files changed, 9 insertions(+), 0 deletions(-)
diff --git a/t/test-lib.sh b/t/test-lib.sh
index bdd9513..1b9c461 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -379,6 +379,15 @@ test_config () {
git config "$@"
}
+# Prepare a script to be used in the test
+write_script () {
+ {
+ echo "#!${2-"$SHELL_PATH"}"
+ cat
+ } >"$1" &&
+ chmod +x "$1"
+}
+
# Use test_set_prereq to tell that a particular prerequisite is available.
# The prerequisite can later be checked for in two ways:
#
^ permalink raw reply related
* [PATCH 2/2] prompt: fall back to terminal if askpass fails
From: Jeff King @ 2012-02-03 22:16 UTC (permalink / raw)
To: Neal Groothuis; +Cc: Feanil Patel, git
In-Reply-To: <20120203213654.GD1890@sigill.intra.peff.net>
The current askpass code simply dies if calling an askpass
helper fails. Worse, in some failure modes it doesn't even
print an error (if start_command fails, then it prints its
own error; if reading fails, we print an error; but if the
command exits non-zero, finish_command fails and we print
nothing!).
Let's be more kind to the user by printing an error message
when askpass doesn't work out, and then falling back to the
terminal (which also may fail, of course, but we die already
there with a nice message).
While we're at it, let's clean up the existing error
messages a bit. Now that our prompts are very long and
contain quotes and colons themselves, our error messages are
hard to read.
So the new failure modes look like:
[before, with a terminal]
$ GIT_ASKPASS=false git push
$ echo $?
128
[before, with no terminal, and we must give up]
$ setsid git push
fatal: could not read 'Password for 'https://peff@github.com': ': No such device or address
[after, with a terminal]
$ GIT_ASKPASS=false git push
error: unable to read askpass response from 'false'
Password for 'https://peff@github.com':
[after, with no terminal, and we must give up]
$ GIT_ASKPASS=false setsid git push
error: unable to read askpass response from 'false'
fatal: could not read Password for 'https://peff@github.com': No such device or address
Signed-off-by: Jeff King <peff@peff.net>
---
Arguably, the terminal-failure error message shouldn't even bother
reporting errno. I can't imagine it failing for any reason other than
ENODEV, and the message would probably be less confusing as:
fatal: could not prompt on terminal for Password for...
prompt.c | 24 +++++++++++++++++-------
1 files changed, 17 insertions(+), 7 deletions(-)
diff --git a/prompt.c b/prompt.c
index 64f817b..d851807 100644
--- a/prompt.c
+++ b/prompt.c
@@ -9,6 +9,7 @@ static char *do_askpass(const char *cmd, const char *prompt)
struct child_process pass;
const char *args[3];
static struct strbuf buffer = STRBUF_INIT;
+ int err = 0;
args[0] = cmd;
args[1] = prompt;
@@ -19,15 +20,21 @@ static char *do_askpass(const char *cmd, const char *prompt)
pass.out = -1;
if (start_command(&pass))
- exit(1);
+ return NULL;
if (strbuf_read(&buffer, pass.out, 20) < 0)
- die("failed to get '%s' from %s\n", prompt, cmd);
+ err = 1;
close(pass.out);
if (finish_command(&pass))
- exit(1);
+ err = 1;
+
+ if (err) {
+ error("unable to read askpass response from '%s'", cmd);
+ strbuf_release(&buffer);
+ return NULL;
+ }
strbuf_setlen(&buffer, strcspn(buffer.buf, "\r\n"));
@@ -36,7 +43,7 @@ static char *do_askpass(const char *cmd, const char *prompt)
char *git_prompt(const char *prompt, int flags)
{
- char *r;
+ char *r = NULL;
if (flags & PROMPT_ASKPASS) {
const char *askpass;
@@ -47,12 +54,15 @@ char *git_prompt(const char *prompt, int flags)
if (!askpass)
askpass = getenv("SSH_ASKPASS");
if (askpass && *askpass)
- return do_askpass(askpass, prompt);
+ r = do_askpass(askpass, prompt);
}
- r = git_terminal_prompt(prompt, flags & PROMPT_ECHO);
if (!r)
- die_errno("could not read '%s'", prompt);
+ r = git_terminal_prompt(prompt, flags & PROMPT_ECHO);
+ if (!r) {
+ /* prompts already contain ": " at the end */
+ die("could not read %s%s", prompt, strerror(errno));
+ }
return r;
}
--
1.7.9.rc1.28.gf4be5
^ permalink raw reply related
* [PATCH 1/2] prompt: clean up strbuf usage
From: Jeff King @ 2012-02-03 22:14 UTC (permalink / raw)
To: Neal Groothuis; +Cc: Feanil Patel, git
In-Reply-To: <20120203213654.GD1890@sigill.intra.peff.net>
The do_askpass function inherited a few bad habits from the
original git_getpass. One, there's no need to strbuf_reset a
buffer which was just initialized. And two, it's a good
habit to use strbuf_detach to claim ownership of a buffer's
string (even though in this case the owning buffer goes out
of scope, so it's effectively the same thing).
Signed-off-by: Jeff King <peff@peff.net>
---
Neither is a big deal, but just some style cleanups while I was in the
area.
prompt.c | 3 +--
1 files changed, 1 insertions(+), 2 deletions(-)
diff --git a/prompt.c b/prompt.c
index 72ab9de..64f817b 100644
--- a/prompt.c
+++ b/prompt.c
@@ -21,7 +21,6 @@ static char *do_askpass(const char *cmd, const char *prompt)
if (start_command(&pass))
exit(1);
- strbuf_reset(&buffer);
if (strbuf_read(&buffer, pass.out, 20) < 0)
die("failed to get '%s' from %s\n", prompt, cmd);
@@ -32,7 +31,7 @@ static char *do_askpass(const char *cmd, const char *prompt)
strbuf_setlen(&buffer, strcspn(buffer.buf, "\r\n"));
- return buffer.buf;
+ return strbuf_detach(&buffer, NULL);
}
char *git_prompt(const char *prompt, int flags)
--
1.7.9.rc1.28.gf4be5
^ permalink raw reply related
* Re: Push from an SSH Terminal
From: Jeff King @ 2012-02-03 22:13 UTC (permalink / raw)
To: Neal Groothuis; +Cc: Feanil Patel, git
In-Reply-To: <20120203213654.GD1890@sigill.intra.peff.net>
On Fri, Feb 03, 2012 at 04:36:54PM -0500, Jeff King wrote:
> > Check to see if the GIT_ASKPASS and/or SSH_ASKPASS environment variables
> > are set, and if the core.askpass config variable is set. If any of these
> > are set, unset them. Git should fall back to a simple password prompt.
>
> Hmm, yeah that is likely the problem. I was thinking git would fall back
> to asking on the terminal, but it does not. We probably should.
We should probably do this:
[1/2]: prompt: clean up strbuf usage
[2/2]: prompt: fall back to terminal if askpass fails
-Peff
^ permalink raw reply
* Re: [PATCH] t0300-credentials: Word around a solaris /bin/sh bug
From: Ben Walton @ 2012-02-03 22:00 UTC (permalink / raw)
To: Jeff King, Junio C Hamano; +Cc: git
In-Reply-To: <20120203215507.GB3472@sigill.intra.peff.net>
Excerpts from Jeff King's message of Fri Feb 03 16:55:07 -0500 2012:
> > write_script () {
> > case "$#" in
> > 1) case "$1" in
> > *.perl | *.pl) echo "#!$PERL_PATH" ;;
> > *) echo "#!$SHELL_PATH" ;;
> > esac
> > 2) echo "#!$2" ;;
> > *) BUG ;;
> > esac >"$1" &&
> > cat >>"$1" &&
> > chmod +x "$1"
> > }
> >
>
> Nice. I was going to suggest a wrapper like "write_sh_script" so you
> didn't have to spell out $SHELL_PATH, but I think the auto-detection
> makes sense (and falling back to shell makes even more sense, as that
> covers 99% of the cases anyway).
This looks like a very nice, general purpose, solution to the problem.
Thanks
-Ben
--
Ben Walton
Systems Programmer - CHASS
University of Toronto
C:416.407.5610 | W:416.978.4302
^ permalink raw reply
* Re: [PATCH] t0300-credentials: Word around a solaris /bin/sh bug
From: Jeff King @ 2012-02-03 21:55 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Ben Walton, git
In-Reply-To: <7vr4ybmvrq.fsf@alter.siamese.dyndns.org>
On Fri, Feb 03, 2012 at 01:50:33PM -0800, Junio C Hamano wrote:
> > write_script foo.sh "$SHELL_PATH" <<-\EOF
> > echo my arguments are "$@"
> > EOF
>
> I first thought that the order of parameters were unusual, but with that
> order, you could even go something fancier like:
>
> write_script () {
> case "$#" in
> 1) case "$1" in
> *.perl | *.pl) echo "#!$PERL_PATH" ;;
> *) echo "#!$SHELL_PATH" ;;
> esac
> 2) echo "#!$2" ;;
> *) BUG ;;
> esac >"$1" &&
> cat >>"$1" &&
> chmod +x "$1"
> }
>
Nice. I was going to suggest a wrapper like "write_sh_script" so you
didn't have to spell out $SHELL_PATH, but I think the auto-detection
makes sense (and falling back to shell makes even more sense, as that
covers 99% of the cases anyway).
-Peff
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox