* Re: [PATCH] Prevent megablobs from gunking up git packs
From: Nicolas Pitre @ 2007-05-22 17:38 UTC (permalink / raw)
To: Dana How; +Cc: Junio C Hamano, Git Mailing List
In-Reply-To: <46528A48.9050903@gmail.com>
On Mon, 21 May 2007, Dana How wrote:
>
> Using fast-import and repack with the max-pack-size patch,
> 3628 commits were imported from Perforce comprising
> 100.35GB (uncompressed) in 38829 blobs, and saved in
> 7 packfiles of 12.5GB total (--window=0 and --depth=0 were
> used due to runtime limits). When using these packfiles,
> several git commands showed very large process sizes,
> and some slowdowns (compared to comparable operations
> on the linux kernel repo) were also apparent.
>
> git stores data in loose blobs or in packfiles. The former
> has essentially now become an exception mechanism, to store
> exceptionally *young* blobs. Why not use this to store
> exceptionally *large* blobs as well? This allows us to
> re-use all the "exception" machinery with only a small change.
>
> Repacking the entire repository with a max-blob-size of 256KB
> resulted in a single 13.1MB packfile, as well as 2853 loose
> objects totaling 15.4GB compressed and 100.08GB uncompressed,
> 11 files per objects/xx directory on average. All was created
> in half the runtime of the previous yet with standard
> --window=10 and --depth=50 parameters. The data in the
> packfile was 270MB uncompressed in 35976 blobs. Operations
> such as "git-log --pretty=oneline" were about 30X faster
> on a cold cache and 2 to 3X faster otherwise. Process sizes
> remained reasonable.
>
> This patch implements the following:
> 1. git pack-objects takes a new --max-blob-size=N flag,
> with the effect that only blobs less than N KB are written
> to the packfiles(s). If a blob was in a pack but violates
> this limit (perhaps the packs were created by fast-import
> or max-blob-size was reduced), then a new loose object
> is written out if needed so the data is not lost.
> 2. git repack inspects repack.maxblobsize . If set, its
> value is passed to git pack-objects on the command line.
> The user should change repack.maxblobsize , NOT specify
> --max-blob-size=N .
> 3. No other caller of git pack-objects supplies this new flag,
> so other callers see no change.
>
> This patch is on top of the earlier max-pack-size patch,
> because I thought I needed some behavior it supplied,
> but could be rebased on master if desired.
I think what this patch is missing is a test after all options have been
parsed to prevent --stdout and --max-blob-size to be used together.
Nicolas
^ permalink raw reply
* Re: [PATCH] Prevent megablobs from gunking up git packs
From: Dana How @ 2007-05-22 16:59 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git, danahow, Junio C Hamano
In-Reply-To: <f2uigr$ufj$1@sea.gmane.org>
On 5/22/07, Jakub Narebski <jnareb@gmail.com> wrote:
> Dana How wrote:
> > There's actually an even more extreme example from my day job.
> > The software team has a project whose files/revisions would be
> > similar to those in the linux kernel (larger commits, I'm sure).
> > But they have *ONE* 500MB file they check in because it takes
> > 2 or 3 days to generate and different people use different versions of it.
> > I'm sure it has 50+ revisions now. If they converted to git and included
> > these blobs in their packfile, that's a 25GB uncompressed increase!
> > *Every* git operation must wade through 10X -- 100X more packfile.
> > Or it could be kept in 50+ loose objects in objects/xx ,
> > requiring a few extra syscalls by each user to get a new version.
> Or keeping those large objects in separate, _kept_ packfile, containing
> only those objects (which can delta well, even if they are large).
Yes, I experimented with various changes to git-repack and
having it create .keep files just before coming up with the maxblobsize
approach. The problem with a 12GB+ repo is not only the large
repack time, but the fact that the repack time keeps growing with
the repo size. So, with split packs, I had repack create .keep
files for all new packs except the last (fragmentary) one. The next
repack would then only repack new stuff plus the single fragmentary
pack, keeping repack time from growing (until you deleted the .keep
files [just the ones with "repack" in them] to start over from scratch).
But this approach is not going to distribute commits and trees all that well.
Last night before signing off Junio proposed some partitioning ideas.
He presented them as ordering things *within* one pack; what I had
tried was making repack operate in 2 passes: the first one would create
pack(s) containing commits+trees+tags, the 2nd would create
pack(s) containing only blobs. Of course the first group would contain
only 1 tiny pack, and the latter 6 or 7 enormous packs. I also combined
this with the previous paragraph, putting .keep files on all but the last
pack in each group. Then the metadata always got repacked,
and the blob data only got its "tail" repacked.
Let's just stipulate that you've convinced me that putting everything
in packs, and not ejecting megablobs, is better or equivalent on
the "central" git repository which will replace (part of) our Perforce
repository. What about the users' repositories?
Each person at my day job has his own workstation. They are all
on a grid and are constantly running jobs in the background.
Each person would have at least one personal repo. What should the
packing strategy be there?
(1) If we must put everything in packs, then we could:
(1a) Repack everything in local repos, incurring large local runtimes.
This extra work then denies the CPU cycles to the grid,
which WILL be noticed and cause much whining.
So the response will be to reduce window and/or turn
on nodelta for some group of objects, worsening packing
and failing to squash the whining. This happens across
20 to 30 workstations. Or we reduce the frequency of
repacking and stagger it across the network. Since daily
pull/fetch/checkout ("sync" in p4 parlance) grabs 400+ new
revisions each day, if we make repacking weekly we have
a policy that results in 400*5/2=1000 extra loose blobs on average,
and there will still be whining. Why not just set maxblobsize
to some size resulting in ~1000 loose blobs, leave window/depth
at default, and enjoy <1hr repacking?
(1b) Repack everything ONLY in the central repo, and have the users' repos
point to it as an alternate. Now we have enormous network traffic.
However, this is better than (1a), and was what I thought I'd be
stuck with. We still do have the possible problem of excessive
packing time on the central repo, but it's easier to solve/hide
in just one place.
(2) We repack everything but leave megablobs loose. Now packfiles
are 13MB, repack time with default window/depth is <1hr, and we
can repack each users' repository from his own cron job. This will
be noticed, but it won't cause too much complaining. Most git
operations by users will be against their local repos, but the
server's db will still be an alternate to fetch at least megablobs.
This is not a problem compared to Perforce, which stores *NO*
repository state locally at all.
I really think megablob ejection from packs makes a lot of sense for local
repos on a network of workstations. It lets me keep almost all repo
state locally very cheaply. It is just another consequence of the tendency
that an adequate solution that operates principally on only 13MB of data
doesn't have to work as hard or as carefully as something
operating on the full 12GB -- three orders of magnitude larger.
If there's interest, I could submit my other alterations to git-repack.
They still have bugs which would take a while to work out since
each run operates on 12GB of data. With quicker runtimes,
maxblobsize was much quicker to debug even though I made
more stupid mistakes at first ;-)
Thanks,
--
Dana L. How danahow@gmail.com +1 650 804 5991 cell
^ permalink raw reply
* Re: [PATCH 2/2] Teach "delta" attribute to pack-objects.
From: Nicolas Pitre @ 2007-05-22 16:04 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <11795608922961-git-send-email-junkio@cox.net>
On Sat, 19 May 2007, Junio C Hamano wrote:
> This teaches pack-objects to use .gitattributes mechanism so
> that the user can specify certain blobs are not worth spending
> CPU cycles to attempt deltification.
[...]
> @@ -1349,6 +1376,10 @@ static void find_deltas(struct object_entry **list, int window, int depth)
>
> if (entry->size < 50)
> continue;
> +
> + if (entry->no_try_delta)
> + continue;
> +
> free_delta_index(n->index);
> n->index = NULL;
> free(n->data);
> @@ -1376,6 +1407,8 @@ static void find_deltas(struct object_entry **list, int window, int depth)
> m = array + other_idx;
> if (!m->entry)
> break;
> + if (m->entry->no_try_delta)
> + continue;
> if (try_delta(n, m, max_depth) < 0)
> break;
> }
This last hunk is unnecessary. Because of the other hunk above, the
no_try_delta objects will never get into the m array.
Well done otherwise. This attribute thing is really nice.
Nicolas
^ permalink raw reply
* Re: [PATCH 2/3] Use stringbuf to fix buffer overflows due to broken use of snprintf()
From: Petr Baudis @ 2007-05-22 13:43 UTC (permalink / raw)
To: Timo Sirainen; +Cc: git
In-Reply-To: <1179627879.32181.1286.camel@hurina>
On Sun, May 20, 2007 at 04:24:39AM CEST, Timo Sirainen wrote:
> @@ -1823,14 +1824,14 @@ static void diff_fill_sha1_info(struct diff_filespec *one)
> static void run_diff(struct diff_filepair *p, struct diff_options *o)
> {
> const char *pgm = external_diff();
> - char msg[PATH_MAX*2+300], *xfrm_msg;
> + stringbuf(msg, PATH_MAX*2+300);
I don't find this style of declaring a variable too clear; I think it
might be worthwhile to make this stand out more and uppercase the
stringbuf() macro.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
-- Samuel Beckett
^ permalink raw reply
* Re: [PATCH 1/3] Added generic string handling code.
From: Petr Baudis @ 2007-05-22 13:40 UTC (permalink / raw)
To: Timo Sirainen; +Cc: git
In-Reply-To: <1179627869.32181.1284.camel@hurina>
On Sun, May 20, 2007 at 04:24:29AM CEST, Timo Sirainen wrote:
> diff --git a/str.c b/str.c
> new file mode 100644
> index 0000000..d46e7f4
> --- /dev/null
> +++ b/str.c
> @@ -0,0 +1,40 @@
> +#include "str.h"
> +
> +void _str_append(struct string *str, const char *cstr)
_ is reserved namespace.
> +{
> + unsigned int avail = str->size - str->len;
> + unsigned int len = strlen(cstr);
> +
> + if (len >= avail) {
> + len = avail - 1;
> + str->overflowed = 1;
> + }
> + memcpy(str->buf + str->len, cstr, len);
> + str->len += len;
> + str->buf[str->len] = '\0';
You can copy len + 1 and avoid this assignment.
> +}
> +
> +void _str_printfa(struct string *str, const char *fmt, ...)
printfA?
> +{
> + unsigned int avail = str->size - str->len;
> + va_list va;
> + int ret;
> +
> + va_start(va, fmt);
> + ret = vsnprintf(str->buf + str->len, avail, fmt, va);
> + if (ret < avail)
> + str->len += ret;
> + else {
> + str->len += avail - 1;
> + str->overflowed = 1;
> + }
> + va_end(va);
> +}
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
-- Samuel Beckett
^ permalink raw reply
* Re: StGIT unhelpfulness
From: Karl Hasselström @ 2007-05-22 14:18 UTC (permalink / raw)
To: Petr Baudis; +Cc: git
In-Reply-To: <20070522141545.GA15546@diana.vm.bytemark.co.uk>
On 2007-05-22 16:15:45 +0200, Karl Hasselström wrote:
> I agree.
... that the check is too strict.
--
Karl Hasselström, kha@treskal.com
www.treskal.com/kalle
^ permalink raw reply
* Re: StGIT unhelpfulness
From: Karl Hasselström @ 2007-05-22 14:15 UTC (permalink / raw)
To: Petr Baudis; +Cc: git
In-Reply-To: <20070522111603.GH4489@pasky.or.cz>
On 2007-05-22 13:16:03 +0200, Petr Baudis wrote:
> stg pull: Rebasing would possibly lose data
I agree. I've also hit this problem, but with "stg rebase".
I just submitted a bug for this:
https://gna.org/bugs/index.php?9181
--
Karl Hasselström, kha@treskal.com
www.treskal.com/kalle
^ permalink raw reply
* Re: [RFC] git-float
From: Michael S. Tsirkin @ 2007-05-22 12:58 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Michael S. Tsirkin, git
In-Reply-To: <7v3b1v7z19.fsf@assigned-by-dhcp.cox.net>
> Quoting Junio C Hamano <junkio@cox.net>:
> Subject: Re: [RFC] git-float
>
> "Michael S. Tsirkin" <mst@dev.mellanox.co.il> writes:
>
> > Here's a simple script I use to float a commit up the history -
> > similiar to what stg float does if I understand it correctly.
> >
> > Is this a good way to implement it?
>
> > git-rebase --onto $ref~1 $ref && git-cherry-pick $ref
>
> Because git-rebase or git-cherry-pick can be interrupted with a
> conflict, this is not a good _implementation_. The whole script
> needs to have the sequencing and continue logic similar to the
> one git-rebase has.
>
> > Would it make sense to have something like this in git tree?
>
> Incidentally, this is closely related to something that people
> have wanted to have for a long time, which is to cherry-pick
> series of commits.
>
> One step of rebase and cherry-pick can be thought of as a
> "rotate a commit" operation. When you cherry-pick a commit C on
> top of where you are, the resulting tree is computed by applying
> the commit C's effect to the current tree via 3-way merge.
>
> git-merge-recursive C^ HEAD C
>
> git-rebase without -m does the equivalent of the above "rotate a
> commit" operation using patch + apply (and fall back to merge if
> the patch does not cleanly apply) for performance reasons, but
> the principle is the same. And the commit message for the
> result is taken from C itself.
>
> git-rebase can be decomposed into three stages:
>
> (1) find the sequence of commits to reapply;
>
> (2) find the commit to start rebuilding onto and reset to it;
>
> (3) one by one, rotate the commits you found in (1), with
> the sequencing support (--abort, --skip and --continue).
>
> There is no reason, other than the fact that there is no other
> commit rotator in git suite that needs sequencing, that these
> three needs to be in a single program git-rebase.
>
> The only difference with the above outline and your float is
> that after you finish step (1), you record "this commit also
> needs to be replayed at the end" information to the sequence.
>
> The implementation of cherry-pick that takes commit range is
> also obvious; instead of the computation git-rebase does for
> step (1) above, we would allow arbitrary series of commits to be
> specified from the command line (most likely using the revision
> list notation A..B) to be replayed with the sequencing
> machinery.
So to summarize: you suggest creating a new low-level command
that gets a list of commits and applies them, with
the sequencing support (--abort, --skip and --continue).
Then rewrite both cherry-pick and rebase on top of these,
implement git float (and maybe git kill which does just
git-rebase --onto $1~1 $1 I posted at some point) on top
of this as well.
--
MST
^ permalink raw reply
* Re: Using git to store /etc, redux
From: David Härdeman @ 2007-05-22 12:16 UTC (permalink / raw)
To: Jan Hudec; +Cc: git
In-Reply-To: <20070521183239.GB5082@efreet.light.src>
On Mon, May 21, 2007 20:32, Jan Hudec wrote:
> Have you looked at IsiSetup (http://www.isisetup.ch/, linked from
> http://git.or.cz/gitwiki/InterfacesFrontendsAndTools) yet? It's a
> front-end
> to git specifically targeted for versioning configuration. From a quick
> glance at it's web I don't see whether it already stores the metadata you
> describe, but in either case it could be interesting for you.
Yes, it was mentioned in the previous thread about tracking /etc in git.
I've looked at it but it seemed to not store any metadata. Also, it seemed
like overkill for just keeping a history of my /etc.
It also used a lot of hairy "grep -v" and similar tricks to run git but to
mangle the output before it's shown to the user. I'd prefer to interact
directly with git which I'm already familiar with...and git already has
everything I need for tracking /etc with the exception of the metadata.
--
David Härdeman
^ permalink raw reply
* [PATCH] gitk: properly resolve ambiguity if argument is both, revision and filename
From: Gerrit Pape @ 2007-05-22 12:42 UTC (permalink / raw)
To: Junio C Hamano, git; +Cc: 425491
If a repository contains a file and a branch with the same name, have
gitk recognize the '--' separator as final command line argument, as
documented in gitk(1); previously 'gitk <name> --' failed. Additionally
have gitk insert '--' automatically for views created or edited through
the View menu.
The bug was noticed and reported by Josh Triplett through
http://bugs.debian.org/425491
Signed-off-by: Gerrit Pape <pape@smarden.org>
---
gitk | 15 +++++++++------
1 files changed, 9 insertions(+), 6 deletions(-)
diff --git a/gitk b/gitk
index a57e84c..7e8d426 100755
--- a/gitk
+++ b/gitk
@@ -19,15 +19,12 @@ proc gitdir {} {
proc start_rev_list {view} {
global startmsecs nextupdate
global commfd leftover tclencoding datemode
- global viewargs viewfiles commitidx
+ global viewargs viewfiles viewargsep commitidx
set startmsecs [clock clicks -milliseconds]
set nextupdate [expr {$startmsecs + 100}]
set commitidx($view) 0
- set args $viewargs($view)
- if {$viewfiles($view) ne {}} {
- set args [concat $args "--" $viewfiles($view)]
- }
+ set args [concat $viewargs($view) $viewargsep($view) $viewfiles($view)]
set order "--topo-order"
if {$datemode} {
set order "--date-order"
@@ -1536,7 +1533,7 @@ proc allviewmenus {n op args} {
proc newviewok {top n} {
global nextviewnum newviewperm newviewname newishighlight
global viewname viewfiles viewperm selectedview curview
- global viewargs newviewargs viewhlmenu
+ global viewargs viewargsep newviewargs viewhlmenu
if {[catch {
set newargs [shellsplit $newviewargs($n)]
@@ -1560,6 +1557,7 @@ proc newviewok {top n} {
set viewperm($n) $newviewperm($n)
set viewfiles($n) $files
set viewargs($n) $newargs
+ set viewargsep($n) "--"
addviewmenu $n
if {!$newishighlight} {
after idle showview $n
@@ -1579,6 +1577,7 @@ proc newviewok {top n} {
if {$files ne $viewfiles($n) || $newargs ne $viewargs($n)} {
set viewfiles($n) $files
set viewargs($n) $newargs
+ set viewargsep($n) "--"
if {$curview == $n} {
after idle updatecommits
}
@@ -6316,10 +6315,12 @@ if {![file isdirectory $gitdir]} {
}
set cmdline_files {}
+set cmdline_sep ""
set i [lsearch -exact $revtreeargs "--"]
if {$i >= 0} {
set cmdline_files [lrange $revtreeargs [expr {$i + 1}] end]
set revtreeargs [lrange $revtreeargs 0 [expr {$i - 1}]]
+ set cmdline_sep "--"
} elseif {$revtreeargs ne {}} {
if {[catch {
set f [eval exec git rev-parse --no-revs --no-flags $revtreeargs]
@@ -6357,6 +6358,7 @@ set selectedhlview None
set viewfiles(0) {}
set viewperm(0) 0
set viewargs(0) {}
+set viewargsep(0) {}
set cmdlineok 0
set stopped 0
@@ -6375,6 +6377,7 @@ if {$cmdline_files ne {} || $revtreeargs ne {}} {
set viewname(1) "Command line"
set viewfiles(1) $cmdline_files
set viewargs(1) $revtreeargs
+ set viewargsep(1) $cmdline_sep
set viewperm(1) 0
addviewmenu 1
.bar.view entryconf Edit* -state normal
--
1.5.2
^ permalink raw reply related
* Re: [PATCH 1/2] Update the bash prompt from 'applied' instead of the obsolete 'current'
From: Karl Hasselström @ 2007-05-22 13:29 UTC (permalink / raw)
To: Catalin Marinas; +Cc: Robin Rosenberg, ydirson, git
In-Reply-To: <b0943d9e0705220511u2b3ed46fg220af2cce26fef96@mail.gmail.com>
On 2007-05-22 13:11:13 +0100, Catalin Marinas wrote:
> On 21/05/07, Karl Hasselström <kha@treskal.com> wrote:
>
> > My suggestion was to have a small stand-alone C program that could
> > do some operations that need to be really fast, such as
> > top/applied/unapplied. It need not have a nice user interface
> > since it's only going to be called by scripts (bash-completion and
> > the like), and it should only handle those operations that _must-
> > avoid the Python startup penalty. And for sanity reasons, it
> > should share code with stgit.
>
> There is one more case to consider - people using NFS-mounted
> directories. The applied/unapplied commands would be even slower and
> the language overhead be negligible.
>
> Another workaround would be to always generate the applied/unapplied
> files when the stack structure changes.
Yes, we could do that. These files would only be accurate when the
stack was last modified with StGIT and not plain git, but that might
be acceptable.
Hmm. Since the only way plain git modifies the stack is by changing
HEAD (we assume the user doesn't manually mess with the patch refs),
we might also write down the value of HEAD for which the
applied/unapplied files are valid, so that the caller could call "stg
applied" if the applied file was out of date. But that's quite a
hassle to have to reimplement every time.
--
Karl Hasselström, kha@treskal.com
www.treskal.com/kalle
^ permalink raw reply
* Re: [StGIT PATCH 5/5] Add --binary flag to commands that generate diffs
From: Karl Hasselström @ 2007-05-22 13:31 UTC (permalink / raw)
To: Catalin Marinas; +Cc: git
In-Reply-To: <b0943d9e0705220515m7eb380cdw277847cf07382761@mail.gmail.com>
On 2007-05-22 13:15:13 +0100, Catalin Marinas wrote:
> On 19/05/07, Karl Hasselström <kha@treskal.com> wrote:
>
> > This just passes the --binary option to git-diff-*, which causes
> > the generated diffs to contain an applyable diff even when binary
> > files differ. It's necessary to do this if you want to mail
> > patches to binary files.
>
> I applied this patch but is there anything wrong if we have this
> option on by default, at least for some commands? Maybe we don't
> need it for 'show' and 'diff' but we definitely need it for 'mail'
> and 'export'.
I'd be fine with that.
> There is also git.apply_diff() which calls git.diff(). This is first
> tried when pushing a patch and followed by a three-way merged if it
> fails. I think we should always have the --binary option in this
> case.
Yes, that sounds good.
--
Karl Hasselström, kha@treskal.com
www.treskal.com/kalle
^ permalink raw reply
* Re: [PATCH] gitweb.perl - Optionally send archives as .zip files
From: Petr Baudis @ 2007-05-22 12:28 UTC (permalink / raw)
To: Mark Levedahl; +Cc: git
In-Reply-To: <11796760064103-git-send-email-mdl123@verizon.net>
Hi,
On Sun, May 20, 2007 at 05:46:46PM CEST, Mark Levedahl wrote:
> git-archive already knows how to generate an archive as a tar or a zip
> file, but gitweb did not. zip archvies are much more usable in a Windows
> environment due to native support and this patch allows a site admin the
> option to deliver zip rather than tar files. The selection is done by
> inserting
>
> $feature{'snapshot'}{'default'} = ['x-zip', 'zip', ''];
>
> in gitweb_config.perl.
>
> Tar files remain the default option.
>
> Signed-off-by: Mark Levedahl <mdl123@verizon.net>
Acked-by: Petr Baudis <pasky@suse.cz>
Maybe the code should decide based on program value being 'zip' rather
than suffix value, but that's just dubious nit-picking.
> @@ -3912,19 +3914,26 @@ sub git_snapshot {
> $hash = git_get_head_hash($project);
> }
>
> - my $filename = decode_utf8(basename($project)) . "-$hash.tar.$suffix";
> + my $git = git_cmd_str();
> + my $name = $project;
> + $name =~ s/\047/\047\\\047\047/g;
By the way, this really looks like an entry to our very own would-be
"git obfuscation of the month" contest. Why did whoever made the code
not just write that ' there? (I know it's been in the original code
too.)
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
-- Samuel Beckett
^ permalink raw reply
* Re: [StGIT RFC] Changing patch@branch syntax
From: Catalin Marinas @ 2007-05-22 12:27 UTC (permalink / raw)
To: Yann Dirson; +Cc: GIT list
In-Reply-To: <20070515220310.GJ16903@nan92-1-81-57-214-146.fbx.proxad.net>
On 15/05/07, Yann Dirson <ydirson@altern.org> wrote:
> The patch@branch syntax is annoying, at least for bash-completion
> purposes: we don't want to provide all possible completions accross
> all branches, yet we'd like to get completion for foreign patches.
>
> Another place where I feel it is bad is when using the full
> "patch@branch//top" syntax: the MSB is in the middle, and the LSB is
> on the right.
I agree that the current syntax is bad.
> Both issues would be solved by switching to a MSB ordering, with a way
> to distinguish branchnames when given. Something similar to pathnames
> would fit well - eg. [/branchname/]patchname[//top]. However, I'm not
> sure using slashes would be a good choice, precisely because of the
> similarity with real pathnames. But we don't have so many separator
> chars that are not special in one way or another, and would require
> quoting them to avoid more user confusion.
We had a discussion some time ago about using slashes for the //top or
//bottom syntax and we ended up using two slashes. We could do the
same to delimitate the branch from the patch - branch//patch//top. The
branch or patch can have (single) slashes in their name.
We couldn't use ":" at that time as it was used for the 'diff -r x:y'
command. I later switched to the git ".." format (when git eventually
defined it). As Karl said, branch:patch@top is another way, unless we
later decide to add another level, the repository, and would like to
have a uniform syntax (maybe always use ":" instead of "//"). We could
run commands like:
stg pick ../../path/to/linux-repo//branch//patch
I don't think using "#" is feasible as bash ignores everything after
it unless you use quotes or escape.
Another nice thing to have is a way to get older versions of a patch
via patchlogs. This should probably follow the current git notation,
i.e. patch^^^ or patch~N.
--
Catalin
^ permalink raw reply
* Re: [StGIT PATCH 5/5] Add --binary flag to commands that generate diffs
From: Catalin Marinas @ 2007-05-22 12:15 UTC (permalink / raw)
To: Karl Hasselström; +Cc: git
In-Reply-To: <20070519001012.4906.86287.stgit@yoghurt>
On 19/05/07, Karl Hasselström <kha@treskal.com> wrote:
> This just passes the --binary option to git-diff-*, which causes the
> generated diffs to contain an applyable diff even when binary files
> differ. It's necessary to do this if you want to mail patches to
> binary files.
I applied this patch but is there anything wrong if we have this
option on by default, at least for some commands? Maybe we don't need
it for 'show' and 'diff' but we definitely need it for 'mail' and
'export'.
There is also git.apply_diff() which calls git.diff(). This is first
tried when pushing a patch and followed by a three-way merged if it
fails. I think we should always have the --binary option in this case.
--
Catalin
^ permalink raw reply
* Re: [PATCH 1/2] Update the bash prompt from 'applied' instead of the obsolete 'current'
From: Catalin Marinas @ 2007-05-22 12:11 UTC (permalink / raw)
To: Karl Hasselström; +Cc: Robin Rosenberg, ydirson, git
In-Reply-To: <20070521153952.GB6474@diana.vm.bytemark.co.uk>
On 21/05/07, Karl Hasselström <kha@treskal.com> wrote:
> My suggestion was to have a small stand-alone C program that could do
> some operations that need to be really fast, such as
> top/applied/unapplied. It need not have a nice user interface since
> it's only going to be called by scripts (bash-completion and the
> like), and it should only handle those operations that _must- avoid
> the Python startup penalty. And for sanity reasons, it should share
> code with stgit.
There is one more case to consider - people using NFS-mounted
directories. The applied/unapplied commands would be even slower and
the language overhead be negligible.
Another workaround would be to always generate the applied/unapplied
files when the stack structure changes.
--
Catalin
^ permalink raw reply
* Re: [PATCH] rename dirlink to gitlink.
From: Petr Baudis @ 2007-05-22 11:53 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Torgil Svensson, Martin Waitz, git
In-Reply-To: <7vodkdbvmk.fsf@assigned-by-dhcp.cox.net>
On Tue, May 22, 2007 at 01:32:51AM CEST, Junio C Hamano wrote:
> "Torgil Svensson" <torgil.svensson@gmail.com> writes:
>
> > On 5/21/07, Martin Waitz <tali@admingilde.org> wrote:
> >> Unify naming of plumbing dirlink/gitlink concept:
> >>
> >> perl -pi -e 's/dirlink/gitlink/g' -e 's/DIRLNK/GITLINK/g'
> >> ---
> >
> > Does this mean that the link doesn't necessarily has to be represented
> > as a directory (even though current code does it) ?
>
> I do not think the patch has that much deep meaning.
>
> Personally I think the patch is similar to renaming "cache" used
> in the code to "index" --- needless code shuffling that does not
> buy you anything.
FWIW, I seriously considered making a patch for that several times - but
always found something better to do; that is, it is not so big an itch
for me to scratch unless I'd be really bored and miss some goot git@
flamewar badly ;-). However, I still think that using "cache" in the
code is a mistake - it's confusing and inconsistent.
(BTW, the other thing I hate about the codebase are the filenames. When
I want to look at the source of some command, I have to try THREE names:
git-something.<tab> (sh, perl, ...), builtin-something.c and
something.c. I wish I could just write git-something.<tab>. :-)
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
-- Samuel Beckett
^ permalink raw reply
* [PATCH] rename gitlink to dirlink.
From: Martin Waitz @ 2007-05-22 11:39 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Torgil Svensson, git
In-Reply-To: <7vd50t9y92.fsf@assigned-by-dhcp.cox.net>
Unify naming of plumbing dirlink/gitlink concept:
perl -pi -e 's/gitlink/dirlink/g'
Signed-off-by: Martin Waitz <tali@admingilde.org>
---
On Mon, May 21, 2007 at 11:19:05PM -0700, Junio C Hamano wrote:
> > But then we also should be consequent and rename the entire
> > low-level plumbing to dirlink. I don't see a reason to keep
> > both.
>
> Ok. Let's do that then before it is too late.
now you can choose which one you want to apply...
builtin-ls-files.c | 2 +-
builtin-update-index.c | 14 +++++++-------
diff.c | 4 ++--
dir.c | 10 +++++-----
entry.c | 2 +-
list-objects.c | 10 +++++-----
read-cache.c | 6 +++---
refs.c | 12 ++++++------
refs.h | 4 ++--
sha1_file.c | 2 +-
tree.c | 2 +-
11 files changed, 34 insertions(+), 34 deletions(-)
diff --git a/builtin-ls-files.c b/builtin-ls-files.c
index f7c066b..ad0267b 100644
--- a/builtin-ls-files.c
+++ b/builtin-ls-files.c
@@ -97,7 +97,7 @@ static void show_other_files(struct dir_struct *dir)
*
* (Matching entries are normally pruned during
* the directory tree walk, but will show up for
- * gitlinks because we don't necessarily have
+ * dirlinks because we don't necessarily have
* dir->show_other_directories set to suppress
* them).
*/
diff --git a/builtin-update-index.c b/builtin-update-index.c
index 8f98991..7fe2466 100644
--- a/builtin-update-index.c
+++ b/builtin-update-index.c
@@ -106,7 +106,7 @@ static int add_one_path(struct cache_entry *old, const char *path, int len, stru
/*
* Handle a path that was a directory. Four cases:
*
- * - it's already a gitlink in the index, and we keep it that
+ * - it's already a dirlink in the index, and we keep it that
* way, and update it if we can (if we cannot find the HEAD,
* we're going to keep it unchanged in the index!)
*
@@ -124,20 +124,20 @@ static int add_one_path(struct cache_entry *old, const char *path, int len, stru
* to try to update it as a directory.
*
* - it doesn't exist at all in the index, but it is a valid
- * git directory, and it should be *added* as a gitlink.
+ * git directory, and it should be *added* as a dirlink.
*/
static int process_directory(const char *path, int len, struct stat *st)
{
unsigned char sha1[20];
int pos = cache_name_pos(path, len);
- /* Exact match: file or existing gitlink */
+ /* Exact match: file or existing dirlink */
if (pos >= 0) {
struct cache_entry *ce = active_cache[pos];
if (S_ISDIRLNK(ntohl(ce->ce_mode))) {
/* Do nothing to the index if there is no HEAD! */
- if (resolve_gitlink_ref(path, "HEAD", sha1) < 0)
+ if (resolve_dirlink_ref(path, "HEAD", sha1) < 0)
return 0;
return add_one_path(ce, path, len, st);
@@ -162,8 +162,8 @@ static int process_directory(const char *path, int len, struct stat *st)
return error("%s: is a directory - add individual files instead", path);
}
- /* No match - should we add it as a gitlink? */
- if (!resolve_gitlink_ref(path, "HEAD", sha1))
+ /* No match - should we add it as a dirlink? */
+ if (!resolve_dirlink_ref(path, "HEAD", sha1))
return add_one_path(NULL, path, len, st);
/* Error out. */
@@ -179,7 +179,7 @@ static int process_file(const char *path, int len, struct stat *st)
struct cache_entry *ce = pos < 0 ? NULL : active_cache[pos];
if (ce && S_ISDIRLNK(ntohl(ce->ce_mode)))
- return error("%s is already a gitlink, not replacing", path);
+ return error("%s is already a dirlink, not replacing", path);
return add_one_path(ce, path, len, st);
}
diff --git a/diff.c b/diff.c
index 33297aa..434947e 100644
--- a/diff.c
+++ b/diff.c
@@ -1432,7 +1432,7 @@ static int populate_from_stdin(struct diff_filespec *s)
return 0;
}
-static int diff_populate_gitlink(struct diff_filespec *s, int size_only)
+static int diff_populate_dirlink(struct diff_filespec *s, int size_only)
{
int len;
char *data = xmalloc(100);
@@ -1468,7 +1468,7 @@ int diff_populate_filespec(struct diff_filespec *s, int size_only)
return 0;
if (S_ISDIRLNK(s->mode))
- return diff_populate_gitlink(s, size_only);
+ return diff_populate_dirlink(s, size_only);
if (!s->sha1_valid ||
reuse_worktree_file(s->path, s->sha1, 0)) {
diff --git a/dir.c b/dir.c
index 11fab7f..1f17fc7 100644
--- a/dir.c
+++ b/dir.c
@@ -300,7 +300,7 @@ enum exist_status {
/*
* The index sorts alphabetically by entry name, which
- * means that a gitlink sorts as '\0' at the end, while
+ * means that a dirlink sorts as '\0' at the end, while
* a directory (which is defined not as an entry, but as
* the files it contains) will sort with the '/' at the
* end.
@@ -343,8 +343,8 @@ static enum exist_status directory_exists_in_index(const char *dirname, int len)
* directory name, we always recurse into the directory to see
* all the files.
*
- * Case 2: If we *already* have that directory name as a gitlink,
- * we always continue to see it as a gitlink, regardless of whether
+ * Case 2: If we *already* have that directory name as a dirlink,
+ * we always continue to see it as a dirlink, regardless of whether
* there is an actual git directory there or not (it might not
* be checked out as a subproject!)
*
@@ -356,7 +356,7 @@ static enum exist_status directory_exists_in_index(const char *dirname, int len)
* also true and the directory is empty, in which case
* we just ignore it entirely.
* (b) if it looks like a git directory, and we don't have
- * 'no_dirlinks' set we treat it as a gitlink, and show it
+ * 'no_dirlinks' set we treat it as a dirlink, and show it
* as a directory.
* (c) otherwise, we recurse into it.
*/
@@ -385,7 +385,7 @@ static enum directory_treatment treat_directory(struct dir_struct *dir,
break;
if (!dir->no_dirlinks) {
unsigned char sha1[20];
- if (resolve_gitlink_ref(dirname, "HEAD", sha1) == 0)
+ if (resolve_dirlink_ref(dirname, "HEAD", sha1) == 0)
return show_directory;
}
return recurse_into_directory;
diff --git a/entry.c b/entry.c
index 82bf725..db66663 100644
--- a/entry.c
+++ b/entry.c
@@ -193,7 +193,7 @@ int checkout_entry(struct cache_entry *ce, const struct checkout *state, char *t
*/
unlink(path);
if (S_ISDIR(st.st_mode)) {
- /* If it is a gitlink, leave it alone! */
+ /* If it is a dirlink, leave it alone! */
if (S_ISDIRLNK(ntohl(ce->ce_mode)))
return 0;
if (!state->force)
diff --git a/list-objects.c b/list-objects.c
index 310f8d3..8027e02 100644
--- a/list-objects.c
+++ b/list-objects.c
@@ -26,7 +26,7 @@ static void process_blob(struct rev_info *revs,
}
/*
- * Processing a gitlink entry currently does nothing, since
+ * Processing a dirlink entry currently does nothing, since
* we do not recurse into the subproject.
*
* We *could* eventually add a flag that actually does that,
@@ -34,20 +34,20 @@ static void process_blob(struct rev_info *revs,
* - is the subproject actually checked out?
* - if so, see if the subproject has already been added
* to the alternates list, and add it if not.
- * - process the commit (or tag) the gitlink points to
+ * - process the commit (or tag) the dirlink points to
* recursively.
*
* However, it's unclear whether there is really ever any
* reason to see superprojects and subprojects as such a
* "unified" object pool (potentially resulting in a totally
* humongous pack - avoiding which was the whole point of
- * having gitlinks in the first place!).
+ * having dirlinks in the first place!).
*
* So for now, there is just a note that we *could* follow
* the link, and how to do it. Whether it necessarily makes
* any sense what-so-ever to ever do that is another issue.
*/
-static void process_gitlink(struct rev_info *revs,
+static void process_dirlink(struct rev_info *revs,
const unsigned char *sha1,
struct object_array *p,
struct name_path *path,
@@ -88,7 +88,7 @@ static void process_tree(struct rev_info *revs,
lookup_tree(entry.sha1),
p, &me, entry.path);
else if (S_ISDIRLNK(entry.mode))
- process_gitlink(revs, entry.sha1,
+ process_dirlink(revs, entry.sha1,
p, &me, entry.path);
else
process_blob(revs,
diff --git a/read-cache.c b/read-cache.c
index d9f46da..3aa92a4 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -86,7 +86,7 @@ static int ce_compare_link(struct cache_entry *ce, size_t expected_size)
return match;
}
-static int ce_compare_gitlink(struct cache_entry *ce)
+static int ce_compare_dirlink(struct cache_entry *ce)
{
unsigned char sha1[20];
@@ -98,7 +98,7 @@ static int ce_compare_gitlink(struct cache_entry *ce)
*
* If so, we consider it always to match.
*/
- if (resolve_gitlink_ref(ce->name, "HEAD", sha1) < 0)
+ if (resolve_dirlink_ref(ce->name, "HEAD", sha1) < 0)
return 0;
return hashcmp(sha1, ce->sha1);
}
@@ -145,7 +145,7 @@ static int ce_match_stat_basic(struct cache_entry *ce, struct stat *st)
case S_IFDIRLNK:
if (!S_ISDIR(st->st_mode))
changed |= TYPE_CHANGED;
- else if (ce_compare_gitlink(ce))
+ else if (ce_compare_dirlink(ce))
changed |= DATA_CHANGED;
return changed;
default:
diff --git a/refs.c b/refs.c
index 89876bf..ee79329 100644
--- a/refs.c
+++ b/refs.c
@@ -285,7 +285,7 @@ static struct ref_list *get_loose_refs(void)
#define MAXDEPTH 5
#define MAXREFLEN (1024)
-static int resolve_gitlink_packed_ref(char *name, int pathlen, const char *refname, unsigned char *result)
+static int resolve_dirlink_packed_ref(char *name, int pathlen, const char *refname, unsigned char *result)
{
FILE *f;
struct cached_refs refs;
@@ -312,7 +312,7 @@ static int resolve_gitlink_packed_ref(char *name, int pathlen, const char *refna
return retval;
}
-static int resolve_gitlink_ref_recursive(char *name, int pathlen, const char *refname, unsigned char *result, int recursion)
+static int resolve_dirlink_ref_recursive(char *name, int pathlen, const char *refname, unsigned char *result, int recursion)
{
int fd, len = strlen(refname);
char buffer[128], *p;
@@ -322,7 +322,7 @@ static int resolve_gitlink_ref_recursive(char *name, int pathlen, const char *re
memcpy(name + pathlen, refname, len+1);
fd = open(name, O_RDONLY);
if (fd < 0)
- return resolve_gitlink_packed_ref(name, pathlen, refname, result);
+ return resolve_dirlink_packed_ref(name, pathlen, refname, result);
len = read(fd, buffer, sizeof(buffer)-1);
close(fd);
@@ -343,10 +343,10 @@ static int resolve_gitlink_ref_recursive(char *name, int pathlen, const char *re
while (isspace(*p))
p++;
- return resolve_gitlink_ref_recursive(name, pathlen, p, result, recursion+1);
+ return resolve_dirlink_ref_recursive(name, pathlen, p, result, recursion+1);
}
-int resolve_gitlink_ref(const char *path, const char *refname, unsigned char *result)
+int resolve_dirlink_ref(const char *path, const char *refname, unsigned char *result)
{
int len = strlen(path), retval;
char *gitdir;
@@ -359,7 +359,7 @@ int resolve_gitlink_ref(const char *path, const char *refname, unsigned char *re
memcpy(gitdir, path, len);
memcpy(gitdir + len, "/.git/", 7);
- retval = resolve_gitlink_ref_recursive(gitdir, len+6, refname, result, 0);
+ retval = resolve_dirlink_ref_recursive(gitdir, len+6, refname, result, 0);
free(gitdir);
return retval;
}
diff --git a/refs.h b/refs.h
index f61f6d9..5cbb711 100644
--- a/refs.h
+++ b/refs.h
@@ -60,7 +60,7 @@ extern int check_ref_format(const char *target);
/** rename ref, return 0 on success **/
extern int rename_ref(const char *oldref, const char *newref, const char *logmsg);
-/** resolve ref in nested "gitlink" repository */
-extern int resolve_gitlink_ref(const char *name, const char *refname, unsigned char *result);
+/** resolve ref in nested "dirlink" repository */
+extern int resolve_dirlink_ref(const char *name, const char *refname, unsigned char *result);
#endif /* REFS_H */
diff --git a/sha1_file.c b/sha1_file.c
index be991ed..c37fbed 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -2393,7 +2393,7 @@ int index_path(unsigned char *sha1, const char *path, struct stat *st, int write
free(target);
break;
case S_IFDIR:
- return resolve_gitlink_ref(path, "HEAD", sha1);
+ return resolve_dirlink_ref(path, "HEAD", sha1);
default:
return error("%s: unsupported file type", path);
}
diff --git a/tree.c b/tree.c
index e4a39aa..d8014eb 100644
--- a/tree.c
+++ b/tree.c
@@ -141,7 +141,7 @@ struct tree *lookup_tree(const unsigned char *sha1)
/*
* NOTE! Tree refs to external git repositories
- * (ie gitlinks) do not count as real references.
+ * (ie dirlinks) do not count as real references.
*
* You don't have to have those repositories
* available at all, much less have the objects
--
1.5.0.3
--
Martin Waitz
^ permalink raw reply related
* StGIT unhelpfulness
From: Petr Baudis @ 2007-05-22 11:16 UTC (permalink / raw)
To: git
Hi,
I tried to stg pull to my git repository at repo.or.cz, but stg pull
only laconically announced
stg pull: Rebasing would possibly lose data
and left me with a confused stare. The same with popping all the patches
first, so there is actually nothing _to_ rebase, but it still fails.
Five minutes of peeking at the stgit code later, I think this is
because I sometimes do stg pull (now) and sometimes stg pop -a, cg
update, stg push -a when I'm feeling nostalgic. orig-base doesn't get
updated and stgit gets unhappy, but frankly, I don't even get _why_ does
it need to record and check orig-base - how does it matter?
Besides, when it doesn't like it, it should give me some more helpful
error message than just the cryptic above...
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
-- Samuel Beckett
^ permalink raw reply
* Re: [PATCH] Teach 'git-apply --whitespace=strip' to remove empty lines at the end of file
From: Marco Costalba @ 2007-05-22 11:13 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Git Mailing List
In-Reply-To: <7vbqgdbq5j.fsf@assigned-by-dhcp.cox.net>
On 5/22/07, Junio C Hamano <junkio@cox.net> wrote:
> "Marco Costalba" <mcostalba@gmail.com> writes:
>
> > On 5/21/07, Junio C Hamano <junkio@cox.net> wrote:
> >> Junio C Hamano <junkio@cox.net> writes:
> >>
> >>
> >> We somehow end up removing one LF too many, like this:
> >>
> >> diff --git a/contrib/emacs/.gitignore b/contrib/emacs/.gitignore
> >> index c531d98..016d3b1 100644
> >> --- a/contrib/emacs/.gitignore
> >> +++ b/contrib/emacs/.gitignore
> >> @@ -1 +1 @@
> >> -*.elc
> >> +*.elc
> >> \ No newline at end of file
> >>
> >
The final, and correct version is:
if (new_whitespace == strip_whitespace && trailing_added_lines) {
int n = 0;
for ( ; n <= trailing_added_lines; n++) { /* counting trailing '\n' */
if (newsize == n) {
n++;
break;
}
if (new[newsize - 1 - n] != '\n')
break;
}
trailing_added_lines = (n>0) ? --n : 0;
} else
trailing_added_lines = 0;
but I understand is ugly as hell. The fact is, it is far easier to
count '\n' *while* they are created then after at the end.
So no problem for me if you drop my patch.
Marco
^ permalink raw reply
* Re: [PATCH] Prevent megablobs from gunking up git packs
From: Jakub Narebski @ 2007-05-22 11:05 UTC (permalink / raw)
To: git
In-Reply-To: <56b7f5510705220100h77e91196r1784b33772911660@mail.gmail.com>
Dana How wrote:
> There's actually an even more extreme example from my day job.
> The software team has a project whose files/revisions would be
> similar to those in the linux kernel (larger commits, I'm sure).
> But they have *ONE* 500MB file they check in because it takes
> 2 or 3 days to generate and different people use different versions of it.
> I'm sure it has 50+ revisions now. If they converted to git and included
> these blobs in their packfile, that's a 25GB uncompressed increase!
> *Every* git operation must wade through 10X -- 100X more packfile.
> Or it could be kept in 50+ loose objects in objects/xx ,
> requiring a few extra syscalls by each user to get a new version.
Or keeping those large objects in separate, _kept_ packfile, containing
only those objects (which can delta well, even if they are large).
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* Re: gitweb - encoding problems
From: Jakub Narebski @ 2007-05-22 7:50 UTC (permalink / raw)
To: David Woodhouse, Martin Koegler; +Cc: git, Junio C Hamano
In-Reply-To: <1179794012.2771.112.camel@shinybook.infradead.org>
On Thu, 22 May 2007, David Woodhouse wrote:
> On Mon, 2007-05-21 at 22:57 +0200, Martin Koegler wrote:
>> I use ISO-8859-1 as my locale, so my blobs, commits and tags are in
>> this encoding.
>
> That's a very strange thing for anyone to do in the 21st century.
> Did you configure this archaic thing correctly in .git/config?
>
> Otherwise, gitweb will assume that you're using utf-8 like any normal
> person would, and of course you'll have problems when it tries to deal
> with your legacy character set as if it were something sensible.
Actually gitweb does not respect i18n.* configuration variables and
happily assumes that everything is in utf-8, with the exception of
*_plain views, which are send :raw.
If you decide to implement supporting encodings other that utf-8 in
gitweb, please remember that some (like git-show, git-log) but not all
parts (like git-rev-list or --pretty=raw) do the decoding/encoding. And
that git can be compiled without iconv support. And that comits might
be in different encodings, which should be given by 'encoding' header,
but there is no way to guess encoding for a blob, or for a file names.
git-commit(1):
i18n.commitEncoding::
Character encoding the commit messages are stored in; git itself
does not care per se, but this information is necessary e.g. when
importing commits from emails or in the gitk graphical history
browser (and possibly at other places in the future or in other
porcelains). See e.g. gitlink:git-mailinfo[1]. Defaults to 'utf-8'.
i18n.logOutputEncoding::
Character encoding the commit messages are converted to when
running `git-log` and friends.
--
Jakub Narebski
Poland
^ permalink raw reply
* do_switch()-enabled SVN binaries for git-svn
From: Eric Wong @ 2007-05-22 9:36 UTC (permalink / raw)
To: David Kastrup; +Cc: git
In-Reply-To: <20070522091619.GA17099@muzzle>
Eric Wong <normalperson@yhbt.net> wrote:
> If recompiling SVN is feasible for you and the branches (as a fresh
> checkout) are as big as trunk, I highly recommend the do_switch patch
> for SVN which lets you transfer only a delta between the branch/tag
> point of trunk:
>
> http://svn.haxx.se/dev/archive-2007-01/0936.shtml
In case you're using Debian x86-32, I've uploaded .deb packages
(with full, Debian sources) here:
http://git-svn.bogomips.org/svn/
The patch by itself is available here:
http://git-svn.bogomips.org/svn/switch-editor-perl.diff
--
Eric Wong
^ permalink raw reply
* Re: [PATCH] git-pack-objects: cache small deltas between big objects
From: Dana How @ 2007-05-22 9:25 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Martin Koegler, git, danahow
In-Reply-To: <7vhcq58et1.fsf@assigned-by-dhcp.cox.net>
On 5/22/07, Junio C Hamano <junkio@cox.net> wrote:
> "Dana How" <danahow@gmail.com> writes:
> > If I simply refuse to insert enormous blobs in the packfiles, and keep
> > them loose, the performance is better. More importantly, my packfiles
> > are now sized like everyone else's, so I'm in an operating regime which
> > everyone is testing and optimizing. This was not true with 12GB+ of packfiles.
> > Of course, loose objects are slower, but slight extra overhead to access
> > something large enough to be noticeable already doesn't bother me.
> >
> > Finally, loose objects don't get deltified. This is a problem, but I would
> > need to repack at least every week, and nonzero window/depth would
> > be prohibitive with large objects included.
>
> Here are a few quick comments before going to bed.
>
> * The objects in the packfile are ordered in "recency" order,
> as "rev-list --objects" feeds you, so it is correct that we
> get trees and blobs mixed. It might be an interesting
> experiment, especially with a repository without huge blobs,
> to see how much improvement we might get if we keep the
> recency order _but_ emit tags, commits, trees, and then
> blobs, in this order. In write_pack_file() we have a single
> loop to call write_one(), but we could make it a nested loop
> that writes only objects of each type.
Already tried that, almost. Added a --types=[ctgb]+ flag to
pack-objects, and changed to git-repack to run in 2 passes
when -a && --max-pack-size. The first pass would create
packfiles with all Commits/Trees/taGs [of course just 1],
the second made packfiles with just Blobs. With a warm cache,
this was 3X to 7X slower than --max-blob-size= approach
(for the git-log --pretty=oneline example). Why? I'm guessing
because each lookup had to go through 7 index files instead of 1,
which would be significant when processing very small blobs
(commits and trees). And the _slower_ one had window/depth=0/0,
so it had no delta expansion to do.
> * Also my earlier "nodelta" attribute thing would be worth
> trying with your repository with huge blobs, with the above
> "group by object type" with further tweak to write blobs
> without "nodelta" marker first and then finally blobs with
> "nodelta" marker.
I started out enthusiastic about "nodelta", causing me to
quickly propose "norepack" as well. However, there is no
simple way in my repository to specify these. Most of the
enormous files have certain suffixes, but each of these
appears on a continuum of file sizes, so I can't write
any *.sfx rules in .gitattributes. I could make rules
specific to specific files, but then I would have to write
scripts to auto-generate them. (At commit time?)
Assuming I *could* get "nodelta" properly specified,
putting these last would help somewhat. But we would
still be left with the problem caused by extra index files
(resulting from 2GB packfile limit).
> I suspect the above two should help "git log" and "git log --
> pathspec..." performance, as these two do not look at blobs at
> all (pathspec limiting does invoke diff machinery, but that is
> only at the tree level).
>
> The "I want to have packs with reasonable size as everybody
> else" (which I think is a reasonable thing to want, but does not
> have much technical meaning as other issues do) wish is
> something we cannot _measure_ to judge pros and cons, ...
?? A depressingly large portion of my career has been spent
fooling optimization programs to work on new problems, by
making the new problem look just like what they're used to. So
wanting a program's input to look "conventional", or similar
to something in a regression, seems pretty reasonable.
It's just the data-side version of preferring small changes in an algorithm.
> ... but with
> the above experiment, you could come up with three set of packs
> such that, all three sets use "nodelta" to leave the huge blobs
> undeltified, and use the default window and depth for others,
> and:
>
> (1) One set has trees and blobs mixed;
>
> (2) Another set has trees and blobs grouped, but "nodelta" blobs
> and others are not separated;
>
> (3) The third set has trees and blobs grouped, and "nodelta"
> blobs and others are separated.
>
> Comparing (1) and (2) would show how bad it is to have huge
> blobs in between trees (which are presumably accessed more
> often). I suspect that comparing (2) and (3) would show that
> for most workloads, the split is not worth it.
>
> And compare (3) with another case where you leave "nodelta"
> blobs loose. That's the true comparison that would demonstrate
> why placing huge blobs in packs is bad and they should be left
> loose. I'm skeptical if there will be significant differences,
> though.
I think the difference will come from at least the different number of
index files,
as pointed out above. I can certainly start on these comparisons.
I must say, getting the whole repository to repack -a under an hour
with great git-log performance after just a 55-line change was a much better
experience than the 10X larger max-pack-size patch...
BTW, why the attachment to keeping *everything* in a packfile?
If I implement the changes above, they will be more extensive than
max-blob-size (even max-pack-size only added *1* new nested loop
to pack-objects), and they'll be climbing uphill due to the packfiles
being THREE orders of magnitude larger and the index files one
order of magnitude more numerous.
Thanks,
--
Dana L. How danahow@gmail.com +1 650 804 5991 cell
^ permalink raw reply
* Re: git-svn or git problem...
From: Eric Wong @ 2007-05-22 9:16 UTC (permalink / raw)
To: David Kastrup; +Cc: git
In-Reply-To: <86abvylihb.fsf@lola.quinscape.zz>
David Kastrup <dak@gnu.org> wrote:
>
> Hi,
>
> I have used git-svn for checking out the trunk of a large Subversion
> archive. The .git subdirectory now sits in the /rep/texlive/trunk
> directory.
>
> But I actually would now want to have the branches (which are pretty
> small in comparison) available in git as well, without checking the
> whole trunk out again.
> Is there a way to move the .git tracking one directory level upwards
> and get the branches into it as well, without checking out the trunk
> again (the trunk is several Gigabytes of size)?
If you're using a 1.5.1 or later version of git-svn, you can do so
pretty easily by editing your .git/config:
If you already have something like:
[svn-remote "svn"]
# url here is the repository root, and 'mpc/trunk' in fetch
# is the relative path within the repository
url = https://svn.musicpd.org
fetch = mpc/trunk:refs/remotes/git-svn
You can just add the following lines to the config in the above section:
branches = mpc/branches/*:refs/remotes/*
tags = mpc/tags/*:refs/remotes/tags/*
If you have something like:
[svn-remote "svn"]
# url here is the full path of what you're tracking,
url = https://svn.musicpd.org/mpc/trunk
fetch = :refs/remotes/git-svn
Change it to something like in the first example (assuming you
have read permissions to the repository root).
If recompiling SVN is feasible for you and the branches (as a fresh
checkout) are as big as trunk, I highly recommend the do_switch patch
for SVN which lets you transfer only a delta between the branch/tag
point of trunk:
http://svn.haxx.se/dev/archive-2007-01/0936.shtml
--
Eric Wong
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox