* Re: [PATCH 07/16] git-read-tree: take --submodules option
From: Junio C Hamano @ 2007-05-24 20:32 UTC (permalink / raw)
To: Sven Verdoolaege; +Cc: Martin Waitz, Jan Hudec, Alex Riesen, git
In-Reply-To: <20070524191438.GZ942MdfPADPa@greensroom.kotnet.org>
Sven Verdoolaege <skimo@kotnet.org> writes:
[side note: I am ignoring your reply-to: liacs.nl as its MTA
seems to use sorbs that has my ISP's outgoing sender identified
as spam source; I'll send the bounce to you privately in a
separate message.]
> On Thu, May 24, 2007 at 11:58:26AM -0700, Junio C Hamano wrote:
>> Sven Verdoolaege <skimo@kotnet.org> writes:
>> > On Thu, May 24, 2007 at 11:26:01AM -0700, Junio C Hamano wrote:
>> >> (2) In superproject .git/, we would have a bare repository for
>> >> each project used by the superproject.
>> >>
>> >> .git/subproject/kernel26/{objects,refs,...}
>> >>
>> >> This is created by making a bare clone from the upstream
>> >> URL, decided by the user with the help from suggested URL
>> >> described in the superproject .gitmodules.
>> >
>> > Do you mean a "pure" clone, i.e., without a working tree,
>> > but with separate-remotes?
>>
>> I meant a bare clone without separate remotes.
>
> Why without separate remotes?
> It has been argued before that changes in the subproject
> may come from different remotes, so the user may want
> to configure extra remotes from which to fetch.
By different remotes, which do you mean?
(1) .git/subprojects/kernel26/ repository has 'origin'
different from any of the suggested URL in .gitmodules, but
as far as it is concerned there is one 'origin';
or
(2) it has 'origin' that is what the superproject suggests, but
the user locally uses additional repositories to pull and
merge from;
If the former that is not an argument, so I'd assume the
latter. I would say in such a case, you are better off having a
usual repository to manage the development of subproject part,
not grafted to any superproject repository, and handle such
merges there (after all, a "subproject" should stand on its own
without having any of the superproject stuff). And treat THAT
repository as the 'origin' used in (1) above. It might be
easier to use non separate-remote layout in the standalone
repository for the subproject, but that is a separate issue.
>> The counter-proposal outline essentially says, for the sake of
>> simplicity, "nuke existing subproject directory whenever we need
>> to replace it with something else, and reclone a new/replacement
>> subproject directory every time we need to check it out, after
>> making sure nothing is lost".
>
> And she can't do it in the clone in his working tree if that's
> going to get nuked from time to time.
And she does not have to. She can do the development/fixes in
(temporarily) checked out subproject tree, and push it back to
the .git/subproject/kernel26/ repository in the superproject
before she leaves (i.e. before branch switching at superproject
level needs to obliterate it). The change stored in the
.git/subproject/kernel26/ repository in the superproject can
further be pushed back to its 'origin', be it the true
"upstream", or "the standalone repository for the subproject" I
mentioned above.
> But you still need figure out _what_ to fetch.
> Before you suggested to just use the default set up by
> clone with separate remotes, but you no longer have that
> in your new proposal.
I do remember saying the "default set up by clone" but I did not
mean separate remotes.
What is fetched by a bare and non-separate-remote repository vs
a repository that uses separate-remote layout from 'origin' is
exactly the same -- the difference is only 'pure/bare' layout
would use
fetch = refs/heads/*:refs/heads/*
while separate-remotes would use
fetch = refs/heads/*:/refs/remotes/origin/*
So I do not think the difference matters for our purpose of
being able to check out commits that are referenced in
superproject trees. As long as we require that the 'origin' for
"longer term repository to keep track of the subproject in
superproject" (aka repository (2) in my message you are
responding to) always contain the commit referenced by tree
objects in the superproject, which I think is a sensible thing
to require (otherwise you cannot even clone and checkout the
whole superproject), both layout would work equally well. It's
just bare/pure layout is easier to understand because it is
essentially a "mirror" of the upstream.
^ permalink raw reply
* Re: [PATCH] Prevent megablobs from gunking up git packs
From: Geert Bosch @ 2007-05-24 20:43 UTC (permalink / raw)
To: Shawn O. Pearce, Geert Bosch; +Cc: Git Mailing List, Junio C Hamano, Dana How
In-Reply-To: <20070524071235.GL28023@spearce.org>
[resent because of malformed headers causing rejection]
On May 24, 2007, at 03:12, Shawn O. Pearce wrote:
> I still don't buy the idea that these megablobs shouldn't be packed.
> I understand Dana's pain here (at least a little bit, my problems
> aren't as bad as his are), but I also hate to see us run away from
> packfiles for these really sick cases just because we have some
> issues in our current packfile handling.
>
> Packfiles give us a lot of benefits:
>
> 1) less inode usage;
Using 1 inode per huge blob can never be an issue
> 2) transport can write directly to local disk;
> 3) transport can (quickly) copy from local disk;
Can do these by re-enabling the new loose object format
> 4) testing for existance is *much* faster;
> 5) deltafication is possible;
Look at it the other way. If we have huge objects (say >1GB),
we should put them in a pack of their own anyway. What's better:
having a pack with a separate index file or just a loose object?
While the one object per file model is awful for many small files
with lots of similarity, it is really quite efficient for large
objects, and the most reasonable model for huge objects.
Such blobs are just too large to do anything useful with.
The only operations done on them will be to check them in
or check them out. Ideally, we should never try to have them
in memory at all, but just stream them to/from disk while
compressing or decompressing.
Trying to deltify huge objects just takes too much time.
Similarly, we don't want to read 100MB to then apply a delta
and maybe throw out half of the data we read in the first place.
It's just too inefficient. If we'd even read the huge blobs once
during "git repack", we'll waste so much time that we're unlikely
to ever gain it back in any real world scenario.
-Geert
^ permalink raw reply
* Re: [PATCH 07/16] git-read-tree: take --submodules option
From: Petr Baudis @ 2007-05-24 20:55 UTC (permalink / raw)
To: Junio C Hamano
Cc: Sven Verdoolaege, Martin Waitz, Jan Hudec, Alex Riesen, git
In-Reply-To: <7vy7jeufmn.fsf@assigned-by-dhcp.cox.net>
On Thu, May 24, 2007 at 10:32:32PM CEST, Junio C Hamano wrote:
> And she does not have to. She can do the development/fixes in
> (temporarily) checked out subproject tree, and push it back to
> the .git/subproject/kernel26/ repository in the superproject
> before she leaves (i.e. before branch switching at superproject
> level needs to obliterate it). The change stored in the
> .git/subproject/kernel26/ repository in the superproject can
> further be pushed back to its 'origin', be it the true
> "upstream", or "the standalone repository for the subproject" I
> mentioned above.
..snip..
> It's just bare/pure layout is easier to understand because it is
> essentially a "mirror" of the upstream.
I OTOH think that it's less confusing to just keep it the same for all
the normal repositories - and you yourself did not rule out the
possibility to have local changes in the repository, in which case we
certainly should use the separate-remotes layout or we go back to last
year's refs mess.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
-- Samuel Beckett
^ permalink raw reply
* Re: [PATCH 07/16] git-read-tree: take --submodules option
From: Petr Baudis @ 2007-05-24 20:57 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Martin Waitz, Jan Hudec, skimo, Alex Riesen, git
In-Reply-To: <7viraixeme.fsf@assigned-by-dhcp.cox.net>
On Thu, May 24, 2007 at 08:26:01PM CEST, Junio C Hamano wrote:
> How about doing something like this, instead?
These discussions are spread over so many posts (and especially threads)
that it's far beyond me to track it all - I hope I won't repeat
something already debunked.
> (1) superproject .gitmodules (in-tree) and .git/config (local
> repository) use the three-level naming in $gmane/47567.
> Namely, (1a) .gitmodules says which subdirectory has a
> checkout of what project, and names the project in
> logical/abstract terms, not with a URL (e.g. "kernel26");
> (1b) .gitmodules also associates a set of suggested URLs
> for each of the logical/abstract project name; (1c)
> .git/config records which project are of interest.
How do you deal with clashes in subproject names? Until now, several
independent projects might live happily in a single repository, this
breaks that. When merging disparate projects, you can resolve name
clashes in .gitmodules, but going back to one of the merged trunks just
won't work. It's ugly.
Now, we can just declare that we don't care about this case. This stance
wouldn't make me comfortable at all, but at least we should make it
clear that we know about this problem and the users are on their own
when it happens.
> (2) In superproject .git/, we would have a bare repository for
> each project used by the superproject.
>
> .git/subproject/kernel26/{objects,refs,...}
>
> This is created by making a bare clone from the upstream
> URL, decided by the user with the help from suggested URL
> described in the superproject .gitmodules.
>
> The idea is to use this repository as a long-term
> subproject state across branch switching.
When not using subproject aliases, you could just name this after some
normalized form of the URL (anything suitable up to an sha1sum of the
URL :).
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
-- Samuel Beckett
^ permalink raw reply
* Re: [PATCH 07/16] git-read-tree: take --submodules option
From: Junio C Hamano @ 2007-05-24 20:59 UTC (permalink / raw)
To: Petr Baudis; +Cc: Sven Verdoolaege, Martin Waitz, Jan Hudec, Alex Riesen, git
In-Reply-To: <20070524205535.GP4489@pasky.or.cz>
Petr Baudis <pasky@suse.cz> writes:
> On Thu, May 24, 2007 at 10:32:32PM CEST, Junio C Hamano wrote:
> ...
>> It's just bare/pure layout is easier to understand because it is
>> essentially a "mirror" of the upstream.
>
> I OTOH think that it's less confusing to just keep it the same for all
> the normal repositories - and you yourself did not rule out the
> possibility to have local changes in the repository, in which case we
> certainly should use the separate-remotes layout or we go back to last
> year's refs mess.
That "local changes in the repository" will happen in the clone
created in kernel-src/ directory and I do not have any problem
using separate remotes there. I was talking about the refs
layout used in .git/subproject/kernel26/, which is a mirror of
the upstream and the clone source for that (temporarily checked
out) repository at kernel-src/ directory. After all, the
"mirror" is where you would push your local changes back into,
and it is more common to use non-separate-remote layout (without
even 'master'->'origin' mapping, hence I had to come up with a
new name 'pure', but I realize that what I meant really is a
'mirror') there.
^ permalink raw reply
* Re: How do we import patches from non-git sources?
From: Yann Dirson @ 2007-05-24 21:22 UTC (permalink / raw)
To: Marc Singer; +Cc: git
In-Reply-To: <1180017010.21181.2.camel@zealous.synapsedev.com>
On Thu, May 24, 2007 at 07:30:10AM -0700, Marc Singer wrote:
> Is there a way to import patches that did not come from git? Remember
> that we'd like to include the functionality of git-am that adds new
> files to the index.
I have written a patch-application tool as part of the (otherwise
stalled) ArcheoloGIT project, clonable from
http://ydirson.free.fr/soft/git/argit.git/.
Side note: it could make sense to finalize this script and integrate
it in git proper. Opinions ?
Although the command-line interface would probably benefit from an
overhaul, it is fully functionning, notably supporting:
- application of several patches in one row
- application of non-incremental patches - eg. the 2.4.19rmk2 patch
applies to plain 2.4.19, not to 2.4.19rmk1
| $ ../argit/ag-import-patch --help
| Usage: ag-import-patch ( [-v] [-n] [-b BASE|-i] [-N NAME] [-p PARENT]... [-t TAG] PATCH )*
|
| Import a revision in a GIT history from a patch.
| Part of the ArcheoloGIT toolkit.
| Copyright (c) Yann Dirson, 2005
| Distributed under version 2 of the GNU GPL.
Since it lacks doc, here is it:
-i - process subsequent paches as incremental (the default)
-b BASE - process subsequent paches as applying to commit BASE
-N NAME - use given NAME instead of patch filename for generating
commit message (useful for /dev/stdin import)
-t TAG - tag after import
-p PARENT - add PARENT to the parents
eg:
$ git checkout v2.4.19
$ ag-import-patch -b v2.4.19 \
-p v2.4.18rmk6 -t v2.4.19rmk1 patch-2.4.19rmk1.diff \
-t v2.4.19rmk2 patch-2.4.19rmk2.diff
Hope this helps,
--
Yann.
^ permalink raw reply
* Problem using git svn clone
From: Gustaf Hendeby @ 2007-05-24 21:35 UTC (permalink / raw)
To: git
Hi!
I have a problem with git-svn, which I hope the list can shed some
light over. I'm quite new to git (but so far I really like it, great
job guys!) and I never use SVN, so don't really know what information
may be needed to be able to help me on the right track. I hope the
below problem description is appropriate.
The situation is as follows:
I plan to use git and git svn to track a module in the middle of a SVN
repository (which I have no control over) where I don't have read
access more than to this specific module and the base directory, ie
the layout is similar to this
https://svn.foo.bar/a/b/c/trunk
https://svn.foo.bar/a/b/c/tags
https://svn.foo.bar/a/b/c/branches
and I have read access to https://svm.foo.bar/a,
https://svn.foo.bar/a/b/c and below, but nothing else.
What I want to do is clone the whole thing, including tags and
branches. I expected to be able to use the following command to do
this (git v1.5.2):
$ git svn clone https://svn.foo.bar/a/b/c -T trunk -t tags -b branches
Initialized empty Git repository in .git/
Using higher level of URL: https://svn.foo.bar/a/b/c => https://svn.foo.bar/a
W: Ignoring error from SVN, path probably does not exist: (175002): RA
layer request failed: REPORT request failed on '/a/!svn/bc/100':
REPORT of '/a/!svn/bc/100': Could not read chunk size: Secure
connection truncated (https://svn.foo.bar)
r306 = 83f0c10b988a8f1e77a3f354126c52f0cfdecf76 (trunk)
[...]
Found possible branch point: https://svn.foo.bar/a/b/c/trunk =>
https://svn.foo.bar/a/b/c/tags/0.1, 314
Found branch parent: (tags/0.1) ad6a2361d0e69d6288ef226bb335bb4bf3bdd12e
Following parent with do_update
Successfully followed parent
r315 = 3b92c4885b9d6f60241533dd99fa5023eebb1c64 (tags/0.1)
r316 = 02e3b828b4f11c5fab9d10d85076a8ff209afa00 (trunk)
[...]
However, it seems I just get the info about the revisions (shows up
nicely in gitk) but no content at all. The config file reads:
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[svn-remote "svn"]
url = https://svn.foo.bar/a/b/c/trunk
fetch = :refs/remotes/git-svn
If I instead do:
$ git svn clone https://svn.foo.bar/a/b/c/trunk
Initialized empty Git repository in .git/
W: Ignoring error from SVN, path probably does not exist: (175007):
HTTP Path Not Found: REPORT request failed on
'/a/!svn/bc/100/b/c/trunk': '/a/!svn/bc/100/b/c/trunk' path not found
W: +empty_dir: source
W: +empty_dir: test
r306 = 83f0c10b988a8f1e77a3f354126c52f0cfdecf76 (git-svn)
A resources/extensions/circuitparts.drext
[...]
r309 = 3a386e9b985c419d129461acbf24978795b36b96 (git-svn)
A tools/source/physical.cc
[...]
I get all content in trunk and their revision history, but as expected
no tag info. The config file reads:
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[svn-remote "svn"]
url = https://svn.foo.bar/a
fetch = b/c/trunk:refs/remotes/trunk
branches = b/c/branches/*:refs/remotes/*
tags = b/c/tags/*:refs/remotes/tags/*
Removing trunk from the URL yields all the files in trunk, tags, and
branches with a similar output.
Please, let me know if information is missing. I'd be very thankful
for any help that I can get to sort this out.
/Gustaf
^ permalink raw reply
* Re: How do we import patches from non-git sources?
From: Linus Torvalds @ 2007-05-24 21:45 UTC (permalink / raw)
To: Marc Singer; +Cc: git
In-Reply-To: <1180017010.21181.2.camel@zealous.synapsedev.com>
On Thu, 24 May 2007, Marc Singer wrote:
>
> Is there a way to import patches that did not come from git? Remember
> that we'd like to include the functionality of git-am that adds new
> files to the index.
The normal thing to do is
git apply --index <patchfile>
which will apply a patch _and_ update the index, so that you can then just
do a simple
git commit -m "my message goes here" --author "the author goes here"
to create the commit, new (or deleted) files and all.
Of course, the reason the "normal" patch format is an email, and not
just a bare patch, is that an email contains so much more: it contains not
just the patch, but the authorship information and the commit message.
So basically:
- no, you _cannot_ just "commit" a patch, since a patch on its own
doesn't contain the required information to be a real commit.
- but yes, you can obviously _apply_ a patch, and then commit it once you
add the proper information, but that does require more information than
just the patch itself includes.
So hopefully that clarified things.
Linus
^ permalink raw reply
* Re: Problem using git svn clone
From: Seth Falcon @ 2007-05-24 21:46 UTC (permalink / raw)
To: Gustaf Hendeby; +Cc: git
In-Reply-To: <bf7b2dda0705241435t1563008o3c47607343a34a56@mail.gmail.com>
"Gustaf Hendeby" <hendeby@gmail.com> writes:
> Hi!
>
> I have a problem with git-svn, which I hope the list can shed some
> light over. I'm quite new to git (but so far I really like it, great
> job guys!) and I never use SVN, so don't really know what information
> may be needed to be able to help me on the right track. I hope the
> below problem description is appropriate.
Although you've described in detail what you did, it isn't clear to me
what the problem is...
> The situation is as follows:
>
> I plan to use git and git svn to track a module in the middle of a SVN
> repository (which I have no control over) where I don't have read
> access more than to this specific module and the base directory, ie
> the layout is similar to this
> https://svn.foo.bar/a/b/c/trunk
> https://svn.foo.bar/a/b/c/tags
> https://svn.foo.bar/a/b/c/branches
> and I have read access to https://svm.foo.bar/a,
> https://svn.foo.bar/a/b/c and below, but nothing else.
>
> What I want to do is clone the whole thing, including tags and
> branches. I expected to be able to use the following command to do
> this (git v1.5.2):
>
> $ git svn clone https://svn.foo.bar/a/b/c -T trunk -t tags -b branches
>
> Initialized empty Git repository in .git/
> Using higher level of URL: https://svn.foo.bar/a/b/c => https://svn.foo.bar/a
>
> W: Ignoring error from SVN, path probably does not exist: (175002): RA
> layer request failed: REPORT request failed on '/a/!svn/bc/100':
> REPORT of '/a/!svn/bc/100': Could not read chunk size: Secure
> connection truncated (https://svn.foo.bar)
> r306 = 83f0c10b988a8f1e77a3f354126c52f0cfdecf76 (trunk)
> [...]
> Found possible branch point: https://svn.foo.bar/a/b/c/trunk =>
> https://svn.foo.bar/a/b/c/tags/0.1, 314
> Found branch parent: (tags/0.1) ad6a2361d0e69d6288ef226bb335bb4bf3bdd12e
> Following parent with do_update
> Successfully followed parent
> r315 = 3b92c4885b9d6f60241533dd99fa5023eebb1c64 (tags/0.1)
> r316 = 02e3b828b4f11c5fab9d10d85076a8ff209afa00 (trunk)
> [...]
>
> However, it seems I just get the info about the revisions (shows up
> nicely in gitk) but no content at all. The config file reads:
>
> [core]
> repositoryformatversion = 0
> filemode = true
> bare = false
> logallrefupdates = true
> [svn-remote "svn"]
> url = https://svn.foo.bar/a/b/c/trunk
> fetch = :refs/remotes/git-svn
>
>
> If I instead do:
>
> $ git svn clone https://svn.foo.bar/a/b/c/trunk
> Initialized empty Git repository in .git/
> W: Ignoring error from SVN, path probably does not exist: (175007):
> HTTP Path Not Found: REPORT request failed on
> '/a/!svn/bc/100/b/c/trunk': '/a/!svn/bc/100/b/c/trunk' path not found
> W: +empty_dir: source
> W: +empty_dir: test
> r306 = 83f0c10b988a8f1e77a3f354126c52f0cfdecf76 (git-svn)
> A resources/extensions/circuitparts.drext
> [...]
> r309 = 3a386e9b985c419d129461acbf24978795b36b96 (git-svn)
> A tools/source/physical.cc
> [...]
>
> I get all content in trunk and their revision history, but as expected
> no tag info. The config file reads:
>
> [core]
> repositoryformatversion = 0
> filemode = true
> bare = false
> logallrefupdates = true
> [svn-remote "svn"]
> url = https://svn.foo.bar/a
> fetch = b/c/trunk:refs/remotes/trunk
> branches = b/c/branches/*:refs/remotes/*
> tags = b/c/tags/*:refs/remotes/tags/*
That looks like what you want. Isn't it?
With that config, I would expect:
git svn fetch
pull updates for trunk and all branches and tags from the svn
server.
git checkout -b trunk remotes/git-svn
checkout the current trunk
git svn rebase
fetch and rebase trunk or current branch. git svn should determine
the right git branch from refs/remotes to rebase against.
Does that help any?
+ seth
--
Seth Falcon | Computational Biology | Fred Hutchinson Cancer Research Center
http://bioconductor.org
^ permalink raw reply
* [PATCH] Teach mailsplit about Maildir's
From: Fernando J. Pereda @ 2007-05-24 22:15 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Git Mailing List
In-Reply-To: <7v8xbevv41.fsf@assigned-by-dhcp.cox.net>
Signed-off-by: Fernando J. Pereda <ferdy@gentoo.org>
---
I built the documentation again, and formatting looks ok with that
blank line there. I think I fixed those declarations after
statements too.
Documentation/git-am.txt | 8 ++-
Documentation/git-mailsplit.txt | 13 +++-
builtin-mailsplit.c | 140 ++++++++++++++++++++++++++++++++-------
builtin.h | 2 +-
4 files changed, 133 insertions(+), 30 deletions(-)
diff --git a/Documentation/git-am.txt b/Documentation/git-am.txt
index ba79773..25cf84a 100644
--- a/Documentation/git-am.txt
+++ b/Documentation/git-am.txt
@@ -12,7 +12,8 @@ SYNOPSIS
'git-am' [--signoff] [--dotest=<dir>] [--keep] [--utf8 | --no-utf8]
[--3way] [--interactive] [--binary]
[--whitespace=<option>] [-C<n>] [-p<n>]
- <mbox>...
+ <mbox>|<Maildir>...
+
'git-am' [--skip | --resolved]
DESCRIPTION
@@ -23,9 +24,10 @@ current branch.
OPTIONS
-------
-<mbox>...::
+<mbox>|<Maildir>...::
The list of mailbox files to read patches from. If you do not
- supply this argument, reads from the standard input.
+ supply this argument, reads from the standard input. If you supply
+ directories, they'll be treated as Maildirs.
-s, --signoff::
Add `Signed-off-by:` line to the commit message, using
diff --git a/Documentation/git-mailsplit.txt b/Documentation/git-mailsplit.txt
index c11d6a5..abb0903 100644
--- a/Documentation/git-mailsplit.txt
+++ b/Documentation/git-mailsplit.txt
@@ -7,12 +7,15 @@ git-mailsplit - Simple UNIX mbox splitter program
SYNOPSIS
--------
-'git-mailsplit' [-b] [-f<nn>] [-d<prec>] -o<directory> [--] [<mbox>...]
+'git-mailsplit' [-b] [-f<nn>] [-d<prec>] -o<directory> [--] [<mbox>|<Maildir>...]
DESCRIPTION
-----------
-Splits a mbox file into a list of files: "0001" "0002" .. in the specified
-directory so you can process them further from there.
+Splits a mbox file or a Maildir into a list of files: "0001" "0002" .. in the
+specified directory so you can process them further from there.
+
+IMPORTANT: Maildir splitting relies upon filenames being sorted to output
+patches in the correct order.
OPTIONS
-------
@@ -20,6 +23,10 @@ OPTIONS
Mbox file to split. If not given, the mbox is read from
the standard input.
+<Maildir>::
+ Root of the Maildir to split. This directory should contain the cur, tmp
+ and new subdirectories.
+
<directory>::
Directory in which to place the individual messages.
diff --git a/builtin-mailsplit.c b/builtin-mailsplit.c
index 3bca855..97ae004 100644
--- a/builtin-mailsplit.c
+++ b/builtin-mailsplit.c
@@ -6,9 +6,10 @@
*/
#include "cache.h"
#include "builtin.h"
+#include "path-list.h"
static const char git_mailsplit_usage[] =
-"git-mailsplit [-d<prec>] [-f<n>] [-b] -o<directory> <mbox>...";
+"git-mailsplit [-d<prec>] [-f<n>] [-b] -o<directory> <mbox>|<Maildir>...";
static int is_from_line(const char *line, int len)
{
@@ -96,44 +97,107 @@ static int split_one(FILE *mbox, const char *name, int allow_bare)
exit(1);
}
-int split_mbox(const char **mbox, const char *dir, int allow_bare, int nr_prec, int skip)
+static int populate_maildir_list(struct path_list *list, const char *path)
{
- char *name = xmalloc(strlen(dir) + 2 + 3 * sizeof(skip));
+ DIR *dir;
+ struct dirent *dent;
+
+ if ((dir = opendir(path)) == NULL) {
+ error("cannot opendir %s (%s)", path, strerror(errno));
+ return -1;
+ }
+
+ while ((dent = readdir(dir)) != NULL) {
+ if (dent->d_name[0] == '.')
+ continue;
+ path_list_insert(dent->d_name, list);
+ }
+
+ closedir(dir);
+
+ return 0;
+}
+
+static int split_maildir(const char *maildir, const char *dir,
+ int nr_prec, int skip)
+{
+ char file[PATH_MAX];
+ char curdir[PATH_MAX];
+ char name[PATH_MAX];
int ret = -1;
+ int i;
+ struct path_list list = {NULL, 0, 0, 1};
- while (*mbox) {
- const char *file = *mbox++;
- FILE *f = !strcmp(file, "-") ? stdin : fopen(file, "r");
- int file_done = 0;
+ snprintf(curdir, sizeof(curdir), "%s/cur", maildir);
+ if (populate_maildir_list(&list, curdir) < 0)
+ goto out;
- if ( !f ) {
- error("cannot open mbox %s", file);
+ for (i = 0; i < list.nr; i++) {
+ FILE *f;
+ snprintf(file, sizeof(file), "%s/%s", curdir, list.items[i].path);
+ f = fopen(file, "r");
+ if (!f) {
+ error("cannot open mail %s (%s)", file, strerror(errno));
goto out;
}
if (fgets(buf, sizeof(buf), f) == NULL) {
- if (f == stdin)
- break; /* empty stdin is OK */
- error("cannot read mbox %s", file);
+ error("cannot read mail %s (%s)", file, strerror(errno));
goto out;
}
- while (!file_done) {
- sprintf(name, "%s/%0*d", dir, nr_prec, ++skip);
- file_done = split_one(f, name, allow_bare);
+ sprintf(name, "%s/%0*d", dir, nr_prec, ++skip);
+ split_one(f, name, 1);
+
+ fclose(f);
+ }
+
+ path_list_clear(&list, 1);
+
+ ret = skip;
+out:
+ return ret;
+}
+
+int split_mbox(const char *file, const char *dir, int allow_bare,
+ int nr_prec, int skip)
+{
+ char name[PATH_MAX];
+ int ret = -1;
+
+ FILE *f = !strcmp(file, "-") ? stdin : fopen(file, "r");
+ int file_done = 0;
+
+ if (!f) {
+ error("cannot open mbox %s", file);
+ goto out;
+ }
+
+ if (fgets(buf, sizeof(buf), f) == NULL) {
+ /* empty stdin is OK */
+ if (f != stdin) {
+ error("cannot read mbox %s", file);
+ goto out;
}
+ file_done = 1;
+ }
- if (f != stdin)
- fclose(f);
+ while (!file_done) {
+ sprintf(name, "%s/%0*d", dir, nr_prec, ++skip);
+ file_done = split_one(f, name, allow_bare);
}
+
+ if (f != stdin)
+ fclose(f);
+
ret = skip;
out:
- free(name);
return ret;
}
+
int cmd_mailsplit(int argc, const char **argv, const char *prefix)
{
- int nr = 0, nr_prec = 4, ret;
+ int nr = 0, nr_prec = 4, num = 0;
int allow_bare = 0;
const char *dir = NULL;
const char **argp;
@@ -186,9 +250,39 @@ int cmd_mailsplit(int argc, const char **argv, const char *prefix)
argp = stdin_only;
}
- ret = split_mbox(argp, dir, allow_bare, nr_prec, nr);
- if (ret != -1)
- printf("%d\n", ret);
+ while (*argp) {
+ const char *arg = *argp++;
+ struct stat argstat;
+ int ret = 0;
+
+ if (arg[0] == '-' && arg[1] == 0) {
+ ret = split_mbox(arg, dir, allow_bare, nr_prec, nr);
+ if (ret < 0) {
+ error("cannot split patches from stdin");
+ return 1;
+ }
+ num += ret;
+ continue;
+ }
+
+ if (stat(arg, &argstat) == -1) {
+ error("cannot stat %s (%s)", arg, strerror(errno));
+ return 1;
+ }
+
+ if (S_ISDIR(argstat.st_mode))
+ ret = split_maildir(arg, dir, nr_prec, nr);
+ else
+ ret = split_mbox(arg, dir, allow_bare, nr_prec, nr);
+
+ if (ret < 0) {
+ error("cannot split patches from %s", arg);
+ return 1;
+ }
+ num += ret;
+ }
+
+ printf("%d\n", num);
- return ret == -1;
+ return 0;
}
diff --git a/builtin.h b/builtin.h
index d3f3a74..39290d1 100644
--- a/builtin.h
+++ b/builtin.h
@@ -8,7 +8,7 @@ extern const char git_usage_string[];
extern void help_unknown_cmd(const char *cmd);
extern int mailinfo(FILE *in, FILE *out, int ks, const char *encoding, const char *msg, const char *patch);
-extern int split_mbox(const char **mbox, const char *dir, int allow_bare, int nr_prec, int skip);
+extern int split_mbox(const char *file, const char *dir, int allow_bare, int nr_prec, int skip);
extern void stripspace(FILE *in, FILE *out);
extern int write_tree(unsigned char *sha1, int missing_ok, const char *prefix);
extern void prune_packed_objects(int);
--
1.5.2
^ permalink raw reply related
* t9107-git-svn-migrate.sh fails
From: Johannes Schindelin @ 2007-05-24 22:18 UTC (permalink / raw)
To: git
Hi,
I recently wanted to use git-svn, and installed the svn perl bindings.
Since then, t9107 is failing:
-- snip --
[...]
Rebuilding .git/svn/git-svn/.rev_db.f944dda1-3e31-0410-9475-8f268450faf6
...
r1 = b9b82a419abdbb54f51a41bc8a3118b28c791ac1
Done rebuilding
.git/svn/git-svn/.rev_db.f944dda1-3e31-0410-9475-8f268450faf6
diff --git
a/home/gene099/my/git/t/trash/.git/svn/trunk/.rev_db.f944dda1-3e31-0410-9475-8f268450faf6
b/home/gene099/my/git/t/trash/.git/svn/trunk/.rev_db
index d3f1b6e..01d8afd 100644
---
a/home/gene099/my/git/t/trash/.git/svn/trunk/.rev_db.f944dda1-3e31-0410-9475-8f268450faf6
+++ b/home/gene099/my/git/t/trash/.git/svn/trunk/.rev_db
@@ -1,2 +1 @@
-0000000000000000000000000000000000000000
-6aa651a66730888e854a8de54199d62ffa402739
+.rev_db.f944dda1-3e31-0410-9475-8f268450faf6
\ No newline at end of file
* FAIL 7: .rev_db auto-converted to .rev_db.UUID
git-svn fetch -i trunk &&
[...]
-- snap --
Usually I try to fix things like this myself, but I really have to get
some dinner now. Besides, other people than me seem to be way more clever
with perl code.
Anybody knows how to fix this?
Ciao,
Dscho
^ permalink raw reply
* [PATCH] sha1_file.c:rearrange_packed_git() should consider packs' object sizes
From: Dana How @ 2007-05-24 22:20 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Git Mailing List, danahow
Shawn O. Pearce wrote:
> We might be able to fix this by altering the sort_pack function
> in sha1_file.c to not only order by mtime, but also by the ratio
> of the size of the .pack to the number of objects stored in it.
> Any packfile with a high size/object ratio is likely to be what
> Dana has been calling a "metadata" pack, holding things like tags,
> commits, trees and small blobs. Its these packfiles that we want
> to search first, as they are the most likely to be accessed.
>
> By pushing the megablob packs to the end of our packed_git search
> list we won't tend to scan their indexes, as most of our objects
> will be found earlier in the search list. Hence we will generally
> avoid any costs associated with their indexes.
So change the sort keys in rearrange_packed_git()/sort_pack() to
local then alternate,
increasing "deviation", <== NEW
decreasing mtime (new then old)
Each packfile has a "rank", which is the log10 of its average
stored object size. "Deviation" is the number of standard deviations
this number exceeds its mean, rounded down and clipped at zero.
Deviation should be 0 in normal use, and positive for packfiles
with significant populations of huge blobs.
This definition of deviation is intended to override mtime
only when sufficiently significant. Putting mtime before deviation
in the sort key list would cause deviation to be irrelevant.
Signed-off-by: Dana L. How <danahow@gmail.com>
---
Makefile | 2 +-
cache.h | 1 +
sha1_file.c | 26 +++++++++++++++++++++++++-
3 files changed, 27 insertions(+), 2 deletions(-)
diff --git a/Makefile b/Makefile
index 29243c6..45f0a52 100644
--- a/Makefile
+++ b/Makefile
@@ -383,7 +383,7 @@ BUILTIN_OBJS = \
builtin-pack-refs.o
GITLIBS = $(LIB_FILE) $(XDIFF_LIB)
-EXTLIBS = -lz
+EXTLIBS = -lz -lm
#
# Platform specific tweaks
diff --git a/cache.h b/cache.h
index cd875bc..630cd89 100644
--- a/cache.h
+++ b/cache.h
@@ -437,6 +437,7 @@ extern struct packed_git {
uint32_t num_objects;
int index_version;
time_t mtime;
+ int deviation;
int pack_fd;
int pack_local;
unsigned char sha1[20];
diff --git a/sha1_file.c b/sha1_file.c
index 12d2ef2..1565d91 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -6,6 +6,7 @@
* This handles basic git sha1 object files - packing, unpacking,
* creation etc.
*/
+#include <math.h>
#include "cache.h"
#include "delta.h"
#include "pack.h"
@@ -869,6 +870,14 @@ static int sort_pack(const void *a_, const void *b_)
return -st;
/*
+ * Packs with large "deviation" should be searched last.
+ * Such packs have significantly larger blobs.
+ */
+ st = a->deviation - b->deviation;
+ if (st)
+ return st;
+
+ /*
* Younger packs tend to contain more recent objects,
* and more recent objects tend to get accessed more
* often.
@@ -884,6 +893,7 @@ static void rearrange_packed_git(void)
{
struct packed_git **ary, *p;
int i, n;
+ float sum1 = 0, sum2 = 0, mean, sigma;
for (n = 0, p = packed_git; p; p = p->next)
n++;
@@ -892,8 +902,22 @@ static void rearrange_packed_git(void)
/* prepare an array of packed_git for easier sorting */
ary = xcalloc(n, sizeof(struct packed_git *));
- for (n = 0, p = packed_git; p; p = p->next)
+ for (n = 0, p = packed_git; p; p = p->next) {
+ /* this is the log10 of the average object size (almost) */
+ float rank = log10((p->pack_size + 1.0) / (p->num_objects + 1.0));
+ sum1 += rank;
+ sum2 += rank * rank;
ary[n++] = p;
+ }
+ mean = sum1 / n;
+ sigma = sqrt(sum2 / n - mean * mean);
+ for (p = packed_git; p; p = p->next) {
+ float rank = log10((p->pack_size + 1.0) / (p->num_objects + 1.0));
+ int deviation = sigma > 0 ? floor((rank - mean) / sigma) : 0;
+ if ( deviation < 0 )
+ deviation = 0;
+ p->deviation = deviation;
+ }
qsort(ary, n, sizeof(struct packed_git *), sort_pack);
--
1.5.2.762.gd8c6-dirty
^ permalink raw reply related
* [PATCH] Split packs from git-repack should have descending timestamps
From: Dana How @ 2007-05-24 22:33 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Git Mailing List, danahow
If git-repack produces multiple split packs because
--max-pack-size was in effect, the first pack written
should have the latest timestamp because:
(1) sha1_file.c:rearrange_packed_git() puts more recent
pack files at the beginning of the search list; and
(2) the most recent objects are written out first
while packing.
This is based on next rather than master to avoid merge
conflicts with changes already in git-repack.sh due to
the --max-pack-size patchset.
Signed-off-by: Dana L. How <danahow@gmail.com>
---
git-repack.sh | 5 +++++
1 files changed, 5 insertions(+), 0 deletions(-)
diff --git a/git-repack.sh b/git-repack.sh
index 4ea6e5b..953de4a 100755
--- a/git-repack.sh
+++ b/git-repack.sh
@@ -68,6 +68,7 @@ names=$(git-pack-objects --non-empty --all --reflog $args </dev/null "$PACKTMP")
if [ -z "$names" ]; then
echo Nothing new to pack.
fi
+restamp=
for name in $names ; do
chmod a-w "$PACKTMP-$name.pack"
chmod a-w "$PACKTMP-$name.idx"
@@ -94,8 +95,12 @@ for name in $names ; do
exit 1
}
rm -f "$PACKDIR/old-pack-$name.pack" "$PACKDIR/old-pack-$name.idx"
+ restamp="$PACKDIR/pack-$name.pack $restamp"
done
+# for split packs, the first created should have most recent timestamp
+for file in $restamp ; do touch $file ; sleep 2; done &
+
if test "$remove_redundant" = t
then
# We know $existing are all redundant.
--
1.5.2.762.gd8c6-dirty
^ permalink raw reply related
* Fwd: Problem using git svn clone
From: Gustaf Hendeby @ 2007-05-24 22:48 UTC (permalink / raw)
Cc: git
In-Reply-To: <bf7b2dda0705241539i56f1b5b3kcd001e871e8688ef@mail.gmail.com>
Apparently gmail does some HTMLing on reply unless you watch out, so
this didn't reach the list.
/Gustaf
---------- Forwarded message ----------
From: Gustaf Hendeby <hendeby@gmail.com>
Date: May 25, 2007 12:39 AM
Subject: Re: Problem using git svn clone
To: Seth Falcon <sethfalcon@gmail.com>
Cc: git@vger.kernel.org
On 5/24/07, Seth Falcon <sethfalcon@gmail.com> wrote:
> "Gustaf Hendeby" <hendeby@gmail.com> writes:
>
> > I have a problem with git-svn, which I hope the list can shed some
> > light over. I'm quite new to git (but so far I really like it, great
> > job guys!) and I never use SVN, so don't really know what information
> > may be needed to be able to help me on the right track. I hope the
> > below problem description is appropriate.
>
> Although you've described in detail what you did, it isn't clear to me
> what the problem is...
Let me clarify. When trying to check out the module with tags, the
first example I only get the revisions, no content at all! I have
added some more info on this below.
The second case works as I expected it, I included it as reference,
and to show that it worked (probably not the smartest thing to do I
guess). Hence, probably just the first example is of interest.
> > The situation is as follows:
> >
> > I plan to use git and git svn to track a module in the middle of a SVN
> > repository (which I have no control over) where I don't have read
> > access more than to this specific module and the base directory, ie
> > the layout is similar to this
> > https://svn.foo.bar/a/b/c/trunk
> > https://svn.foo.bar/a/b/c/tags
> > https://svn.foo.bar/a/b/c/branches
> > and I have read access to https://svm.foo.bar/a,
> > https://svn.foo.bar/a/b/c and below, but nothing else.
> >
> > What I want to do is clone the whole thing, including tags and
> > branches. I expected to be able to use the following command to do
> > this (git v1.5.2):
> >
> > $ git svn clone https://svn.foo.bar/a/b/c -T trunk -t tags -b branches
> >
> > Initialized empty Git repository in .git/
> > Using higher level of URL: https://svn.foo.bar/a/b/c => https://svn.foo.bar/a
> >
> > W: Ignoring error from SVN, path probably does not exist: (175002): RA
> > layer request failed: REPORT request failed on '/a/!svn/bc/100':
> > REPORT of '/a/!svn/bc/100': Could not read chunk size: Secure
> > connection truncated (https://svn.foo.bar)
> > r306 = 83f0c10b988a8f1e77a3f354126c52f0cfdecf76 (trunk)
> > [...]
> > Found possible branch point: https://svn.foo.bar/a/b/c/trunk =>
> > https://svn.foo.bar/a/b/c/tags/0.1 , 314
> > Found branch parent: (tags/0.1) ad6a2361d0e69d6288ef226bb335bb4bf3bdd12e
> > Following parent with do_update
> > Successfully followed parent
> > r315 = 3b92c4885b9d6f60241533dd99fa5023eebb1c64 (tags/0.1)
> > r316 = 02e3b828b4f11c5fab9d10d85076a8ff209afa00 (trunk)
> > [...]
> >
> > However, it seems I just get the info about the revisions (shows up
> > nicely in gitk) but no content at all. The config file reads:
I'm really sorry, I mixed up the config files. :( So I actually get
this for the first example not the second:
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[svn-remote "svn"]
url = https://foo.bar/a
fetch = b/c/trunk:refs/remotes/trunk
branches = b/c/branches/*:refs/remotes/*
tags = b/c/tags/*:refs/remotes/tags/*
Which looks good I guess... But to be a bit more clear about what
this actually gives me in the directory I just cloned to, and this is
the problem:
$ git checkout -f master
Already on branch "master"
$ ls -a
. .. .git
$ du -hs .git/objects
186K .git/objects
There should be hundreds of files here, several MB of code. So
something has gone seriously wrong here --- or at least did not work
the way I expected it to. The .git/objects in the second example
contains 5.7 MB data. More stats:
$ git rev-list --all | wc -l
99
$ git tag -l
$ git branch -a
* master
tags/0.1
tags/0.1.1
tags/0.2.0
tags/0.2.1
trunk
No tags (I mistook some of the branches for tags in the first post),
but some branches all containing no files, though, still claiming to
be clean...
> >
> >
> > If I instead do:
> >
> > $ git svn clone https://svn.foo.bar/a/b/c/trunk
> > I get all content in trunk and their revision history, but as expected
> That looks like what you want. Isn't it?
Second example works just as expected.
I assume the listed commands below are assumed to work with the second
example? They doesn't seem to do any good to my first example.
> With that config, I would expect:
>
> git svn fetch
> pull updates for trunk and all branches and tags from the svn
> server.
>
> git checkout -b trunk remotes/git-svn
> checkout the current trunk
>
> git svn rebase
> fetch and rebase trunk or current branch. git svn should determine
> the right git branch from refs/remotes to rebase against.
>
> Does that help any?
Given the bad info I posted you did a great job, I really didn't
intend to waste the lists time. Thanks! Though, my problem still
remains. Given the new information, do you get any other ideas on
what I do wrong?
^ permalink raw reply
* Re: [PATCH] Prevent megablobs from gunking up git packs
From: Dana How @ 2007-05-24 23:29 UTC (permalink / raw)
To: Shawn O. Pearce; +Cc: Junio C Hamano, Git Mailing List, danahow
In-Reply-To: <20070524071235.GL28023@spearce.org>
On 5/24/07, Shawn O. Pearce <spearce@spearce.org> wrote:
> Junio C Hamano <junkio@cox.net> wrote:
> > "Dana How" <danahow@gmail.com> writes:
> > > We have three options in this case:
> > > (1) Drop the object (do not put it in the new pack(s)).
> > > (2) Pass the object into the new pack(s).
> > > (3) Write out the object as a new loose object.
> > > Option (1) is unacceptable. When you call git-repack -a,
> > > it blindly deletes all the non-kept packs at the end. So
> > > the megablobs would be lost.
> > Ok, I can buy that -- (1) nor (2) are unacceptable and (3) is
> > the only sane thing to do for a previously packed objects that
> > exceed the size limit.
>
> I still don't buy the idea that these megablobs shouldn't be packed.
> I understand Dana's pain here (at least a little bit, my problems
> aren't as bad as his are), but I also hate to see us run away from
> packfiles for these really sick cases just because we have some
> issues in our current packfile handling.
>
> Packfiles give us a lot of benefits:
>
> 1) less inode usage;
I agree with Geert that blowing an inode on a 100MB+ object
is no big deal.
> 2) transport can write directly to local disk;
> 3) transport can (quickly) copy from local disk;
For (2) and (3) see comments on next para plus NFS discussion.
> 4) testing for existance is *much* faster;
This is true. But I don't care about this cost if it
is only incurred on large objects which are leaf nodes
in the git "data relationship tree" (tags->commits->trees->blobs) anyway.
> 5) deltafication is possible;
Again Geert made a good argument that didn't occur to me that
you definitely DON'T want to do deltification on such large objects.
Junio recently added delta/nodelta attribute; this would be useful
to me, but unfortunately I have several continua of files, each with
the same suffix, but with largely varying sizes, so attributes won't
help me unless the name globs in .gitattributes are expanded to full
expressions similar to find(1) [i.e. include testing based on size,
perms, type], which I think would be insane.
> Now #3 is actually really important here. Don't forget that we
> *just* disabled the fancy "new loose object format". It doesn't
> exist. We can read the packfile-like loose objects, but we cannot
> write them anymore. So lets say we explode a megablob into a loose
> object, and its 800 MiB by itself. Now we have to send that object
> to a client. Yes, that's right, we must *RECOMPRESS* 800 MiB for
> no reason. Not the best choice. Maybe we shouldn't have deleted
> that packfile formatted loose object writer...
I completely agree with your argument. I do not suggest that repositories
that communicate with others via packs should use this feature.
Our repositories will communicate via alternates/NFS in one direction
and probably packs in the other. In the latter case the packs would
be generated with maxblobsize=0. See comments on next para.
> Now one argument to work around that recompression problem would
> be to NFS share out the loose objects directory, and let clients
> mount that volume and add it to their .git/objects/info/alternates
> list. But this doesn't work in the very general distributed case,
> such as me getting huge files from kernel.org. Last I checked,
> the kernel.org admins did not offer up NSF mounts. Besides, the
> round-trip latency between me and kernel.org is too large for it
> to be useful anyway over NFS. :)
The NFS case is exactly what I want to use. I want each repo to
have their own packfiles to reduce load on the central alternate,
but these local repos would not include megablobs. I do not have
as strong a feeling about whether the central alternate should
pack its megablobs or not [but I don't want to do it if it costs me
deltification for everybody], but I need a way to exclude megablobs
from getting into local packs. WIth such exclusion, git-gc/repack
is extremely quick. There is NO WAY this can be true if
several GB have to be copied around, which again comes from Geert.
I think this conversation does suggest one alteration to my patch:
as submitted it writes out a loose object if the object is packed
and has no loose object. It should really do this only if the
object is packed LOCALLY and has no loose object.
> So I think this "explode out megablobs" is a bad idea. Its violating
> other things that make us fast, like #3's being able to reuse large
> parts of an existing packfile during transfer.
As I said, your true argument doesn't apply in my case.
> Dana pointed out the megablobs make access slower because their
> packfile indexes must still be searched to locate a commit; but if
> the megablob packfile(s) contain only blobs then there is no value
> in looking at those packfiles.
>
> We might be able to fix this by altering the sort_pack function
> in sha1_file.c to not only order by mtime, but also by the ratio
> of the size of the .pack to the number of objects stored in it.
> Any packfile with a high size/object ratio is likely to be what
> Dana has been calling a "metadata" pack, holding things like tags,
> commits, trees and small blobs. Its these packfiles that we want
> to search first, as they are the most likely to be accessed.
>
> By pushing the megablob packs to the end of our packed_git search
> list we won't tend to scan their indexes, as most of our objects
> will be found earlier in the search list. Hence we will generally
> avoid any costs associated with their indexes.
Good argument and I submitted a patch to do this.
Let's see who chokes on the floating arithmetic ;-)
> Huge packfiles probably should be scheduled for keeping with a .keep
> automatically. We probably should teach pack-objects to generate a
> .keep file if the resulting .pack was over a certain size threshold
> (say 1.5 GiB by default) and teach git-repack to rename the .keep
> file as it also renames the .idx and .pack.
I have experimented with this, and Jakub Narebski made related
suggestions. I find this quite hokey, but even if I do it in my central
alternate, I still do not want to be packing megablobs in individual user's
repos EVER, and need some way to exclude them.
> Better that we degrade gracefully when faced with massive inputs
> than we do something stupid by default and make the poor user pay
> for their mistake of not throughly reading plumbing documentation
> before use.
Unnecessary copying of several GB is not degrading gracefully in my view.
In fact having repack.maxblobsize = 2000 (K) in the default "config"
strikes me as degrading much more gracefully than what the code
would currently do.
This silly patch took my packfile sets from 12GB+ to 13MB,
and it's difficult to describe how relieved I now feel.
It's also difficult for me to believe that a setup that treats 12GB
(almost) equally is going to be as efficient as one which concentrates on 13MB.
That's three orders of magnitude, a point I've made before.
But I think you have an understandable motivation:
you want packfiles to be as good as possible, and any escape
mechanism from them decreases the motivation to "fix" packing.
Now I agree with this, which is why I just submitted some other patches,
but I don't share your goal of the universality of all packfiles --
just the ones used for transport. Don't your packv4 plans introduce
mods which won't be used for transport as well?
> Now I would agree that we should punt on deltification of anything
> that is just too large, and let the user decide what too large means,
> and default it around 500 or 1024 MiB. But I would still stuff it
> into a packfile.
>
> Maybe it still makes sense to have a limit on the maximum size of a
> loose object to pack, but I think that's only to avoid the sick case
> of a very simple no-argument "git repack" taking a long while because
> there's 8 loose objects and 6 of them are 900 MiB image files.
Perhaps we _will_ make progress if we all agree to describe
my situation as "sick" ;-) . In this paragraph you seem to agree that
there is some argument for keeping megablobs from _entering_ packs?
One reason I like my patch is because I do view megablobs
as perverting the system, and just keeping them out of the optimized
packfile system is a big step forward.
> Once in a packfile, I'd keep it there, even if the user decreases
> the threshold, as the advantages of it being in the packfile outweigh
> the disadvantages of it being in the packfile. And there's like no
> advantage to being loose once packed.
To (almost) follow this suggestion I would need git-fast-import to respect
repack.maxblobsize as well. Is that OK with you?
I previously offered to Junio that the "write loose object" thing
could be restricted: it would only happen if -f were supplied to
git-repack, otherwise the bad blob would pass through to the new pack.
Does this "reduction in strength" make this feature more palatable to you?
If the stats on a repo change significantly, "write loose object"
becomes more important if you have to make a significant reduction
to repack.maxblobsize (or specify it for the first time).
I don't agree that once in a packfile, a blob should stay there.
Its presence is degrading access to "normal" blobs co-habiting with it.
So you will want to repack to separate them in different packs
(the various .keep-related ideas) or just write them out loose.
To conclude:
the patch wrote out a new loose object when it was previously
packed and is larger then repack.maxblobsize.
I could change this to only happen when the object is
(1) packed AND
(2) locally packed AND
(3) -f/--no-object-reuse was specified to git-repack/git-pack-objects.
The previous behavior that a megablob never _enters_ the pack
would remain unchanged.
This more restrictive behavior would be sufficient for me,
and I think I *need* it at least in the users' repositories
in an NFS/alternates setup.
What do you think?
Thanks,
--
Dana L. How danahow@gmail.com +1 650 804 5991 cell
^ permalink raw reply
* Re: [PATCH] Split packs from git-repack should have descending timestamps
From: Shawn O. Pearce @ 2007-05-25 0:46 UTC (permalink / raw)
To: Dana How; +Cc: Junio C Hamano, Git Mailing List
In-Reply-To: <465612CE.4080605@gmail.com>
Dana How <danahow@gmail.com> wrote:
>
> If git-repack produces multiple split packs because
> --max-pack-size was in effect, the first pack written
> should have the latest timestamp because:
> (1) sha1_file.c:rearrange_packed_git() puts more recent
> pack files at the beginning of the search list; and
> (2) the most recent objects are written out first
> while packing.
>
> This is based on next rather than master to avoid merge
> conflicts with changes already in git-repack.sh due to
> the --max-pack-size patchset.
Ack. Given our mtime based sorting routine, even without your
recent patch to improve it, I think we definately want this type
of behavior built into git-repack.sh. Good follow-on to your
--max-pack-size series.
--
Shawn.
^ permalink raw reply
* Re: [PATCH 11/22] entry.c: optionally checkout submodules
From: Shawn O. Pearce @ 2007-05-25 0:49 UTC (permalink / raw)
To: Martin Waitz; +Cc: Alex Riesen, skimo@liacs.nl, git, Junio C Hamano
In-Reply-To: <20070524162106.GN5412@admingilde.org>
Martin Waitz <tali@admingilde.org> wrote:
>
> On Thu, May 24, 2007 at 03:18:19AM -0400, Shawn O. Pearce wrote:
> > I'm actually really unhappy with our !istty(2) means disable
> > progress thing. git-gui knows how to read and show the progress
> > meters, but nobody prints them anymore as 2 is a pipe. I have the
> > same problem with a Java build tool that sometimes starts up an
> > expensive Git operation (like a clone over SSH of a 60+ MiB project).
> >
> > I've been considering adding a GIT_ISTTY environment variable to
> > forcefully override the istty result, just to get the progress
> > meters turned back on...
>
> or perhaps introduce GIT_PROGRESS to name a filedescriptor which then
> _only_ gets all the progress information, in a format easily parseable
> by other tools?
Unfortunately that isn't Tcl friendly, and its *really* not Tcl
on Windows friendly as there we have a difficult time passing
environment variables from Tcl down into Cygwin forked processes.
That problem appears to be a glitch in how Cygwin's Tcl happens to be
implemented on Windows; its actually more a native Tcl than a Cygwin
process, especially when it comes to the builtin Tcl "exec" command.
But its a good idea. Its just hard for me to take advantage of
it up in git-gui. ;-)
--
Shawn.
^ permalink raw reply
* Re: [PATCH] Prevent megablobs from gunking up git packs
From: Shawn O. Pearce @ 2007-05-25 0:55 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: david, Junio C Hamano, Dana How, Git Mailing List
In-Reply-To: <Pine.LNX.4.64.0705241828160.4648@racer.site>
Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> On Thu, 24 May 2007, david@lang.hm wrote:
> > On Thu, 24 May 2007, Shawn O. Pearce wrote:
> >
> > > Now #3 is actually really important here. Don't forget that we
> > > *just* disabled the fancy "new loose object format". It doesn't
> > > exist. We can read the packfile-like loose objects, but we cannot
> > > write them anymore. So lets say we explode a megablob into a loose
> > > object, and its 800 MiB by itself. Now we have to send that object
> > > to a client. Yes, that's right, we must *RECOMPRESS* 800 MiB for
> > > no reason. Not the best choice. Maybe we shouldn't have deleted
> > > that packfile formatted loose object writer...
> >
> > when did the object store get changed so that loose objects aren't
> > compressed?
>
> That never happened. But we had a different file format for loose objects,
> which was meant to make it easier to copy as-is into a pack. That file
> format went away, since it was not as useful as we hoped.
That "different file format" thing was added exactly for this type
of problem. Someone added a bunch of large blobs to their repository
and then spent a lot of time decompressing and recompressing them
during their next repack.
The reason that recompress must happen is the deflate stream in a
standard (aka legacy) loose object contains both the Git object
header and the raw data; in a packfile the Git object header is
stored external from the deflate stream. The "different file format"
used the packfile format, allowing us to store the Git object header
external from the deflate stream. That meant we could just copy
the raw bytes as-is from the loose object into the packfile.
So we still store loose objects compressed, its just that we can
no longer create loose objects that can be copied directly into
a packfile without recompression. And that is sort of Dana's
problem here. OK, not entirely, but whatever.
--
Shawn.
^ permalink raw reply
* Re: [PATCH] Split packs from git-repack should have descending timestamps
From: Junio C Hamano @ 2007-05-25 1:04 UTC (permalink / raw)
To: Shawn O. Pearce; +Cc: Dana How, Git Mailing List
In-Reply-To: <20070525004610.GP28023@spearce.org>
"Shawn O. Pearce" <spearce@spearce.org> writes:
> Dana How <danahow@gmail.com> wrote:
>>
>> If git-repack produces multiple split packs because
>> --max-pack-size was in effect, the first pack written
>> should have the latest timestamp because:
>> (1) sha1_file.c:rearrange_packed_git() puts more recent
>> pack files at the beginning of the search list; and
>> (2) the most recent objects are written out first
>> while packing.
>>
>> This is based on next rather than master to avoid merge
>> conflicts with changes already in git-repack.sh due to
>> the --max-pack-size patchset.
>
> Ack. Given our mtime based sorting routine, even without your
> recent patch to improve it, I think we definately want this type
> of behavior built into git-repack.sh. Good follow-on to your
> --max-pack-size series.
Gee, I do not want to touch this, unless we can do something
about that sleep 2, even if you have & at the end (actually,
especially because you have that -- it makes me worried).
At the minimum, I think you do not have to restamp at all if the
result is a single pack (i.e. the usual case), like so:
case "$restamp" in
?*' '?*)
# we have more than one.
# for split packs, the first created should have most recent timestamp
for file in $restamp ; do touch $file; sleep 2; done &
;;
esac
Come to think of it, can't you do this "re-touching" business at
the end of pack-objects without sleeping? You could keep track
of the names of the packs you produced, and if you have produced
5, like so:
1
2
3
4
5
you would swap timestamp of #1 and #5, #2 and #4 using stat()
and utime(), and you are done. Each of these huge packs would
take more than one second to write it out, but if that is not
the case, you could even start with timestamp of #5, subtract 1
and stamp #4, subtract 1 and stamp #3, ... You may end up using
timestamp from the past, but that would not be a problem.
And I am really hoping that the other "use object density in
reordering" patch would make this irrelevant. You would have
commit and then the rest in the normal input object stream, and
recenty ordering done by git-pack-objects should keep commits
together early in the resulting split pack, and earlier parts
that have the commits would be hopefully denser.
^ permalink raw reply
* Pulling from refs/remotes/ ?
From: Han-Wen Nienhuys @ 2007-05-25 1:12 UTC (permalink / raw)
To: git
Hi,
why can't I pull from a remote? I can do
git log REMOTE-BRANCH
git diff BRANCH REMOTE-BRANCH
etc. But pulling yields
[hanwen@haring foobar]$ git pull . origin/nonrandr-setup
error: no such remote ref refs/heads/origin/nonrandr-setup
Fetch failure: .
also, "Fetch failure: .", what is that supposed to mean?
--
Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen
^ permalink raw reply
* Re: Pulling from refs/remotes/ ?
From: Junio C Hamano @ 2007-05-25 1:19 UTC (permalink / raw)
To: hanwen; +Cc: git
In-Reply-To: <f35d9n$21e$1@sea.gmane.org>
Han-Wen Nienhuys <hanwen@xs4all.nl> writes:
> why can't I pull from a remote? I can do
>
> git log REMOTE-BRANCH
> git diff BRANCH REMOTE-BRANCH
>
> etc. But pulling yields
>
> [hanwen@haring foobar]$ git pull . origin/nonrandr-setup
> error: no such remote ref refs/heads/origin/nonrandr-setup
> Fetch failure: .
>
> also, "Fetch failure: .", what is that supposed to mean?
You are treating your local repository as if it is a remote
repository somewhere else, namely, '.' (current repository).
Notice "git log" and "git diff" are LOCAL operations? "git
pull" is "git fetch" which is a remote operation (i.e. it
interacts with a remote repository) followed by "git merge"
which is a local operation to merge in what was fetched or what
you already have locally.
I think you would want:
git merge REMOTE-BRANCH
e.g. "git merge origin/nonrandr-setup".
^ permalink raw reply
* Re: Pulling from refs/remotes/ ?
From: Han-Wen Nienhuys @ 2007-05-25 1:35 UTC (permalink / raw)
To: git; +Cc: git
In-Reply-To: <7v7iqxvgx8.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano escreveu:
> Han-Wen Nienhuys <hanwen@xs4all.nl> writes:
>
>> why can't I pull from a remote? I can do
>>
>> git log REMOTE-BRANCH
>> git diff BRANCH REMOTE-BRANCH
>>
>> etc. But pulling yields
>>
>> [hanwen@haring foobar]$ git pull . origin/nonrandr-setup
>> error: no such remote ref refs/heads/origin/nonrandr-setup
>> Fetch failure: .
>>
>> also, "Fetch failure: .", what is that supposed to mean?
>
> You are treating your local repository as if it is a remote
> repository somewhere else, namely, '.' (current repository).
>
> Notice "git log" and "git diff" are LOCAL operations? "git
> pull" is "git fetch" which is a remote operation (i.e. it
> interacts with a remote repository) followed by "git merge"
> which is a local operation to merge in what was fetched or what
> you already have locally.
>
> I think you would want:
>
> git merge REMOTE-BRANCH
>
> e.g. "git merge origin/nonrandr-setup".
Yes, that works, thanks. Wouldn't it be more consistent with this
reasoning to disallow
git pull . LOCAL-BRANCH
too?
I still think that "fetch failure" is a bad error message.
What information does it convey?
--
Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen
^ permalink raw reply
* [PATCH] fix memory leak in parse_object when check_sha1_signature fails
From: Carlos Rica @ 2007-05-25 1:46 UTC (permalink / raw)
To: git, Junio C Hamano
When check_sha1_signature fails, program is not terminated:
it prints an error message and returns NULL, so the
buffer returned by read_sha1_file should be freed before.
Signed-off-by: Carlos Rica <jasampler@gmail.com>
---
object.c | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/object.c b/object.c
index 78a44a6..ccd7dd7 100644
--- a/object.c
+++ b/object.c
@@ -185,6 +185,7 @@ struct object *parse_object(const unsigned char *sha1)
if (buffer) {
struct object *obj;
if (check_sha1_signature(sha1, buffer, size, typename(type)) < 0) {
+ free(buffer);
error("sha1 mismatch %s\n", sha1_to_hex(sha1));
return NULL;
}
--
1.5.0
^ permalink raw reply related
* Re: [PATCH] Make sure an autogenerated version has at least four parts
From: Sam Vilain @ 2007-05-25 1:42 UTC (permalink / raw)
To: Martin Waitz; +Cc: Junio C Hamano, git
In-Reply-To: <46563CE1.9020007@vilain.net>
I wrote:
> dpkg uses "-" in version numbers for its own uses - to delimit the
> packager's packaging version from the software version. The change I
> posted keeps original behaviour - just fills out the .0's.
>
> Perhaps the munging should go in git-describe instead?
>
> Subject: [PATCH] describe: add --levels option
...and here's the corresponding GIT-VERSION-GEN patch:
Subject: [PATCH] Make sure an autogenerated version has at least four parts
Otherwise, a custom "v1.5.2.42.gd00b" is considered newer than a
"v1.5.2.1.69.gcafe".
Contains a workaround for the chicken-and-egg problem it would
otherwise introduce.
Signed-off-by: Sam Vilain <sam@vilain.net>
---
GIT-VERSION-GEN | 3 ++-
1 files changed, 2 insertions(+), 1 deletions(-)
diff --git a/GIT-VERSION-GEN b/GIT-VERSION-GEN
index 06c360b..91e8966 100755
--- a/GIT-VERSION-GEN
+++ b/GIT-VERSION-GEN
@@ -12,7 +12,8 @@ if test -f version
then
VN=$(cat version) || VN="$DEF_VER"
elif test -d .git &&
- VN=$(git describe --abbrev=4 HEAD 2>/dev/null) &&
+ VN=$(git describe --abbrev=4 --levels=4 HEAD 2>/dev/null ||
+ git describe --abbrev=4 HEAD 2>/dev/null) &&
case "$VN" in
*$LF*) (exit 1) ;;
v[0-9]*) : happy ;;
--
1.5.2.0.45.gfea6d-dirty
^ permalink raw reply related
* Re: [PATCH] Make sure an autogenerated version has at least four parts
From: Sam Vilain @ 2007-05-25 1:33 UTC (permalink / raw)
To: Martin Waitz; +Cc: Sam Vilain, Junio C Hamano, git
In-Reply-To: <20070521073650.GV5412@admingilde.org>
Martin Waitz wrote:
> hoi :)
>
> On Mon, May 21, 2007 at 02:52:21PM +1200, Sam Vilain wrote:
>> Otherwise, a custom "v1.5.2.42.gd00b" is considered newer than a
>> "v1.5.2.1.69.gcafe".
>
> or just use git describe output without replacing "-" with "."?
>
dpkg uses "-" in version numbers for its own uses - to delimit the
packager's packaging version from the software version. The change I
posted keeps original behaviour - just fills out the .0's.
Perhaps the munging should go in git-describe instead?
Subject: [PATCH] describe: add --levels option
Some projects might want git describe to always give a result that
has a given number of version levels. ie, if you say --levels=4
and describe finds a name like 'v1.5.2', the result will be 'v1.5.2.0'.
This does mean that on exact tag matches, the returned version
is not a resolvable ref - but that is probably caveat emptor.
Signed-off-by: Sam Vilain <sam@vilain.net>
---
builtin-describe.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++----
t/t6120-describe.sh | 8 ++++++++
2 files changed, 54 insertions(+), 4 deletions(-)
diff --git a/builtin-describe.c b/builtin-describe.c
index 165917e..05eabc5 100644
--- a/builtin-describe.c
+++ b/builtin-describe.c
@@ -15,6 +15,7 @@ static int all; /* Default to annotated tags only */
static int tags; /* But allow any tags if --tags is specified */
static int abbrev = DEFAULT_ABBREV;
static int max_candidates = 10;
+static int num_levels = 0;
struct commit_name {
int prio; /* annotated tag = 2, tag = 1, head = 0 */
@@ -134,6 +135,7 @@ static void describe(const char *arg, int last_one)
struct possible_tag all_matches[MAX_TAGS];
unsigned int match_cnt = 0, annotated_cnt = 0, cur_match;
unsigned long seen_commits = 0;
+ char* chosen;
if (get_sha1(arg, sha1))
die("Not a valid object name %s", arg);
@@ -148,8 +150,9 @@ static void describe(const char *arg, int last_one)
n = cmit->util;
if (n) {
- printf("%s\n", n->path);
- return;
+ chosen = n->path;
+ abbrev = 0;
+ goto show;
}
if (debug)
@@ -228,10 +231,44 @@ static void describe(const char *arg, int last_one)
sha1_to_hex(gave_up_on->object.sha1));
}
}
+
+ chosen = all_matches[0].name->path;
+
+ /* make the described version have the desired number of
+ * levels in it */
+ show:
+ if (num_levels) {
+ int found = 1;
+ char* idx = chosen;
+ int i;
+ while ((idx = index(idx, '.'))) {
+ found++;
+ idx++;
+ }
+ if (found > num_levels) {
+ idx = chosen;
+ for (i = 0; i < num_levels; i++) {
+ if (i)
+ idx++;
+ idx = index(idx, '.');
+ }
+ *idx = '\0';
+ }
+ else if (found < num_levels) {
+ int extra = 2 * (num_levels - found);
+ char* new = xmalloc(strlen(chosen) + extra + 1);
+ chosen = strcpy(new, chosen);
+ while (found < num_levels) {
+ strcat(chosen, ".0");
+ found++;
+ }
+ }
+ }
+
if (abbrev == 0)
- printf("%s\n", all_matches[0].name->path );
+ printf("%s\n", chosen);
else
- printf("%s-%d-g%s\n", all_matches[0].name->path,
+ printf("%s-%d-g%s\n", chosen,
all_matches[0].depth,
find_unique_abbrev(cmit->object.sha1, abbrev));
@@ -266,6 +303,11 @@ int cmd_describe(int argc, const char **argv, const char *prefix)
else if (max_candidates > MAX_TAGS)
max_candidates = MAX_TAGS;
}
+ else if (!prefixcmp(arg, "--levels=")) {
+ num_levels = strtoul(arg + 9, NULL, 10);
+ if (num_levels < 0)
+ num_levels = 0;
+ }
else
usage(describe_usage);
}
diff --git a/t/t6120-describe.sh b/t/t6120-describe.sh
index 3e9edda..0336ddd 100755
--- a/t/t6120-describe.sh
+++ b/t/t6120-describe.sh
@@ -33,6 +33,7 @@ test_expect_success setup '
test_tick &&
echo two >file && git-add file && git-commit -m second &&
+ git-tag -a -m v1.1 v1.1 &&
two=$(git-rev-parse HEAD) &&
test_tick &&
@@ -94,4 +95,11 @@ check_describe D-* --tags HEAD^^
check_describe A-* --tags HEAD^^2
check_describe B --tags HEAD^^2^
+check_describe A.0-* --tags --levels=2 HEAD
+check_describe A.0.0-* --tags --levels=3 HEAD
+check_describe v1 --tags --levels=1 v1.1
+check_describe v1.1 --tags --levels=2 v1.1
+check_describe v1.1.0 --tags --levels=3 v1.1
+check_describe v1-* --tags --levels=1 A^1
+
test_done
--
1.5.2.0.45.gfea6d-dirty
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox