* Re: Smart fetch via HTTP?
From: Matthieu Moy @ 2007-05-17 11:28 UTC (permalink / raw)
To: git
In-Reply-To: <Pine.LNX.4.64.0705170152470.6410@racer.site>
Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> Hi,
>
> On Thu, 17 May 2007, Martin Langhoff wrote:
>
>> On 5/16/07, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>> > On Wed, 16 May 2007, Martin Langhoff wrote:
>> > > Do the indexes have enough info to use them with http ranges? It'd be
>> > > chunkier than a smart protocol, but it'd still work with dumb servers.
>> > It would not be really performant, would it? Besides, not all Web servers
>> > speak HTTP/1.1...
>>
>> Performant compared to downloading a huge packfile to get 10% of it?
>> Sure! It'd probably take a few trips, and you'd end up fetching 20% of
>> the file, still better than 100%.
>
> Don't forget that those 10% probably do not do you the favour to be in
> large chunks. Chances are that _every_ _single_ wanted object is separate
> from the others.
FYI, bzr uses HTTP range requests, and the introduction of this
feature lead to significant performance improvement for them (bzr is
more dumb-protocol oriented than git is, so that's really important
there). They have this "index file+data file" system too, so you
download the full index file, and then send an HTTP range request to
get only the relevant parts of the data file.
The thing is, AAUI, they don't send N range requests to get N chunks,
but one HTTP request, requesting the N ranges at a time, and get the N
chunks a a whole (IIRC, a kind of MIME-encoded response from the
server). So, you pay the price of a longer HTTP request, but not the
price of N networks round-trips.
That's surely not as efficient as anything smart on the server, but
might really help for the cases where the server is /not/ smart.
--
Matthieu
^ permalink raw reply
* Re: [PATCH] Git.pm: Add remote_refs() git-ls-remote frontend
From: Petr Baudis @ 2007-05-17 11:30 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vk5v89ek8.fsf@assigned-by-dhcp.cox.net>
On Thu, May 17, 2007 at 07:58:31AM CEST, Junio C Hamano wrote:
> Petr Baudis <pasky@suse.cz> writes:
>
> > .... Too bad that
> >
> > git-ls-remote --heads .
> >
> > is subtly different from
> >
> > git-ls-remote . refs/heads/
> >
> > so we have to provide the interface for specifying both.
>
> I've already heard you say the above elsewhere, but I am not
> sure what you exactly mean here. Mind substantiating it a bit
> more clearly?
Because the arguments are matched like
case "/$path" in
*/$pat )
so if you have remote "refs" and refs/remote/refs/heads/x, git-ls-remote
refs/heads/ is ambiguous. OTOH, --heads always match start of the path.
It's somewhat obscure and probably doesn't really matter for user
interface, but API should be free even of obscure misbehaviour, I
believe.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
-- Samuel Beckett
^ permalink raw reply
* Re: Smart fetch via HTTP?
From: Theodore Tso @ 2007-05-17 12:36 UTC (permalink / raw)
To: Shawn O. Pearce
Cc: david, Johannes Schindelin, Martin Langhoff, Jan Hudec, git
In-Reply-To: <20070517014542.GW3141@spearce.org>
On Wed, May 16, 2007 at 09:45:42PM -0400, Shawn O. Pearce wrote:
> Its not perfect. The hotter parts of the object database is almost
> always the recent stuff, as that's what people are actively trying
> to fetch, or are using as a base when they are trying to fetch from
> someone else. The hotter parts are also probably too new to be
> in the shared store offered by kernel.org admins, which means you
> cannot get good IO buffering. Back to the current set of problems.
Actually, as long as objects/info/alternates is pointing at Linus's
kernel.org tree, I would think that it should work relatively well,
since everyone is normally basing their work on top of his tree as a
starting point.
- Ted
^ permalink raw reply
* Re: Smart fetch via HTTP?
From: Petr Baudis @ 2007-05-17 12:40 UTC (permalink / raw)
To: Jan Hudec; +Cc: git
In-Reply-To: <20070515201006.GD3653@efreet.light.src>
Hi,
On Tue, May 15, 2007 at 10:10:06PM CEST, Jan Hudec wrote:
> Did anyone already think about fetching over HTTP working similarly to the
> That is rather than reading the raw content of the repository, there would be
> a CGI script (could be integrated to gitweb), that would negotiate what the
> client needs and then generate and send a single pack with it.
frankly, I'm not that excited. I'm not disputing that this would be
useful, but I have my doubts on just how *much* useful it would be - I'm
not so sure the set of users affected is really all that large. So I'm
just cooling people down here. ;-))
> Mercurial and bzr both have this option. It would IMO have three benefits:
> - Fast access for people behind paranoid firewalls, that only let http and
> https (you can tunel anything through, but only to port 443) through.
How many users really have this problem? I'm not so sure. There are
certainly some, but enough for this to be a viable argument?
> - Can be run on shared machine. If you have web space on machine shared
> by many people, you can set up your own gitweb, but cannot/are not allowed
> to start your own network server for git native protocol.
You need to have CGI-enabled hosting, set up the CGI script etc. -
overally, the setup is similarly complicated as git-daemon setup, so
it's not "zero-setup" solution anymore.
Again, I'm not sure just how many people are in the situation that
they can run real CGI (not just PHP) but not git-daemon.
> - Less things to set up. If you are setting up gitweb anyway, you'd not need
> to set up additional thing for providing fetch access.
Except, well, how do you "set it up"? You need to make sure
git-update-server-info is run, yes, but that shouldn't be a problem (I'm
not so sure if git does this for you automagically - Cogito would...).
I think 95% of people don't set up gitweb.cgi either for their small
HTTP repositories. :-)
Then again, it's not that it would be really technically complicated -
adding "give me a bundle" support to gitweb should be pretty easy.
However, this support has some "social" costs as well: no compatibility
with older git versions, support cost, confusion between dumb HTTP and
gitweb HTTP transports, more lack of motivation for improving dumb HTTP
transport...
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
-- Samuel Beckett
^ permalink raw reply
* Re: [StGIT PATCH] Don't use patches/<branch>/current
From: Catalin Marinas @ 2007-05-17 12:43 UTC (permalink / raw)
To: Karl Hasselström; +Cc: git
In-Reply-To: <20070516204029.GB17805@diana.vm.bytemark.co.uk>
[-- Attachment #1: Type: text/plain, Size: 1483 bytes --]
On 16/05/07, Karl Hasselström <kha@treskal.com> wrote:
> I set up a kernel repository with 100 applied and 100 unapplied
> patches:
[...]
> Then repeatedly:
>
> $ time stg series >/dev/null
I ran 'git repack -a -d' and 'git prune'. There are no other objects
apart from the generated pack:
$ du -sh .git
211M .git
And then repeatedly 'time stg series > /dev/null':
real 0m1.638s
user 0m1.422s
sys 0m0.088s
real 0m2.542s
user 0m1.436s
sys 0m0.078s
real 0m2.916s
user 0m1.424s
sys 0m0.083s
real 0m2.940s
user 0m1.425s
sys 0m0.081s
real 0m1.614s
user 0m1.421s
sys 0m0.081s
real 0m1.587s
user 0m1.423s
sys 0m0.081s
real 0m2.653s
user 0m1.427s
sys 0m0.075s
> But my kernel repository is _much_ smaller than yours:
>
> $ du -sh .git
> 183M .git
>
> Do you perchance have a bunch of loose objects in there?
It got smaller after repacking but it is still bigger than yours.
Maybe the reason is that I have 14 branches with various patches, some
of them just for historical reasons but going back to 2.6.12. There
are also several commits generated for the patch logs.
The CPU is a P4 at 2.5GHz and the 'stg series' operation seems to be
CPU bound rather than IO. I'm also using Python 2.3 on this PC and for
this reason I changed 2 generator constructs (x for x in ...) with
list comprehension (see the attached patch).
--
Catalin
[-- Attachment #2: list-comprehension.patch --]
[-- Type: text/x-patch, Size: 1101 bytes --]
commit f7cc1f7fa6b2ee4f42e812970760ba5db5d2cafc
Author: Catalin Marinas <catalin.marinas@gmail.com>
Date: Wed May 16 13:08:45 2007 +0100
refresh b5003aa7bb5ec384e1dbe2887c07885a791384c2
diff --git a/stgit/stack.py b/stgit/stack.py
index 13eb692..0ef26b8 100644
--- a/stgit/stack.py
+++ b/stgit/stack.py
@@ -349,8 +349,8 @@ def unapplied_patches(ref2hash):
unapplied = Set()
for line in git._output_lines(
'git-rev-list --stdin',
- ('%s%s\n' % (['', '^'][ref == None], sha1)
- for ref, sha1 in ref2hash.iteritems())):
+ ['%s%s\n' % (['', '^'][ref == None], sha1)
+ for ref, sha1 in ref2hash.iteritems()]):
for ref in hash2refs.get(line.strip(), []):
unapplied.add(ref)
return unapplied
@@ -364,7 +364,7 @@ def sort_applied_patches(ref2hash):
if r != None:
hash2refs.setdefault(h, Set()).add(r)
- missing = Set(ref for ref in ref2hash.iterkeys() if ref != None)
+ missing = Set([ref for ref in ref2hash.iterkeys() if ref != None])
if not missing:
return []
applied = []
^ permalink raw reply related
* Re: [3/4] What's not in 1.5.2 (new topics)
From: Petr Baudis @ 2007-05-17 12:46 UTC (permalink / raw)
To: Alex Riesen; +Cc: Junio C Hamano, Andy Parkins, git
In-Reply-To: <20070517110225.GA3334@steel.home>
On Thu, May 17, 2007 at 01:02:25PM CEST, Alex Riesen wrote:
> why can't I just have _repo_ configuration:
>
> [subproject "kernel/"]
> URL = http://www.kernel.org/pub/linux-2.6.git
> ?
> It can be first-time cloned from the upstream, but it stays after
> people change it to suit their systems. They can depend on it not to
> be broken by upstream.
Because kernel/ can get removed, moved around, or point at entirely
*different* projects over time and branches - kernel/ can switch from
linux-2.4 to linux-2.6, libc/ can switch between glibc and uClibc, ...
> Can I suggest a part of repo configuration to be clonable? So that
> there is a something in .git/config.dist, which is _cloned_ with
> git-clone. The obviuos thing to put there would be subproject
> configuration, and maybe there will be something else in the future
> (I'd think of description, which is a separate file now, and as for
> now, the only way to get this description is to use gitweb or ssh).
> git-ls-remote could be made to show this "remote-accessible"
> configuration, in case someone have to update/compare local copy of
> this config.
This is troublesome because then you will also need a way to update the
configuration in the future, otherwise you will run into some
embarassing situations, and since we don't even support any motds while
fetching, when something *needs* to be changed you don't even have a
good way to tell your users. (Actually, I've been thinking about adding
motd support to the fetchers. :-)
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
-- Samuel Beckett
^ permalink raw reply
* Re: Smart fetch via HTTP?
From: Matthieu Moy @ 2007-05-17 12:48 UTC (permalink / raw)
To: git
In-Reply-To: <20070517124006.GO4489@pasky.or.cz>
Petr Baudis <pasky@suse.cz> writes:
>> Mercurial and bzr both have this option. It would IMO have three benefits:
>> - Fast access for people behind paranoid firewalls, that only let http and
>> https (you can tunel anything through, but only to port 443) through.
>
> How many users really have this problem? I'm not so sure.
Many (if not most?) of the people working in a big company, I'd say.
Year, it sucks, but people having used a paranoid firewall with a
not-less-paranoid and broken proxy understand what I mean.
>> - Can be run on shared machine. If you have web space on machine shared
>> by many people, you can set up your own gitweb, but cannot/are not allowed
>> to start your own network server for git native protocol.
>
> You need to have CGI-enabled hosting, set up the CGI script etc. -
> overally, the setup is similarly complicated as git-daemon setup, so
> it's not "zero-setup" solution anymore.
>
> Again, I'm not sure just how many people are in the situation that
> they can run real CGI (not just PHP) but not git-daemon.
Any volunteer to write a full-PHP version of git? ;-)
--
Matthieu
^ permalink raw reply
* Re: Smart fetch via HTTP?
From: Martin Langhoff @ 2007-05-17 13:10 UTC (permalink / raw)
To: git
In-Reply-To: <vpq8xbnlmdv.fsf@bauges.imag.fr>
On 5/17/07, Matthieu Moy <Matthieu.Moy@imag.fr> wrote:
> FYI, bzr uses HTTP range requests, and the introduction of this
> feature lead to significant performance improvement for them (bzr is
> more dumb-protocol oriented than git is, so that's really important
> there). They have this "index file+data file" system too, so you
> download the full index file, and then send an HTTP range request to
> get only the relevant parts of the data file.
That's the kind of thing I was imagining. Between the index and an
additional "index-supplement-for-dumb-protocols" maintained by
update-server-info, http ranges can be bent to our evil purposes.
Of course it won't be as network-efficient as the git proto, or even
as the git-over-cgi proto, but it'll surely be server-cpu-and-memory
efficient. And people will benefit from it without having to do any
additional setup.
It might be hard to come up with a usable approach to http ranges. But
I do think it's worth considering carefully.
cheers,
m
^ permalink raw reply
* Re: [3/4] What's not in 1.5.2 (new topics)
From: Nicolas Pitre @ 2007-05-17 13:45 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Andy Parkins, git
In-Reply-To: <7v4pmcauu3.fsf@assigned-by-dhcp.cox.net>
On Wed, 16 May 2007, Junio C Hamano wrote:
> Andy Parkins <andyparkins@gmail.com> writes:
>
> > Our in-tree .gitmodules will have the same problem. I recognise that
> > you've mitigated that with some "confirm with the user, store in the
> > config" hand waving; but that is just hiding the problem: the submodule
> > URL is not something that should be version controlled; it is an
> > all-of-history property; when it changes for revision N it changes for
> > revision N-1, N-2, N-3, etc. Storing it in .gitmodules implies that
> > it's value in the past has meaning - it doesn't.
>
> I think that depends _WHY_ the URL recorded .gitmodules are
> updated. It would perfectly be reasonable for release #1 of an
> appliance project to bind linux 2.4 tree at kernel/ subdirectory
> while release #2 source to have 2.6 one; they come from two
> different repository URLs. When you seek the superproject back
> to release #1, you would still want to fetch from 2.4 upstream
> if you are updating.
I don't know if the above example should make sense. In practice that
would mean you'll have to _replace_ the repo within the submodule
directory which is quite different from merely checking out a different
version of the same repository.
Nicolas
^ permalink raw reply
* Re: [3/4] What's not in 1.5.2 (new topics)
From: Jeff King @ 2007-05-17 13:46 UTC (permalink / raw)
To: Petr Baudis; +Cc: Alex Riesen, Junio C Hamano, Andy Parkins, git
In-Reply-To: <20070517124622.GP4489@pasky.or.cz>
On Thu, May 17, 2007 at 02:46:22PM +0200, Petr Baudis wrote:
> > why can't I just have _repo_ configuration:
> >
> > [subproject "kernel/"]
> > URL = http://www.kernel.org/pub/linux-2.6.git
> > ?
> > It can be first-time cloned from the upstream, but it stays after
> > people change it to suit their systems. They can depend on it not to
> > be broken by upstream.
>
> Because kernel/ can get removed, moved around, or point at entirely
> *different* projects over time and branches - kernel/ can switch from
> linux-2.4 to linux-2.6, libc/ can switch between glibc and uClibc, ...
I think we clearly need a 2-level system: a tracked pointer to the repo,
with an optional local override.
However, I don't quite like Junio's idea of using the URL as a key,
since it is intended to change. IOW, if I am overriding your URL via
.git/config, if you change your URL then my config is now broken.
Instead, why not:
1. url location is supplied in configuration as
[subproject "kernel/"]
url = git://git.kernel.org/pub/linux-2.4.git
2. .gitmodules is simply read as a lower-priority version of
configuration
One advantage of this approach is that it's totally general; instead of
.gitmodules, we could in fact be talking about .gitconfig, a mechanism
for projects to contain tracked configuration that can be overridden by
individual repos. For some projects, I imagine some of the commit
encoding config options might make sense.
Thoughts?
-Peff
^ permalink raw reply
* Re: Smart fetch via HTTP?
From: Johannes Schindelin @ 2007-05-17 13:47 UTC (permalink / raw)
To: Martin Langhoff; +Cc: git
In-Reply-To: <46a038f90705170610mf9c9b0eu7b40af709469a601@mail.gmail.com>
Hi,
[I missed this mail, because Matthieu culled the Cc list again]
On Fri, 18 May 2007, Martin Langhoff wrote:
> On 5/17/07, Matthieu Moy <Matthieu.Moy@imag.fr> wrote:
>
> > FYI, bzr uses HTTP range requests, and the introduction of this
> > feature lead to significant performance improvement for them (bzr is
> > more dumb-protocol oriented than git is, so that's really important
> > there). They have this "index file+data file" system too, so you
> > download the full index file, and then send an HTTP range request to
> > get only the relevant parts of the data file.
>
> That's the kind of thing I was imagining. Between the index and an
> additional "index-supplement-for-dumb-protocols" maintained by
> update-server-info, http ranges can be bent to our evil purposes.
>
> Of course it won't be as network-efficient as the git proto, or even
> as the git-over-cgi proto, but it'll surely be server-cpu-and-memory
> efficient. And people will benefit from it without having to do any
> additional setup.
Of course, the problem is that only the server can know beforehand which
objects are needed. Imagine this:
X - Y - Z
\
A
Client has "X", wants "Z", but not "A". Client needs "Y" and "Z". But
client cannot know that it needs "Y" before getting "Z", except if the
server says so.
If you have a solution for that problem, please enlighten me: I don't.
Ciao,
Dscho
^ permalink raw reply
* Re: [PATCH] Allow the ident attribute to include a length specifier
From: Andy Parkins @ 2007-05-17 13:53 UTC (permalink / raw)
To: git
In-Reply-To: <200705141405.29550.andyparkins@gmail.com>
On Monday 2007 May 14, Andy Parkins wrote:
> When the ident attribute is found for a path, then git replaces $ident$
> with:
>
> $ident: df2a1fd3ebce86876721bd7e12ce02ac89c885db $
>
> With this patch, you can put the following in your attribute file:
>
> somepath ident=10
>
> And get expansions like this:
>
> $ident: df2a1fd3eb $
>
> There is no change to existing behaviour. With no parameter, the
> expansion is all 40 hex digits.
Were there any thoughts on this patch? I think it might have got drowned in
the noise I made about $ident$ -> $Id$.
Andy
--
Dr Andy Parkins, M Eng (hons), MIET
andyparkins@gmail.com
^ permalink raw reply
* Re: Smart fetch via HTTP?
From: Matthieu Moy @ 2007-05-17 14:05 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: Martin Langhoff, git
In-Reply-To: <Pine.LNX.4.64.0705171445100.6410@racer.site>
Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> Hi,
>
> [I missed this mail, because Matthieu culled the Cc list again]
Sorry about that, miss-configuration of my mailer. I didn't find time
to solve it before.
OTOH, since most people actually complain when you Cc them on a
mailing list, the choice "To Cc or not to Cc" has no universal
solution ;-).
--
Matthieu
^ permalink raw reply
* Re: Smart fetch via HTTP?
From: Martin Langhoff @ 2007-05-17 14:09 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: git
In-Reply-To: <Pine.LNX.4.64.0705171445100.6410@racer.site>
On 5/18/07, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> If you have a solution for that problem, please enlighten me: I don't.
Ok - worst case scenario - have a minimal hints file that tells me the
ranges to fetch all commits and all trees. To reduce that Add to the
hints file data to name the hashes (or even better - offsets) for the
delta chains that contain commits+trees relevant to all the heads -
minus 10, 20, 30, 40 commits and 1,2,4,8 and 16 days.
So there's a good chance the client can get the commits+trees needed
efficiently. For blobs, all you need is the index to mark the delta
chains you need.
cheers,
m
^ permalink raw reply
* Re: [PATCH] Document core.excludesfile for git-add
From: Michael Hendricks @ 2007-05-17 14:35 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vwsz89ftr.fsf@assigned-by-dhcp.cox.net>
On Wed, May 16, 2007 at 10:31:12PM -0700, Junio C Hamano wrote:
> Thanks, but wouldn't this belong to Documentation/config.txt
> instead, I wonder?
It appears to my untrained eye that core.excludesfiles only works for
git-add. It seems to have no effect on git-status and the only code
that mentions core.excludesfile is builtin-add.c
--
Michael
^ permalink raw reply
* Re: Smart fetch via HTTP?
From: Nicolas Pitre @ 2007-05-17 14:41 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: Shawn O. Pearce, Martin Langhoff, Jan Hudec, git
In-Reply-To: <Pine.LNX.4.64.0705171143350.6410@racer.site>
[-- Attachment #1: Type: TEXT/PLAIN, Size: 4719 bytes --]
On Thu, 17 May 2007, Johannes Schindelin wrote:
> Hi,
>
> On Wed, 16 May 2007, Nicolas Pitre wrote:
>
> > Still... I wonder if this could be actually workable. A typical daily
> > update on the Linux kernel repository might consist of a couple hundreds
> > or a few tousands objects. This could still be faster to fetch parts of
> > a pack than the whole pack if the size difference is above a certain
> > treshold. It is certainly not worse than fetching loose objects.
> >
> > Things would be pretty horrid if you think of fetching a commit object,
> > parsing it to find out what tree object to fetch, then parse that tree
> > object to find out what other objects to fetch, and so on.
> >
> > But if you only take the approach of fetching the pack index files,
> > finding out about the objects that the remote has that are not available
> > locally, and then fetching all those objects from within pack files
> > without even looking at them (except for deltas), then it should be
> > possible to issue a couple requests in parallel and possibly have decent
> > performances. And if it turns out that more than, say, 70% of a
> > particular pack is to be fetched (you can determine that up front), then
> > it might be decided to fetch the whole pack.
> >
> > There is no way to sensibly keep those objects packed on the receiving
> > end of course, but storing them as loose objects and repacking them
> > afterwards should be just fine.
> >
> > Of course you'll get objects from branches in the remote repository you
> > might not be interested in, but that's a price to pay for such a hack.
> > On average the overhead shouldn't be that big anyway if branches within
> > a repository are somewhat related.
> >
> > I think this is something worth experimenting.
>
> I am a bit wary about that, because it is so complex. IMHO a cgi which
> gets, say, up to a hundred refs (maybe something like ref~0, ref~1, ref~2,
> ref~4, ref~8, ref~16, ... for the refs), and then makes a bundle for that
> case on the fly, is easier to do.
And if you have 1) the permission and 2) the CPU power to execute such a
cgi on the server and obviously 3) the knowledge to set it up properly,
then why aren't you running the Git daemon in the first place? After
all, they both boil down to running git-pack-objects and sending out the
result. I don't think such a solution really buys much.
On the other hand, if the client does all the work and provides the
server with a list of ranges within a pack it wants to be sent, then you
simply have zero special setup to perform on the hosting server and you
keep the server load down due to not running pack-objects there. That,
at least, is different enough from the Git daemon to be worth
considering. Not only does it provide an advantage to those who cannot
do anything but http out of their segregated network, but it also
provide many advantages on the server side too while the cgi approach
doesn't.
And actually finding out the list of objects the remote has that you
don't have is not that complex. It could go as follows:
1) Fetch every .idx files the remote has.
2) From those .idx files, keep only a list of objects that are unknown
locally. A good starting point for doing this really efficiently is
the code for git-pack-redundant.
3) From the .idx files we got in (1), create a reverse index to get each
object's size in the remote pack. The code to do this already exists
in builtin-pack-objects.c.
4) With the list of missing objects from (2) along with their offset and
size within a given pack file, fetch those objects from the remote
server. Either perform multiple requests in parallel, or as someone
mentioned already, provide the server with a list of ranges you want
to be sent.
5) Store the received objects as loose objects locally. If a given
object is a delta, verify if its base is available locally, or if it
is listed amongst those objects to be fetched from the server. If
not, add it to the list. In most cases, delta base objects will be
objects already listed to be fetched anyway. To greatly simplify
things, the loose delta object type from 2 years ago could be revived
(commit 91d7b8afc2) since a repack will get rid of them.
6 Repeat (4) and (5) until everything has been fetched.
7) Run git-pack-objects with the list of fetched objects.
Et voilà. Oh, and of course update your local refs from the remote's.
Actually there is nothing really complex in the above operations. And
with this the server side remains really simple with no special setup
nor extra load beyond the simple serving of file content.
Nicolas
^ permalink raw reply
* Re: Smart fetch via HTTP?
From: Nicolas Pitre @ 2007-05-17 14:50 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: Martin Langhoff, git
In-Reply-To: <Pine.LNX.4.64.0705171445100.6410@racer.site>
On Thu, 17 May 2007, Johannes Schindelin wrote:
> Hi,
>
> [I missed this mail, because Matthieu culled the Cc list again]
>
> On Fri, 18 May 2007, Martin Langhoff wrote:
>
> > On 5/17/07, Matthieu Moy <Matthieu.Moy@imag.fr> wrote:
> >
> > > FYI, bzr uses HTTP range requests, and the introduction of this
> > > feature lead to significant performance improvement for them (bzr is
> > > more dumb-protocol oriented than git is, so that's really important
> > > there). They have this "index file+data file" system too, so you
> > > download the full index file, and then send an HTTP range request to
> > > get only the relevant parts of the data file.
> >
> > That's the kind of thing I was imagining. Between the index and an
> > additional "index-supplement-for-dumb-protocols" maintained by
> > update-server-info, http ranges can be bent to our evil purposes.
> >
> > Of course it won't be as network-efficient as the git proto, or even
> > as the git-over-cgi proto, but it'll surely be server-cpu-and-memory
> > efficient. And people will benefit from it without having to do any
> > additional setup.
>
> Of course, the problem is that only the server can know beforehand which
> objects are needed.
But the whole idea is that we don't care.
> Imagine this:
>
> X - Y - Z
> \
> A
>
>
> Client has "X", wants "Z", but not "A". Client needs "Y" and "Z". But
> client cannot know that it needs "Y" before getting "Z", except if the
> server says so.
>
> If you have a solution for that problem, please enlighten me: I don't.
We're talking about a _dumb_ protocol here. If you want something
fancy, just use the Git daemon.
Otherwise, you'll simply get everything the remote has that you don't
have, including A.
In practice this shouldn't be a problem because people tend to have
clean repositories on machines they want their stuff to be published,
meaning that those public repos are usually the result of pushes, hence
they contain only the minimum set of needed objects. Of course you get
every branches and not only a particular one, but that's the price to
pay with a dumb protocol.
Nicolas
^ permalink raw reply
* Re: [StGIT PATCH] Don't use patches/<branch>/current
From: Karl Hasselström @ 2007-05-17 14:57 UTC (permalink / raw)
To: Catalin Marinas; +Cc: git
In-Reply-To: <b0943d9e0705170543g4d8c7542p6b44a4e6fa54676a@mail.gmail.com>
On 2007-05-17 13:43:35 +0100, Catalin Marinas wrote:
> I ran 'git repack -a -d' and 'git prune'. There are no other objects
> apart from the generated pack:
>
> $ du -sh .git
> 211M .git
>
> And then repeatedly 'time stg series > /dev/null':
Hmm, it seems there is a problem then. :-(
> It got smaller after repacking but it is still bigger than yours.
> Maybe the reason is that I have 14 branches with various patches,
> some of them just for historical reasons but going back to 2.6.12.
> There are also several commits generated for the patch logs.
OK. That shouldn't matter, though, since that extra history shouldn't
be examined anyway.
> The CPU is a P4 at 2.5GHz and the 'stg series' operation seems to be
> CPU bound rather than IO. I'm also using Python 2.3 on this PC and
> for this reason I changed 2 generator constructs (x for x in ...)
> with list comprehension (see the attached patch).
I don't think that's the problem, since those lists are both small.
The only possibility I can think of that might explain this is that
some of your unapplied patches are attached to a place in the commit
DAG that's far away from the branch head (e.g. you have rebased to
some entirely different place since you last had them applied), so
that "git-rev-list patch ^branch" outputs a large part of the commit
DAG.
Could you put counters in unapplied_patches() and
sort_applied_patches() to see how many lines each of them reads from
git-rev-list? The expected number (if it had taken just a little time,
like it did for me) is a small constant times the number of patches in
both cases.
--
Karl Hasselström, kha@treskal.com
www.treskal.com/kalle
^ permalink raw reply
* Re: Smart fetch via HTTP?
From: Nicolas Pitre @ 2007-05-17 15:01 UTC (permalink / raw)
To: Martin Langhoff; +Cc: Johannes Schindelin, git
In-Reply-To: <46a038f90705170709j7eb23d4fy6811fc2985dd888d@mail.gmail.com>
On Fri, 18 May 2007, Martin Langhoff wrote:
> On 5/18/07, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > If you have a solution for that problem, please enlighten me: I don't.
>
> Ok - worst case scenario - have a minimal hints file that tells me the
> ranges to fetch all commits and all trees. To reduce that Add to the
> hints file data to name the hashes (or even better - offsets) for the
> delta chains that contain commits+trees relevant to all the heads -
> minus 10, 20, 30, 40 commits and 1,2,4,8 and 16 days.
NO !
This is unreliable, unnecessary, and actually kills the beauty of
the solution's simplicity.
You get updates for every branches the remote has, period.
No server side extra files, no guesses, no arbitrary ranges, no backward
compatibility issues, no crap!
Nicolas
^ permalink raw reply
* Re: Smart fetch via HTTP?
From: Martin Langhoff @ 2007-05-17 15:24 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Johannes Schindelin, Shawn O. Pearce, Jan Hudec, git
In-Reply-To: <alpine.LFD.0.99.0705170954200.24220@xanadu.home>
On 5/18/07, Nicolas Pitre <nico@cam.org> wrote:
> And if you have 1) the permission and 2) the CPU power to execute such a
> cgi on the server and obviously 3) the knowledge to set it up properly,
> then why aren't you running the Git daemon in the first place?
And you probably _are_ running git daemon. But some clients may be on
shitty connections that only allow http. That's one of the scenarios
we're discussing.
cheers,
m
^ permalink raw reply
* Re: Smart fetch via HTTP?
From: Nicolas Pitre @ 2007-05-17 15:34 UTC (permalink / raw)
To: Martin Langhoff; +Cc: Johannes Schindelin, Shawn O. Pearce, Jan Hudec, git
In-Reply-To: <46a038f90705170824g4ef8c800w826ada3964b711a@mail.gmail.com>
On Fri, 18 May 2007, Martin Langhoff wrote:
> On 5/18/07, Nicolas Pitre <nico@cam.org> wrote:
> > And if you have 1) the permission and 2) the CPU power to execute such a
> > cgi on the server and obviously 3) the knowledge to set it up properly,
> > then why aren't you running the Git daemon in the first place?
>
> And you probably _are_ running git daemon. But some clients may be on
> shitty connections that only allow http. That's one of the scenarios
> we're discussing.
That's not what I'm disputing at all.
I'm disputing the vertue of an HTTP solution involving a cgi with Git
bundles vs an HTTP solution involving static file range serving. The
clients on shitty connections don't care either ways.
Nicolas
^ permalink raw reply
* Re: [PATCH] Document core.excludesfile for git-add
From: Petr Baudis @ 2007-05-17 15:39 UTC (permalink / raw)
To: Junio C Hamano, git
In-Reply-To: <20070517143542.GB15709@ginosko.local>
On Thu, May 17, 2007 at 04:35:44PM CEST, Michael Hendricks wrote:
> On Wed, May 16, 2007 at 10:31:12PM -0700, Junio C Hamano wrote:
> > Thanks, but wouldn't this belong to Documentation/config.txt
> > instead, I wonder?
>
> It appears to my untrained eye that core.excludesfiles only works for
> git-add. It seems to have no effect on git-status and the only code
> that mentions core.excludesfile is builtin-add.c
I believe the current policy should be that all config options are at
least briefly listed in config.txt and possibly are described in more
detail in the documentation of relevant command.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
-- Samuel Beckett
^ permalink raw reply
* Re: [PATCH] Document core.excludesfile for git-add
From: Johan Herland @ 2007-05-17 15:45 UTC (permalink / raw)
To: git; +Cc: Michael Hendricks, Junio C Hamano
In-Reply-To: <20070517143542.GB15709@ginosko.local>
On Thursday 17 May 2007, Michael Hendricks wrote:
> On Wed, May 16, 2007 at 10:31:12PM -0700, Junio C Hamano wrote:
> > Thanks, but wouldn't this belong to Documentation/config.txt
> > instead, I wonder?
>
> It appears to my untrained eye that core.excludesfiles only works for
> git-add. It seems to have no effect on git-status and the only code
> that mentions core.excludesfile is builtin-add.c
If so, we should probably fix that. The set of ignore files parsed by
relevant git commands should be consistent. We don't want a situation
where exclude patterns in a file are picked up by some git commands, but
not by others.
...Johan
--
Johan Herland, <johan@herland.net>
www.herland.net
^ permalink raw reply
* Re: [3/4] What's not in 1.5.2 (new topics)
From: Petr Baudis @ 2007-05-17 16:10 UTC (permalink / raw)
To: Jeff King; +Cc: Alex Riesen, Junio C Hamano, Andy Parkins, git
In-Reply-To: <20070517134649.GA20853@coredump.intra.peff.net>
On Thu, May 17, 2007 at 03:46:49PM CEST, Jeff King wrote:
> On Thu, May 17, 2007 at 02:46:22PM +0200, Petr Baudis wrote:
>
> > > why can't I just have _repo_ configuration:
> > >
> > > [subproject "kernel/"]
> > > URL = http://www.kernel.org/pub/linux-2.6.git
> > > ?
> > > It can be first-time cloned from the upstream, but it stays after
> > > people change it to suit their systems. They can depend on it not to
> > > be broken by upstream.
> >
> > Because kernel/ can get removed, moved around, or point at entirely
> > *different* projects over time and branches - kernel/ can switch from
> > linux-2.4 to linux-2.6, libc/ can switch between glibc and uClibc, ...
>
> I think we clearly need a 2-level system: a tracked pointer to the repo,
> with an optional local override.
>
> However, I don't quite like Junio's idea of using the URL as a key,
> since it is intended to change. IOW, if I am overriding your URL via
> .git/config, if you change your URL then my config is now broken.
>
> Instead, why not:
> 1. url location is supplied in configuration as
> [subproject "kernel/"]
> url = git://git.kernel.org/pub/linux-2.4.git
> 2. .gitmodules is simply read as a lower-priority version of
> configuration
But, did you read what you actually quoted? Because I can only repeat my
argument in the face of (1), and you didn't seem to dispute any part of
it at all.
"kernel/" has _no_ meaning. Only a (treeid,"kernel/") pair has meaning,
nothing less - a particular tree contains a submodule in given subtree.
Different trees can have different submodules in different subtrees.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
-- Samuel Beckett
^ permalink raw reply
* Re: [3/4] What's not in 1.5.2 (new topics)
From: Jeff King @ 2007-05-17 16:25 UTC (permalink / raw)
To: Petr Baudis; +Cc: Alex Riesen, Junio C Hamano, Andy Parkins, git
In-Reply-To: <20070517161002.GR4489@pasky.or.cz>
On Thu, May 17, 2007 at 06:10:02PM +0200, Petr Baudis wrote:
> But, did you read what you actually quoted? Because I can only repeat my
> argument in the face of (1), and you didn't seem to dispute any part of
> it at all.
You said:
> Because kernel/ can get removed, moved around, or point at entirely
> *different* projects over time and branches - kernel/ can switch from
> linux-2.4 to linux-2.6, libc/ can switch between glibc and uClibc, ...
which I took to mean that we must be able to track changes to the URL
which is pointed to by the kernel/ submodule, and therefore this
configuration must be in a tracked file. Which is _precisely_ what I
advocated: it goes in a .gitmodules (or .gitconfig) file in the tracked
directory. This is counter to what Alex says, which is that one should
simply pull the config down during clone time and never change it.
However, I think we _must_ have an override mechanism, since I don't
necessarily use the same URLs that you do. I propose that such overrides
should go into the local repo config. The only difference between what I
have proposed and what Junio mentioned is that I would base the config
override key on the directory name, not the URL. This means that if
upstream changes their pointer to the URL, yours will change with it
_unless you have an override_. With Junio's, their change of URL will
override your change (since the key will no longer match your config).
How do you propose to handle overrides?
> "kernel/" has _no_ meaning. Only a (treeid,"kernel/") pair has meaning,
> nothing less - a particular tree contains a submodule in given subtree.
> Different trees can have different submodules in different subtrees.
Right. In my proposal (unlike Alex's), it _is_ tied to the tree, since
that tree has a particular .gitmodules. But I also think you should be
able to override the submodule URL for kernel/ _for all time_ if you
want.
-Peff
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox