From: Eric Wong <e@80x24.org>
To: "Jakub Narębski" <jnareb@gmail.com>
Cc: git@vger.kernel.org
Subject: Re: [RFC 0/3] dumb HTTP transport speedups
Date: Sun, 24 Jul 2016 11:20:27 +0000 [thread overview]
Message-ID: <20160724112027.GB25507@starla> (raw)
In-Reply-To: <57949467.60904@gmail.com>
Jakub Narębski <jnareb@gmail.com> wrote:
> W dniu 2016-07-11 o 22:51, Eric Wong pisze:
>
> > TL;DR: dumb HTTP clone from a certain badly-packed repo goes from
> > ~2 hours to ~30 min memory usage drops from 2G to 360M
> >
> >
> > I hadn't packed the public repo at https://public-inbox.org/git
> > for a few weeks. As an admin of a small server limited memory
> > and CPU resources but fairly good bandwidth, I prefer clients
> > use dumb HTTP for initial clones.
>
> Hopefully the solution / workaround for large initial clone
> problem utilizing bundles (`git bundle`), which can be resumably
> transferred, would get standarized and automated.
I've been hoping to look at this more in coming weeks/months.
It would be nice if bundles and packs could be unified somehow
to avoid doubling storage on the server.
> Do you use bitmap indices for speeding up fetches?
Yes, but slow clients are still a problem since big responses
keeps memory-hungry processes running while trickling
(or waste disk space buffering the pack output up front)
Static packfiles/bundles are nice since all the clients can
share the same data on the server side as it's trickled out.
> BTW. IMVHO the problem with dumb HTTP is the latency, not extra
> bandwidth needed...
I enabled persistent connections for 404s on loose objects for
this reason :) We should probably be doing it across the board
on 404s, just haven't gotten around to it...
Increasing default parallelism should also help; but might hurt
some servers which can't handle many connections...
Hard to imagine people using antiquated prefork servers for
slow clients in a post-Slowloris world, but maybe it happens?
prev parent reply other threads:[~2016-07-24 11:20 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-07-11 20:51 [RFC 0/3] dumb HTTP transport speedups Eric Wong
2016-07-11 20:51 ` [PATCH 1/3] http-walker: remove unused parameter from fetch_object Eric Wong
2016-07-11 20:51 ` [PATCH 2/3] http: avoid disconnecting on 404s for loose objects Eric Wong
2016-07-11 20:51 ` [PATCH 3/3] http-walker: reduce O(n) ops with doubly-linked list Eric Wong
2016-07-11 21:02 ` [REJECT 4/3] http-walker: use hashmap to reduce list scan Eric Wong
2016-07-24 10:11 ` [RFC 0/3] dumb HTTP transport speedups Jakub Narębski
2016-07-24 11:20 ` Eric Wong [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160724112027.GB25507@starla \
--to=e@80x24.org \
--cc=git@vger.kernel.org \
--cc=jnareb@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).