From: Zenaan Harkness <zen@freedbms.net>
To: git@vger.kernel.org
Subject: git torrent - sane deterministic pack files
Date: Tue, 2 Jun 2015 15:07:03 +1000 [thread overview]
Message-ID: <CAOsGNSSK6WwVvftc1Wx14gRyZ7Mx2qnXsDxducCCA0VBvqyK8w@mail.gmail.com> (raw)
Please CC me when replying, if you think of it. Thanks.
LWN discussion, this particular thread starts here:
https://lwn.net/Articles/646758/
Some extracts:
Deterministic packfile creation is required for parallel git downloads.
Example git pack file parameter/ configuration variations:
- compression on X number of CPUs
- maximum packfile size
- more trees/ branches in this repo than on that repo
A way is needed to capture these packfile config variations and
distribute them to other git servers (perhaps on a standardized branch
name or ??).
---
Related mail from 4 years ago:
http://article.gmane.org/gmane.comp.version-control.git/164643
Conceptually it may not be hard, but implementation is hard. By
forcing certain object layout rules, you may have lower compression
ratio, or slower pack access, and may consume more power. Git tries to
reuse deltas from existing packs to produce a new pack. This makes it
quick to assemble a pack, but also underterministic. There's also
threads stealing jobs from one another in the above link. Resumable
clone is a frequent request, and we still don't have it now.
---
Deterministic pack file parameter sets and compression can always be
tuned over time even though they change the format - that's just local
policy for the authoritative git torrent server.
Also for scenarios which benefit from pack file torrents, the marginal
reduction in compression (increase in pack file size) due to the need
for determinism may very well be valuable (marginal increase in local
storage in order to distribute downloads) - local policy strikes
again.
...
It's up to the "authoritative git server" admin to make the policy
decision of how long to keep with a current deterministic torrentable
pack file parameter set, and when to update to a new/tuned set. This
is always a local policy matter! "We can't do that because it's not
the best policy for maximum compression" is not the right answer
here...
...
If I am a torrent repo mirror, the "authoritative torrent upstream" is
merely a local config.
This not only sounds easy, it is easy - even in the face of
compression technology changes and "tuning" over time - that's merely
a "version" increase or new parameter set provided by the
"authoritative" repo server, and is local policy to that server.
Regards
Zenaan
reply other threads:[~2015-06-02 5:07 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAOsGNSSK6WwVvftc1Wx14gRyZ7Mx2qnXsDxducCCA0VBvqyK8w@mail.gmail.com \
--to=zen@freedbms.net \
--cc=git@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).