* Re: SHA1 collisions found
From: Jeff King @ 2017-02-23 18:42 UTC (permalink / raw)
To: Joey Hess; +Cc: Linus Torvalds, Git Mailing List
In-Reply-To: <20170223182147.hbsyxsmyijgkqu75@kitenet.net>
On Thu, Feb 23, 2017 at 02:21:47PM -0400, Joey Hess wrote:
> Linus Torvalds wrote:
> > What you describe pretty much already requires a pre-image attack,
> > which the new attack is _not_.
> >
> > It's not clear that the "good" object can be anything sane.
>
> Generate a regular commit object; use the entire commit object + NUL as the
> chosen prefix, and use the identical-prefix collision attack to generate
> the colliding good/bad objects.
FWIW, git-fsck complains about those (and transfer.fsck rejects them):
$ (git cat-file commit HEAD; printf '\0more stuff') |
git hash-object -w --stdin -t commit
ecb2e5165c184f9025cb4c49d8f75901f4830354
$ git fsck
warning in commit ecb2e5165c184f9025cb4c49d8f75901f4830354: nulInCommit: NUL byte in the commit object body
So as long as either your "good" or "evil" commit has binary junk in it,
you are likely to be noticed (not everybody turns on transfer.fsck, but
GitHub does).
-Peff
^ permalink raw reply
* Re: [PATCH] http(s): automatically try NTLM authentication first
From: Junio C Hamano @ 2017-02-23 19:06 UTC (permalink / raw)
To: Johannes Schindelin
Cc: Jeff King, David Turner, git@vger.kernel.org,
sandals@crustytoothpaste.net, Eric Sunshine
In-Reply-To: <alpine.DEB.2.20.1702231806340.3767@virtualbox>
Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> On Wed, 22 Feb 2017, Jeff King wrote:
>> This patch drops the useless probe request:
> ...
>> but setting http.emptyauth adds back in the useless request. I think
>> that could be fixed by skipping the empty-auth thing when
>> http_auth_methods does not have CURLAUTH_NEGOTIATE in it (or perhaps
>> other methods need it to, so maybe skip it if _just_ BASIC is set).
>>
>> I suspect the patch above could probably be generalized as:
>>
>> /* cut out methods we know the server doesn't support */
>> http_auth_methods &= results.auth_avail;
>>
>> and let curl figure it out from there.
>
> Maybe this patch (or a variation thereof) would also be able to fix this
> problem with the patch:
>
> https://github.com/git-for-windows/git/issues/1034
>
> Short version: for certain servers (that do *not* advertise Negotiate),
> setting emptyauth to true will result in a failed fetch, without letting
> the user type in their credentials.
The issue described in that page looks rather serious.
I believe that a "variation" has become the first part of a
two-patch series that appear in the downthread from here. Perhaps
you can ask them to test it out (or even better if you have a setup
you can easily test against yourself)?
^ permalink raw reply
* Re: SHA1 collisions found
From: Joey Hess @ 2017-02-23 18:31 UTC (permalink / raw)
To: Linus Torvalds, Git Mailing List
In-Reply-To: <20170223182147.hbsyxsmyijgkqu75@kitenet.net>
[-- Attachment #1: Type: text/plain, Size: 886 bytes --]
Joey Hess wrote:
> Linus Torvalds wrote:
> > What you describe pretty much already requires a pre-image attack,
> > which the new attack is _not_.
> >
> > It's not clear that the "good" object can be anything sane.
>
> Generate a regular commit object; use the entire commit object + NUL as the
> chosen prefix, and use the identical-prefix collision attack to generate
> the colliding good/bad objects.
>
> (The size in git's object header is a minor complication. Set the size
> field to something sufficiently large, and then pad out the colliding
> objects to that size once they're generated.)
Sorry! While that would work, it's a useless attack because the good and bad
commit objects still point to the same tree.
It would be interesting to have such colliding objects, to see what beaks,
but probably not worth $75k to generate them.
--
see shy jo
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: [PATCH v2] send-email: only allow one address per body tag
From: Junio C Hamano @ 2017-02-23 18:53 UTC (permalink / raw)
To: Matthieu Moy; +Cc: Johan Hovold, Jeff King, Kevin Daudt, Larry Finger, git
In-Reply-To: <vpqo9xxkqqo.fsf@anie.imag.fr>
Matthieu Moy <Matthieu.Moy@grenoble-inp.fr> writes:
> Johan Hovold <johan@kernel.org> writes:
>
>> --- a/git-send-email.perl
>> +++ b/git-send-email.perl
>> @@ -1563,7 +1563,7 @@ foreach my $t (@files) {
>> # Now parse the message body
>> while(<$fh>) {
>> $message .= $_;
>> - if (/^(Signed-off-by|Cc): (.*)$/i) {
>> + if (/^(Signed-off-by|Cc): ([^>]*>?)/i) {
>
> I think this is acceptable, but this doesn't work with trailers like
>
> Cc: "Some > Body" <Some.Body@example.com>
>
> A proper management of this kind of weird address should be doable by
> reusing the regexp parsing "..." in parse_mailbox:
>
> my $re_quote = qr/"(?:[^\"\\]|\\.)*"/;
>
> So the final regex would look like
>
> if (/^(Signed-off-by|Cc): (([^>]*|"(?:[^\"\\]|\\.)*")>?)/i) {
>
> I don't think that should block the patch inclusion, but it may be worth
> considering.
>
> Anyway, thanks for the patch!
Somehow this fell off the radar. So your reviewed-by: and then
we'll cook this in 'next' for a while?
Thanks.
^ permalink raw reply
* [PATCH] upload-pack: report "not our ref" to client
From: Jonathan Tan @ 2017-02-23 18:43 UTC (permalink / raw)
To: git; +Cc: Jonathan Tan, gitster
In-Reply-To: <xmqqo9xtajcu.fsf@gitster.mtv.corp.google.com>
Make upload-pack report "not our ref" errors to the client as an "ERR" line.
(If not, the client would be left waiting for a response when the server is
already dead.)
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
---
Thanks, here is the independent patch.
upload-pack.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/upload-pack.c b/upload-pack.c
index 7597ba340..ffb028d62 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -822,9 +822,13 @@ static void receive_needs(void)
use_include_tag = 1;
o = parse_object(sha1_buf);
- if (!o)
+ if (!o) {
+ packet_write_fmt(1,
+ "ERR upload-pack: not our ref %s",
+ sha1_to_hex(sha1_buf));
die("git upload-pack: not our ref %s",
sha1_to_hex(sha1_buf));
+ }
if (!(o->flags & WANTED)) {
o->flags |= WANTED;
if (!((allow_unadvertised_object_request & ALLOW_ANY_SHA1) == ALLOW_ANY_SHA1
--
2.11.0.483.g087da7b7c-goog
^ permalink raw reply related
* Re: SHA1 collisions found
From: Jeff King @ 2017-02-23 18:46 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Joey Hess, Git Mailing List
In-Reply-To: <CA+55aFxckeEW1ePcebrgG4iN4Lp62A2vU6tA=xnSDC_BnKQiCQ@mail.gmail.com>
On Thu, Feb 23, 2017 at 10:40:48AM -0800, Linus Torvalds wrote:
> > Generate a regular commit object; use the entire commit object + NUL as the
> > chosen prefix, and use the identical-prefix collision attack to generate
> > the colliding good/bad objects.
>
> So I agree with you that we need to make git check for the opaque
> data. I think I was the one who brought that whole argument up.
We do already.
> But even then, what you describe doesn't work. What you describe just
> replaces the opaque data - that git doesn't actually *use*, and that
> nobody sees - with another piece of opaque data.
>
> You also need to make the non-opaque data of the bad object be
> something that actually encodes valid git data with interesting hashes
> in it (for the parent/tree/whatever pointers).
>
> So you don't have just that "chosen prefix". You actually need to also
> fill in some very specific piece of data *in* the attack parts itself.
> And you need to do this in the exact same size (because that's part of
> the prefix), etc etc.
It's not an identical prefix, but I think collision attacks generally
are along the lines of selecting two prefixes followed by garbage, and
then mutating the garbage on both sides. That would "work" in this case
(modulo the fact that git would complain about the NUL).
I haven't read the paper yet to see if that is the case here, though.
A related case is if you could stick a "cruft ...." header at the end of
the commit headers, and mutate its value (avoiding newlines). fsck
doesn't complain about that.
-Peff
^ permalink raw reply
* Re: SHA1 collisions found
From: Linus Torvalds @ 2017-02-23 18:40 UTC (permalink / raw)
To: Joey Hess; +Cc: Git Mailing List
In-Reply-To: <20170223182147.hbsyxsmyijgkqu75@kitenet.net>
On Thu, Feb 23, 2017 at 10:21 AM, Joey Hess <id@joeyh.name> wrote:
> Linus Torvalds wrote:
>> What you describe pretty much already requires a pre-image attack,
>> which the new attack is _not_.
>>
>> It's not clear that the "good" object can be anything sane.
>
> Generate a regular commit object; use the entire commit object + NUL as the
> chosen prefix, and use the identical-prefix collision attack to generate
> the colliding good/bad objects.
So I agree with you that we need to make git check for the opaque
data. I think I was the one who brought that whole argument up.
But even then, what you describe doesn't work. What you describe just
replaces the opaque data - that git doesn't actually *use*, and that
nobody sees - with another piece of opaque data.
You also need to make the non-opaque data of the bad object be
something that actually encodes valid git data with interesting hashes
in it (for the parent/tree/whatever pointers).
So you don't have just that "chosen prefix". You actually need to also
fill in some very specific piece of data *in* the attack parts itself.
And you need to do this in the exact same size (because that's part of
the prefix), etc etc.
So I think it's challenging.
... and then we can discover it trivially.
Ok, so "git fsck" right now takes a couple of minutes for me and I
don't actually run it very often (I used to run it religiously back in
the days), but afaik kernel.org actually runs it nightly. So it's
pretty much "trivially discoverable" - imagine spending thousands of
CPU-hours and lots of social capital to get an attack in, and then the
next night the kernel.org fsck complains about the strange commit you
added?
Linus
^ permalink raw reply
* Re: SHA1 collisions found
From: Junio C Hamano @ 2017-02-23 18:38 UTC (permalink / raw)
To: Joey Hess; +Cc: Linus Torvalds, Git Mailing List
In-Reply-To: <20170223181018.ns4vyosgzmuoyiva@kitenet.net>
Joey Hess <id@joeyh.name> writes:
> For example, git fsck does warn about a commit message with opaque
> data hidden after a NUL. But, git show/merge/pull give no indication
> that something funky is going on when working with such commits.
Would
$ git config transfer.fsckobjects true
help?
^ permalink raw reply
* Re: SHA1 collisions found
From: Linus Torvalds @ 2017-02-23 18:29 UTC (permalink / raw)
To: Joey Hess; +Cc: Git Mailing List
In-Reply-To: <20170223181018.ns4vyosgzmuoyiva@kitenet.net>
On Thu, Feb 23, 2017 at 10:10 AM, Joey Hess <id@joeyh.name> wrote:
>
> It would cost 6500 CPU years + 100 GPU years to generate valid colliding
> git objects using the methods of the paper's authors. That might be cost
> effective if it helped get a backdoor into eg, the kernel.
I still think it also needs to be interesting enough data, not just
random noise that is then trivial to find with automated tools.
Because for the kernel, it's not just that an attacker needs to do the
CPU time. Yes, first he needs the technical resources to just do just
the attack and create the situation you described.
But then he *also* needs to build up the social capital to get the end
result pulled into the tree (ie if he depends on the hidden spaces, he
needs somebody to actually do a git pull, not just apply a patch).
.. and if we then have a tool that then finds the problem trivially
(ie "git fsck"), he's not only wasted all those technical resources,
he's also burned his identity.
>> (b) we can probably easily add some extra sanity checks to the opaque
>> data we do have, to make it much harder to do the hiding of random
>> data that these attacks pretty much always depend on.
>
> For example, git fsck does warn about a commit message with opaque
> data hidden after a NUL. But, git show/merge/pull give no indication
> that something funky is going on when working with such commits.
I do agree that we might want to do some of the fsck checks
particularly at fetch time. That's when doing checks is both relevant
and cheap.
So we could do the opaque data checks, but we could/should probably
also add the attack pattern ("disturbance vectors") checks.
And the thing is, adding those checks is really cheap, and basically
makes the whole attack vector pointless against git.
Because unlike some "signing a pdf" attack, git doesn't fundamentally
depend on the SHA1 as some kind of absolute security. If we have the
minimal machinery in git to just notice the attack, the attack
essentially goes away. Attackers can waste infinite amounts of CPU
time, and if it's cheap for us to notice, it completely disarms all
that attack work.
Again, I'm not arguing that people shouldn't work on extending git to
a new (and bigger) hash. I think that's a no-brainer, and we do want
to have a path to eventually move towards SHA3-256 or whatever.
But I'm very definitely arguing that the current attack doesn't
actually sound like it really even _matters_, because it should be so
easy to mitigate against.
Linus
^ permalink raw reply
* Re: SHA1 collisions found
From: Joey Hess @ 2017-02-23 18:21 UTC (permalink / raw)
To: Linus Torvalds, Git Mailing List
In-Reply-To: <CA+55aFzFEpi1crykZ33r9f7BsvLt_kiB-CHXOkuCAX=fd4BU-w@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 591 bytes --]
Linus Torvalds wrote:
> What you describe pretty much already requires a pre-image attack,
> which the new attack is _not_.
>
> It's not clear that the "good" object can be anything sane.
Generate a regular commit object; use the entire commit object + NUL as the
chosen prefix, and use the identical-prefix collision attack to generate
the colliding good/bad objects.
(The size in git's object header is a minor complication. Set the size
field to something sufficiently large, and then pad out the colliding
objects to that size once they're generated.)
--
see shy jo
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: SHA1 collisions found
From: David Lang @ 2017-02-23 17:52 UTC (permalink / raw)
To: Joey Hess; +Cc: Junio C Hamano, Git Mailing List
In-Reply-To: <20170223173547.qljypk7sdqi37oha@kitenet.net>
On Thu, 23 Feb 2017, Joey Hess wrote:
> Junio C Hamano wrote:
>> On Thu, Feb 23, 2017 at 8:43 AM, Joey Hess <id@joeyh.name> wrote:
>>>
>>> Since we now have collisions in valid PDF files, collisions in valid git
>>> commit and tree objects are probably able to be constructed.
>>
>> That may be true, but
>> https://public-inbox.org/git/Pine.LNX.4.58.0504291221250.18901@ppc970.osdl.org/
>
> That's about someone replacing an valid object in Linus's repository
> with an invalid random blob they found that collides. This SHA1
> break doesn't allow generating such a blob anyway. Linus is right,
> that's an impractical attack.
>
> Attacks using this SHA1 break will look something more like:
>
> * I push a "bad" object to a repo on github I set up under a
> pseudonym.
> * I publish a "good" object in a commit and convince the maintainer to
> merge it.
> * I wait for the maintainer to push to github.
> * I wait for github to deduplicate and hope they'll replace the good
> object with the bad one I pre-uploaded, thus silently changing the
> content of the good commit the maintainer reviewed and pushed.
> * The bad object is pulled from github and deployed.
> * The maintainer still has the good object. They may not notice the bad
> object is out there for a long time.
>
> Of course, it doesn't need to involve Github, and doesn't need to
> rely on internal details of their deduplication[1];
> that only let me publish the bad object under a psydonym.
read that e-mail again, it covers the case where a central server gets a blob
replaced in it.
tricking a maintainerinto accepting a file that contains huge amounts of binary
data in it is going to be a non-trivial task, and even after you trick them into
accepting one bad file, you then need to replace the file they accepted with a
new one (breaking into github or assuming that github is putting both files into
the same repo, both of which are fairly unlikely)
David Lang
^ permalink raw reply
* Re: SHA1 collisions found
From: Joey Hess @ 2017-02-23 18:10 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Git Mailing List
In-Reply-To: <CA+55aFxJGDpJXqpcoPnwvzcn_fB-zaggj=w7P2At-TOt4buOqw@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1699 bytes --]
Linus Torvalds wrote:
> I haven't seen the attack yet, but git doesn't actually just hash the
> data, it does prepend a type/length field to it. That usually tends to
> make collision attacks much harder, because you either have to make
> the resulting size the same too, or you have to be able to also edit
> the size field in the header.
I have some sha1 collisions (and other fun along these lines) in
https://github.com/joeyh/supercollider
That includes two files with the same SHA and size, which do get
different blobs thanks to the way git prepends the header to the
content.
joey@darkstar:~/tmp/supercollider>sha1sum bad.pdf good.pdf
d00bbe65d80f6d53d5c15da7c6b4f0a655c5a86a bad.pdf
d00bbe65d80f6d53d5c15da7c6b4f0a655c5a86a good.pdf
joey@darkstar:~/tmp/supercollider>git ls-tree HEAD
100644 blob ca44e9913faf08d625346205e228e2265dd12b65 bad.pdf
100644 blob 5f90b67523865ad5b1391cb4a1c010d541c816c1 good.pdf
While appending identical data to these colliding files does generate
other collisions, prepending data does not.
It would cost 6500 CPU years + 100 GPU years to generate valid colliding
git objects using the methods of the paper's authors. That might be cost
effective if it helped get a backdoor into eg, the kernel.
> (b) we can probably easily add some extra sanity checks to the opaque
> data we do have, to make it much harder to do the hiding of random
> data that these attacks pretty much always depend on.
For example, git fsck does warn about a commit message with opaque
data hidden after a NUL. But, git show/merge/pull give no indication
that something funky is going on when working with such commits.
--
see shy jo
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: SHA1 collisions found
From: Linus Torvalds @ 2017-02-23 17:52 UTC (permalink / raw)
To: Joey Hess; +Cc: Junio C Hamano, Git Mailing List
In-Reply-To: <20170223173547.qljypk7sdqi37oha@kitenet.net>
On Thu, Feb 23, 2017 at 9:35 AM, Joey Hess <id@joeyh.name> wrote:
>
> Attacks using this SHA1 break will look something more like:
We don't actually know what the break is, but it's likely that you
can't actually do what you think you can do:
> * I push a "bad" object to a repo on github I set up under a
> pseudonym.
> * I publish a "good" object in a commit and convince the maintainer to
> merge it.
It's not clear that the "good" object can be anything sane.
What you describe pretty much already requires a pre-image attack,
which the new attack is _not_.
The new attack doesn't have a controlled "good" case, you need two
different objects that both have "near-collision" blocks in the
middle. I don't know what the format of those near-collision blocks
are, but it's a big problem.
You blithely just say "I create a good object". It's not that simple.
If it was, this would be a pre-image attack.
So basically, the attack needs some kind of random binary garbage in
*both* objects in the middle.
That's why pdf's are the classic model for showing these attacks: it's
easy to insert garbage in the middle of a pdf that is invisible.
In a psf, you can just define a bitmap that you don't use for printing
- but you can use them to then make a decision about what to print -
making the printed version of the pdf look radically different in ways
that are not so much _directly_ about the invisible block itself.
Linus
^ permalink raw reply
* Re: SHA1 collisions found
From: Linus Torvalds @ 2017-02-23 17:29 UTC (permalink / raw)
To: Joey Hess; +Cc: Git Mailing List
In-Reply-To: <CA+55aFxJGDpJXqpcoPnwvzcn_fB-zaggj=w7P2At-TOt4buOqw@mail.gmail.com>
On Thu, Feb 23, 2017 at 9:19 AM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> I don't think you'd necessarily want to change the size of the hash.
> You can use a different hash and just use the same 160 bits from it.
Side note: I do believe that in practice you should just change the
size of the hash too, I'm just saying that the size of the hash and
the choice of the hash algorithm are independent issues.
So you *could* just use something like SHA3-256, but then pick the
first 160 bits.
Realistically, changing the few hardcoded sizes internally in git is
likely the least problem in switching hashes.
So what you'd probably do is switch to a 256-bit hash, use that
internally and in the native git database, and then by default only
*show* the hash as a 40-character hex string (kind of like how we
already abbreviate things in many situations).
That way tools around git don't even see the change unless passed in
some special "--full-hash" argument (or "--abbrev=64" or whatever -
the default being that we abbreviate to 40).
Linus
^ permalink raw reply
* Re: SHA1 collisions found
From: Joey Hess @ 2017-02-23 17:35 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Git Mailing List
In-Reply-To: <CAPc5daVZ79WWKSw76kxHgDra9a7fSR1AibZa_pvK9aUuuVawLQ@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1602 bytes --]
Junio C Hamano wrote:
> On Thu, Feb 23, 2017 at 8:43 AM, Joey Hess <id@joeyh.name> wrote:
> >
> > Since we now have collisions in valid PDF files, collisions in valid git
> > commit and tree objects are probably able to be constructed.
>
> That may be true, but
> https://public-inbox.org/git/Pine.LNX.4.58.0504291221250.18901@ppc970.osdl.org/
That's about someone replacing an valid object in Linus's repository
with an invalid random blob they found that collides. This SHA1
break doesn't allow generating such a blob anyway. Linus is right,
that's an impractical attack.
Attacks using this SHA1 break will look something more like:
* I push a "bad" object to a repo on github I set up under a
pseudonym.
* I publish a "good" object in a commit and convince the maintainer to
merge it.
* I wait for the maintainer to push to github.
* I wait for github to deduplicate and hope they'll replace the good
object with the bad one I pre-uploaded, thus silently changing the
content of the good commit the maintainer reviewed and pushed.
* The bad object is pulled from github and deployed.
* The maintainer still has the good object. They may not notice the bad
object is out there for a long time.
Of course, it doesn't need to involve Github, and doesn't need to
rely on internal details of their deduplication[1];
that only let me publish the bad object under a psydonym.
--
see shy jo
[1] Which I'm only guessing about, but now that we have colliding
objects, we can upload them to different repos and see if such
dedupication happens.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: SHA1 collisions found
From: David Lang @ 2017-02-23 17:12 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Joey Hess, Git Mailing List
In-Reply-To: <CAPc5daVZ79WWKSw76kxHgDra9a7fSR1AibZa_pvK9aUuuVawLQ@mail.gmail.com>
On Thu, 23 Feb 2017, Junio C Hamano wrote:
> On Thu, Feb 23, 2017 at 8:43 AM, Joey Hess <id@joeyh.name> wrote:
>>
>> Since we now have collisions in valid PDF files, collisions in valid git
>> commit and tree objects are probably able to be constructed.
>
> That may be true, but
> https://public-inbox.org/git/Pine.LNX.4.58.0504291221250.18901@ppc970.osdl.org/
>
it doesn't help that the Google page on this explicitly says that this shows
that it's possible to create two different git repos that have the same hash but
different contents.
https://shattered.it/
How is GIT affected?
GIT strongly relies on SHA-1 for the identification and integrity checking of
all file objects and commits. It is essentially possible to create two GIT
repositories with the same head commit hash and different contents, say a benign
source code and a backdoored one. An attacker could potentially selectively
serve either repository to targeted users. This will require attackers to
compute their own collision.
David Lang
^ permalink raw reply
* Re: SHA1 collisions found
From: Linus Torvalds @ 2017-02-23 17:19 UTC (permalink / raw)
To: Joey Hess; +Cc: Git Mailing List
In-Reply-To: <20170223164306.spg2avxzukkggrpb@kitenet.net>
On Thu, Feb 23, 2017 at 8:43 AM, Joey Hess <id@joeyh.name> wrote:
>
> IIRC someone has been working on parameterizing git's SHA1 assumptions
> so a repository could eventually use a more secure hash. How far has
> that gotten? There are still many "40" constants in git.git HEAD.
I don't think you'd necessarily want to change the size of the hash.
You can use a different hash and just use the same 160 bits from it.
> Since we now have collisions in valid PDF files, collisions in valid git
> commit and tree objects are probably able to be constructed.
I haven't seen the attack yet, but git doesn't actually just hash the
data, it does prepend a type/length field to it. That usually tends to
make collision attacks much harder, because you either have to make
the resulting size the same too, or you have to be able to also edit
the size field in the header.
pdf's don't have that issue, they have a fixed header and you can
fairly arbitrarily add silent data to the middle that just doesn't get
shown.
So pdf's make for a much better attack vector, exactly because they
are a fairly opaque data format. Git has opaque data in some places
(we hide things in commit objects intentionally, for example, but by
definition that opaque data is fairly secondary.
Put another way: I doubt the sky is falling for git as a source
control management tool. Do we want to migrate to another hash? Yes.
Is it "game over" for SHA1 like people want to say? Probably not.
I haven't seen the attack details, but I bet
(a) the fact that we have a separate size encoding makes it much
harder to do on git objects in the first place
(b) we can probably easily add some extra sanity checks to the opaque
data we do have, to make it much harder to do the hiding of random
data that these attacks pretty much always depend on.
Linus
^ permalink raw reply
* Re: SHA1 collisions found
From: Junio C Hamano @ 2017-02-23 17:18 UTC (permalink / raw)
To: Joey Hess; +Cc: Git Mailing List
In-Reply-To: <CAPc5daVZ79WWKSw76kxHgDra9a7fSR1AibZa_pvK9aUuuVawLQ@mail.gmail.com>
On Thu, Feb 23, 2017 at 9:02 AM, Junio C Hamano <gitster@pobox.com> wrote:
> On Thu, Feb 23, 2017 at 8:43 AM, Joey Hess <id@joeyh.name> wrote:
>>
>> Since we now have collisions in valid PDF files, collisions in valid git
>> commit and tree objects are probably able to be constructed.
>
> That may be true, but
> https://public-inbox.org/git/Pine.LNX.4.58.0504291221250.18901@ppc970.osdl.org/
IOW, we want to continue the work to switch from SHA-1, but today's announcement
does not fundamentally change anything and we do not panic.
^ permalink raw reply
* Re: SHA1 collisions found
From: David Lang @ 2017-02-23 17:00 UTC (permalink / raw)
To: Joey Hess; +Cc: git
In-Reply-To: <20170223164306.spg2avxzukkggrpb@kitenet.net>
On Thu, 23 Feb 2017, Joey Hess wrote:
> https://shattered.io/static/shattered.pdf
> https://freedom-to-tinker.com/2017/02/23/rip-sha-1/
>
> IIRC someone has been working on parameterizing git's SHA1 assumptions
> so a repository could eventually use a more secure hash. How far has
> that gotten? There are still many "40" constants in git.git HEAD.
>
> In the meantime, git commit -S, and checks that commits are signed,
> seems like the only way to mitigate against attacks such as
> the ones described in the threads at
> https://joeyh.name/blog/sha-1/ and
> https://joeyh.name/blog/entry/size_of_the_git_sha1_collision_attack_surface/
>
> Since we now have collisions in valid PDF files, collisions in valid git
> commit and tree objects are probably able to be constructed.
keep in mind that there is a huge difference between
creating a collision between two documents you create, both of which contain a
huge amount of arbitrary binary data that can be changed at will without
affecting the results
and
creating a collision betwen an existing document that someone else created and a
new document that is also valid C code without huge amounts of binary in it.
So, it's not time to panic, but it is one more push to make the changes to
support something else.
David Lang
^ permalink raw reply
* Re: [PATCH] http(s): automatically try NTLM authentication first
From: Johannes Schindelin @ 2017-02-23 17:08 UTC (permalink / raw)
To: Jeff King
Cc: Junio C Hamano, David Turner, git@vger.kernel.org,
sandals@crustytoothpaste.net, Eric Sunshine
In-Reply-To: <20170222213410.iak43asq775tzr42@sigill.intra.peff.net>
Hi Peff,
On Wed, 22 Feb 2017, Jeff King wrote:
> On Wed, Feb 22, 2017 at 01:16:33PM -0800, Junio C Hamano wrote:
>
> > David Turner <David.Turner@twosigma.com> writes:
> >
> > > Always, no. For failed authentication (or authorization),
> > > apparently, yes. I tested this by setting the variable to false
> > > and then true, and trying to Push to a github repository which I
> > > didn't have write access to, with both an empty username
> > > (https://@:github.com/...) and no username (http://github.com/...).
> > > I ran this under GIT_CURL_VERBOSE=1 and I saw two 401 responses in
> > > the "http.emptyauth=true" case and one in the false case. I also
> > > tried with a repo that I did have access to (first configuring the
> > > necessary tokens for HTTPS push access), and saw two 401 responses
> > > in *both* cases.
> >
> > Thanks; that matches my observation. I do not think we care about
> > an extra roundtrip for the failure case, but as long as we do not
> > increase the number of roundtrip in the normal case, we can declare
> > that this is an improvement. I am not quite sure where that extra
> > 401 comes from in the normal case, and that might be an indication
> > that we already are doing something wrong, though.
>
> This patch drops the useless probe request:
>
> diff --git a/http.c b/http.c
> index 943e630ea..7b4c2db86 100644
> --- a/http.c
> +++ b/http.c
> @@ -1663,6 +1663,9 @@ static int http_request(const char *url,
> curlinfo_strbuf(slot->curl, CURLINFO_EFFECTIVE_URL,
> options->effective_url);
>
> + if (results.auth_avail == CURLAUTH_BASIC)
> + http_auth_methods = CURLAUTH_BASIC;
> +
> curl_slist_free_all(headers);
> strbuf_release(&buf);
>
>
> but setting http.emptyauth adds back in the useless request. I think
> that could be fixed by skipping the empty-auth thing when
> http_auth_methods does not have CURLAUTH_NEGOTIATE in it (or perhaps
> other methods need it to, so maybe skip it if _just_ BASIC is set).
>
> I suspect the patch above could probably be generalized as:
>
> /* cut out methods we know the server doesn't support */
> http_auth_methods &= results.auth_avail;
>
> and let curl figure it out from there.
Maybe this patch (or a variation thereof) would also be able to fix this
problem with the patch:
https://github.com/git-for-windows/git/issues/1034
Short version: for certain servers (that do *not* advertise Negotiate),
setting emptyauth to true will result in a failed fetch, without letting
the user type in their credentials.
Ciao,
Johannes
^ permalink raw reply
* Re: SHA1 collisions found
From: Junio C Hamano @ 2017-02-23 17:02 UTC (permalink / raw)
To: Joey Hess; +Cc: Git Mailing List
In-Reply-To: <20170223164306.spg2avxzukkggrpb@kitenet.net>
On Thu, Feb 23, 2017 at 8:43 AM, Joey Hess <id@joeyh.name> wrote:
>
> Since we now have collisions in valid PDF files, collisions in valid git
> commit and tree objects are probably able to be constructed.
That may be true, but
https://public-inbox.org/git/Pine.LNX.4.58.0504291221250.18901@ppc970.osdl.org/
^ permalink raw reply
* SHA1 collisions found
From: Joey Hess @ 2017-02-23 16:43 UTC (permalink / raw)
To: git
[-- Attachment #1: Type: text/plain, Size: 748 bytes --]
https://shattered.io/static/shattered.pdf
https://freedom-to-tinker.com/2017/02/23/rip-sha-1/
IIRC someone has been working on parameterizing git's SHA1 assumptions
so a repository could eventually use a more secure hash. How far has
that gotten? There are still many "40" constants in git.git HEAD.
In the meantime, git commit -S, and checks that commits are signed,
seems like the only way to mitigate against attacks such as
the ones described in the threads at
https://joeyh.name/blog/sha-1/ and
https://joeyh.name/blog/entry/size_of_the_git_sha1_collision_attack_surface/
Since we now have collisions in valid PDF files, collisions in valid git
commit and tree objects are probably able to be constructed.
--
see shy jo
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* RE: [PATCH 2/2] http: add an "auto" mode for http.emptyauth
From: David Turner @ 2017-02-23 16:31 UTC (permalink / raw)
To: 'Jeff King'
Cc: Junio C Hamano, git@vger.kernel.org, sandals@crustytoothpaste.net,
Johannes Schindelin, Eric Sunshine
In-Reply-To: <20170223013746.lturqad7lnehedb4@sigill.intra.peff.net>
> -----Original Message-----
> From: Jeff King [mailto:peff@peff.net]
> Sent: Wednesday, February 22, 2017 8:38 PM
> To: David Turner <David.Turner@twosigma.com>
> Cc: Junio C Hamano <gitster@pobox.com>; git@vger.kernel.org;
> sandals@crustytoothpaste.net; Johannes Schindelin
> <johannes.schindelin@gmx.de>; Eric Sunshine <sunshine@sunshineco.com>
> Subject: Re: [PATCH 2/2] http: add an "auto" mode for http.emptyauth
>
> On Thu, Feb 23, 2017 at 01:16:33AM +0000, David Turner wrote:
>
> > I don't know enough about how libcurl handles authentication to know
> > whether these patches are a good idea, but I have a minor comment
> anyway.
>
> As somebody who is using non-Basic auth, can you apply these patches and
> show us the output of:
>
> GIT_TRACE_CURL=1 \
> git ls-remote https://your-server 2>&1 >/dev/null |
> egrep '(Send|Recv) header: (GET|HTTP|Auth)'
>
> (without http.emptyauth turned on, obviously).
The results appear to be identical with and without
the patch. With http.emptyauth turned off,
16:27:28.208924 http.c:524 => Send header: GET /info/refs?service=git-upload-pack HTTP/1.1
16:27:28.212872 http.c:524 <= Recv header: HTTP/1.1 401 Authorization Required
Username for 'http://git': [I just pressed enter]
Password for 'http://git': [ditto]
16:27:29.928872 http.c:524 => Send header: GET /info/refs?service=git-upload-pack HTTP/1.1
16:27:29.929787 http.c:524 <= Recv header: HTTP/1.1 401 Authorization Required
(if someone else wants to replicate this, delete >/dev/null bit
from Jeff's shell snippet)
^ permalink raw reply
* SHAttered (the first practical SHA1 attack)
From: Santiago Torres @ 2017-02-23 15:50 UTC (permalink / raw)
To: Git
[-- Attachment #1: Type: text/plain, Size: 321 bytes --]
Hello all,
I ran into this website presenting the "first practical attack on
sha1"[1]. I don't recall seeing this on the ML, so I'm sharing this just
in case. I know there are proposals to move out of sha1 already. I
wonder if this affects the timeline for their adoption?
Thanks,
-Santiago.
[1] https://shattered.io/
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* RE: [RFC] Add support for downloading blobs on demand
From: Ben Peart @ 2017-02-23 15:39 UTC (permalink / raw)
To: 'Christian Couder'
Cc: 'Jeff King', 'git', 'Johannes Schindelin',
'Ben Peart'
In-Reply-To: <002701d2816e$f4682fa0$dd388ee0$@gmail.com>
I've completed the work of switching our read_object proposal to use a
background process (refactored from the LFS code) and have extricated it
from the rest of our GVFS fork so that it can be examined/tested
separately. It is currently based on a Git For Windows fork that I've
pushed to GitHub for anyone who is interested in viewing it at:
https://github.com/benpeart/git/tree/read-object-process
After some additional conversations with Christian, we're working to
combine our RFC/patch series into a single solution that should meet the
requirements of both.
The combined solution needs to have an "info" function which requests
info about a single object instead of a "have" function which must
return information on all objects the ODB knows as this doesn't scale
when the number of objects is large.
This means the "info" call has to be fast so spawning a process on every
call won't work. The background process with a versioned interface that
allows you to negotiate capabilities should solve this problem.
Ben
> -----Original Message-----
> From: Ben Peart [mailto:peartben@gmail.com]
> Sent: Tuesday, February 7, 2017 1:21 PM
> To: 'Christian Couder' <christian.couder@gmail.com>
> Cc: 'Jeff King' <peff@peff.net>; 'git' <git@vger.kernel.org>; 'Johannes
> Schindelin' <Johannes.Schindelin@gmx.de>; Ben Peart
> <benpeart@microsoft.com>
> Subject: RE: [RFC] Add support for downloading blobs on demand
>
> No worries about a late response, I'm sure this is the start of a long
> conversation. :)
>
> > -----Original Message-----
> > From: Christian Couder [mailto:christian.couder@gmail.com]
> > Sent: Sunday, February 5, 2017 9:04 AM
> > To: Ben Peart <peartben@gmail.com>
> > Cc: Jeff King <peff@peff.net>; git <git@vger.kernel.org>; Johannes
> > Schindelin <Johannes.Schindelin@gmx.de>
> > Subject: Re: [RFC] Add support for downloading blobs on demand
> >
> > (Sorry for the late reply and thanks to Dscho for pointing me to this
> > thread.)
> >
> > On Tue, Jan 17, 2017 at 10:50 PM, Ben Peart <peartben@gmail.com>
> wrote:
> > >> From: Jeff King [mailto:peff@peff.net] On Fri, Jan 13, 2017 at
> > >> 10:52:53AM -0500, Ben Peart wrote:
> > >>
> > >> > Clone and fetch will pass a --lazy-clone flag (open to a better
> > >> > name
> > >> > here) similar to --depth that instructs the server to only
> > >> > return commits and trees and to ignore blobs.
> > >> >
> > >> > Later during git operations like checkout, when a blob cannot be
> > >> > found after checking all the regular places (loose, pack,
> > >> > alternates, etc), git will download the missing object and place
> > >> > it into the local object store (currently as a loose object) then
> > >> > resume the
> > operation.
> > >>
> > >> Have you looked at the "external odb" patches I wrote a while ago,
> > >> and which Christian has been trying to resurrect?
> > >>
> > >>
> > >>
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpub
> > >> li
> > >> c-
> > >> inbox.org%2Fgit%2F20161130210420.15982-1-
> > >>
> >
> chriscool%40tuxfamily.org%2F&data=02%7C01%7CBen.Peart%40microsoft.c
> > >>
> >
> om%7C9596d3bf32564f123e0c08d43f08a9e1%7C72f988bf86f141af91ab2d7c
> > >>
> >
> d011db47%7C1%7C0%7C636202753822020527&sdata=a6%2BGOAQoRhjFoxS
> > >> vftY8JZAVUssmrXuDZ9OBy3xqNZk%3D&reserved=0
> > >>
> > >> This is a similar approach, though I pushed the policy for "how do
> > >> you get the objects" out into an external script. One advantage
> > >> there is that large objects could easily be fetched from another
> > >> source entirely (e.g., S3 or equivalent) rather than the repo itself.
> > >>
> > >> The downside is that it makes things more complicated, because a
> > >> push or a fetch now involves three parties (server, client, and the
> > >> alternate object store). So questions like "do I have all the
> > >> objects I need" are hard to reason about.
> > >>
> > >> If you assume that there's going to be _some_ central Git repo
> > >> which has all of the objects, you might as well fetch from there
> > >> (and do it over normal git protocols). And that simplifies things a
> > >> bit, at the cost of
> > being less flexible.
> > >
> > > We looked quite a bit at the external odb patches, as well as lfs
> > > and even using alternates. They all share a common downside that
> > > you must maintain a separate service that contains _some_ of the files.
> >
> > Pushing the policy for "how do you get the objects" out into an
> > external helper doesn't mean that the external helper cannot use the main
> service.
> > The external helper is still free to do whatever it wants including
> > calling the main service if it thinks it's better.
>
> That is a good point and you're correct, that means you can avoid having to
> build out multiple services.
>
> >
> > > These
> > > files must also be versioned, replicated, backed up and the service
> > > itself scaled out to handle the load. As you mentioned, having
> > > multiple services involved increases flexability but it also
> > > increases the complexity and decreases the reliability of the
> > > overall version control service.
> >
> > About reliability, I think it depends a lot on the use case. If you
> > want to get very big files over an unreliable connection, it can
> > better if you send those big files over a restartable protocol and
> > service like HTTP/S on a regular web server.
> >
>
> My primary concern about reliability was the multiplicative effect of making
> multiple requests across multiple servers to complete a single request.
> Having putting this all in a single service like you suggested above brings us
> back to parity on the complexity.
>
> > > For operational simplicity, we opted to go with a design that uses a
> > > single, central git repo which has _all_ the objects and to focus on
> > > enhancing it to handle large numbers of files efficiently. This
> > > allows us to focus our efforts on a great git service and to avoid
> > > having to build out these other services.
> >
> > Ok, but I don't think it prevents you from using at least some of the
> > same mechanisms that the external odb series is using.
> > And reducing the number of mechanisms in Git itself is great for its
> > maintainability and simplicity.
>
> I completely agree with the goal of reducing the number of mechanisms in
> Git itself. Our proposal is primarily targeting speeding operations when
> dealing with large numbers of files. ObjectDB is primarily targeting large
> objects but there is a lot of similarity in how we're approaching the solution.
> I hope/believe we can come to a common solution that will solve both.
>
> >
> > >> > To prevent git from accidentally downloading all missing blobs,
> > >> > some git operations are updated to be aware of the potential for
> > missing blobs.
> > >> > The most obvious being check_connected which will return success
> > >> > as if everything in the requested commits is available locally.
> > >>
> > >> Actually, Git is pretty good about trying not to access blobs when
> > >> it doesn't need to. The important thing is that you know enough
> > >> about the blobs to fulfill has_sha1_file() and sha1_object_info()
> > >> requests without actually fetching the data.
> > >>
> > >> So the client definitely needs to have some list of which objects
> > >> exist, and which it _could_ get if it needed to.
> >
> > Yeah, and the external odb series handles that already, thanks to
> > Peff's initial work.
> >
>
> I'm currently working on a patch series that will reimplement our current
> read-object hook to use the LFS model for long running background
> processes. As part of that, I am building a versioned interface that will
> support multiple commands (like get, have, put). In my initial
> implementation, I'm only supporting the "get" verb as that is what we
> currently need but my intent is to build it so that we could add have and put
> in future versions. When I have the first iteration ready, I'll push it up to our
> fork on github for review as code is clearer than my description in email.
>
> Moving forward, the "have" verb is a little problematic as we would "have"
> 3+ million shas that we'd be required to fetch from the server and then pass
> along to git when requested. It would be nice to come up with a way to
> avoid or reduce that cost.
>
> > >> The one place you'd probably want to tweak things is in the diff
> > >> code, as a single "git log -Sfoo" would fault in all of the blobs.
> > >
> > > It is an interesting idea to explore how we could be smarter about
> > > preventing blobs from faulting in if we had enough info to fulfill
> > > has_sha1_file() and sha1_object_info(). Given we also heavily prune
> > > the working directory using sparse-checkout, this hasn't been our
> > > top focus but it is certainly something worth looking into.
> >
> > The external odb series doesn't handle preventing blobs from faulting
> > in yet, so this could be a common problem.
> >
>
> Agreed. This is one we've been working on quite a bit out of necessity. If
> you look at our patch series, most of the changes are related to dealing with
> missing objects.
>
> > [...]
> >
> > >> One big hurdle to this approach, no matter the protocol, is how you
> > >> are going to handle deltas. Right now, a git client tells the
> > >> server "I have this commit, but I want this other one". And the
> > >> server knows which objects the client has from the first, and which
> > >> it needs from the second. Moreover, it knows that it can send
> > >> objects in delta form directly from disk if the other side has the delta
> base.
> > >>
> > >> So what happens in this system? We know we don't need to send any
> > >> blobs in a regular fetch, because the whole idea is that we only
> > >> send blobs on demand. So we wait for the client to ask us for blob
> > >> A. But then what do we send? If we send the whole blob without
> > >> deltas, we're going to waste a lot of bandwidth.
> > >>
> > >> The on-disk size of all of the blobs in linux.git is ~500MB. The
> > >> actual data size is ~48GB. Some of that is from zlib, which you get
> > >> even for non-deltas. But the rest of it is from the delta
> > >> compression. I don't think it's feasible to give that up, at least
> > >> not for "normal" source repos like linux.git (more on that in a minute).
> > >>
> > >> So ideally you do want to send deltas. But how do you know which
> > >> objects the other side already has, which you can use as a delta
> > >> base? Sending the list of "here are the blobs I have" doesn't scale.
> > >> Just the sha1s start to add up, especially when you are doing
> > >> incremental
> > fetches.
> >
> > To initialize some paths that the client wants, it could perhaps just
> > ask for some pack files, or maybe bundle files, related to these paths.
> > Those packs or bundles could be downloaded either directly from the
> > main server or from other web or proxy servers.
> >
> > >> I think this sort of things performs a lot better when you just
> > >> focus on large objects. Because they don't tend to delta well
> > >> anyway, and the savings are much bigger by avoiding ones you don't
> > >> want. So a directive like "don't bother sending blobs larger than
> > >> 1MB" avoids a lot of these issues. In other words, you have some
> > >> quick shorthand to communicate between the client and server: this
> > >> what I have, and what I
> > don't.
> > >> Normal git relies on commit reachability for that, but there are
> > >> obviously other dimensions. The key thing is that both sides be
> > >> able to express the filters succinctly, and apply them efficiently.
> > >
> > > Our challenge has been more the sheer _number_ of files that exist
> > > in the repo rather than the _size_ of the files in the repo. With
> > > >3M source files and any typical developer only needing a small
> > > percentage of those files to do their job, our focus has been
> > > pruning the tree as much as possible such that they only pay the
> > > cost for the files they actually need. With typical text source
> > > files being 10K - 20K in size, the overhead of the round trip is a
> > > significant part of the overall transfer time so deltas don't help
> > > as much. I agree that large files are also a problem but it isn't my top
> focus at this point in time.
> >
> > Ok, but it would be nice if both problems could be solved using some
> > common mechanisms.
> > This way it could probably work better in situations where there are
> > both a large number of files _and_ some big files.
> > And from what I am seeing, there could be no real downside from using
> > some common mechanisms.
> >
>
> Agree completely. I'm hopeful that we can come up with some common
> mechanisms that will allow us to solve both problems.
>
> > >> If most of your benefits are not from avoiding blobs in general,
> > >> but rather just from sparsely populating the tree, then it sounds
> > >> like sparse clone might be an easier path forward. The general idea
> > >> is to restrict not just the checkout, but the actual object
> > >> transfer and reachability (in the tree dimension, the way shallow
> > >> clone limits it in the time dimension, which will require
> > >> cooperation between the client
> > and server).
> > >>
> > >> So that's another dimension of filtering, which should be expressed
> > >> pretty
> > >> succinctly: "I'm interested in these paths, and not these other
> > >> ones." It's pretty easy to compute on the server side during graph
> > >> traversal (though it interacts badly with reachability bitmaps, so
> > >> there would need to be some hacks there).
> > >>
> > >> It's an idea that's been talked about many times, but I don't
> > >> recall that there were ever working patches. You might dig around
> > >> in the list archive under the name "sparse clone" or possibly "narrow
> clone".
> > >
> > > While a sparse/narrow clone would work with this proposal, it isn't
> > > required. You'd still probably want all the commits and trees but
> > > the clone would also bring down the specified blobs. Combined with
> > > using "depth" you could further limit it to those blobs at tip.
> > >
> > > We did run into problems with this model however as our usage
> > > patterns are such that our working directories often contain very
> > > sparse trees and as a result, we can end up with thousands of
> > > entries in the sparse checkout file. This makes it difficult for
> > > users to manually specify a sparse-checkout before they even do a
> > > clone. We have implemented a hashmap based sparse-checkout to deal
> > > with the performance issues of having that many entries but that's a
> > > different RFC/PATCH. In short, we found that a "lazy-clone" and
> > > downloading blobs on demand provided a better developer experience.
> >
> > I think both ways are possible using the external odb mechanism.
> >
> > >> > Future Work
> > >> > ~~~~~~~~~~~
> > >> >
> > >> > The current prototype calls a new hook proc in
> > >> > sha1_object_info_extended and read_object, to download each
> > >> > missing blob. A better solution would be to implement this via a
> > >> > long running process that is spawned on the first download and
> > >> > listens for requests to download additional objects until it
> > >> > terminates when the parent git operation exits (similar to the
> > >> > recent long running smudge and clean filter
> > >> work).
> > >>
> > >> Yeah, see the external-odb discussion. Those prototypes use a
> > >> process per object, but I think we all agree after seeing how the
> > >> git-lfs interface has scaled that this is a non-starter. Recent
> > >> versions of git-lfs do the single- process thing, and I think any
> > >> sort of external-odb hook should be modeled on that protocol.
> >
> > I agree that the git-lfs scaling work is great, but I think it's not
> > necessary in the external odb work to have the same kind of
> > single-process protocol from the beginning (though it should be possible
> and easy to add it).
> > For example if the external odb work can be used or extended to handle
> > restartable clone by downloading a single bundle when cloning, this
> > would not need that kind of protocol.
> >
> > > I'm looking into this now and plan to re-implement it this way
> > > before sending out the first patch series. Glad to hear you think
> > > it is a good protocol to model it on.
> >
> > Yeah, for your use case on Windows, it looks really worth it to use
> > this kind of protocol.
> >
> > >> > Need to investigate an alternate batching scheme where we can
> > >> > make a single request for a set of "related" blobs and receive
> > >> > single a packfile (especially during checkout).
> > >>
> > >> I think this sort of batching is going to be the really hard part
> > >> to retrofit onto git. Because you're throwing out the procedural
> > >> notion that you can loop over a set of objects and ask for each
> individually.
> > >> You have to start deferring computation until answers are ready.
> > >> Some operations can do that reasonably well (e.g., checkout), but
> > >> something like "git log -p" is constantly digging down into history.
> > >> I suppose you could just perform the skeleton of the operation
> > >> _twice_, once to find the list of objects to fault in, and the
> > >> second time to
> > actually do it.
> >
> > In my opinion, perhaps we can just prevent "git log -p" from faulting
> > in blobs and have it show a warning saying that it was performed only
> > on a subset of all the blobs.
> >
>
> You might be surprised at how many other places end up faulting in blobs. :)
> Rename detection is one we've recently been working on.
>
> > [...]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox