Detecting pushes originating from shallow clones?

Detecting pushes originating from shallow clones?

[Owen Jacobson]

[-- Attachment #1: Type: text/plain, Size: 1066 bytes --]

Hi folks,

Here at Heroku, we ingest code from users by supplying them with a Git server they can `git push` into. Occasionally, users will attempt to push to us from a shallow clone, which causes numerous problems for us and, often, for our users (incomplete repositories, gaps in history, failed builds, and a host of other problems). We’ve been investigating what we can do to give these users clearer advice on how to fix the problem.

Within the constraints that

* we cannot control which version of Git our users have installed, and
* we run Git v1.9.1, obtained from the Ubuntu 14.04 LTS .deb repositories

what can we do in an update/pre-receive hook to detect that an incoming push originates from a shallow repository and reject it?

Right now, the best strategy we have is to observe whether

    git rev-list OLD NEW

fails, and if it does fail, whether the stderr output includes the phrase "revision walk setup failed”. This feels like a fairly weak fix, so I thought I’d see if anyone here has a better idea.

Thanks,

-o


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: Detecting pushes originating from shallow clones?

[Junio C Hamano]

Owen Jacobson <ojacobson@heroku.com> writes:

> Hi folks,
>
> Here at Heroku, we ingest code from users by supplying them with a
> Git server they can `git push` into. Occasionally, users will
> attempt to push to us from a shallow clone, which causes numerous
> problems for us and, often, for our users (incomplete
> repositories, gaps in history, failed builds, and a host of other
> problems). We’ve been investigating what we can do to give these
> users clearer advice on how to fix the problem.
>
> Within the constraints that
>
> * we cannot control which version of Git our users have installed, and
> * we run Git v1.9.1, obtained from the Ubuntu 14.04 LTS .deb repositories
>
> what can we do in an update/pre-receive hook to detect that an
> incoming push originates from a shallow repository and reject it?

Hmm, I would have suggested to set receive.fsckObjects which has
been around since early 2008 (it is in v1.5.6, so it would likely be
in v1.9.1 as well).

But even without that configuration set, "push" shouldn't leave the
receiving repository in an inconsistent state (e.g. incomplete
repository, gaps in history) in the first place.

Does anybody recall us having such a bug in the distant past in
1.9.1 and fixing it?  I do not offhand recall but I wouldn't be
surprised.

> Right now, the best strategy we have is to observe whether
>
>     git rev-list OLD NEW
>
> fails, and if it does fail, whether the stderr output includes the
> phrase "revision walk setup failed”. This feels like a fairly
> weak fix ...

Actually, that (with "--objects" option) is essentially the test the
receiving end does internally to detect the "gaps in history" when
receive.fsckObjects configuration is set.

Re: Detecting pushes originating from shallow clones?

[Owen Jacobson]

[-- Attachment #1: Type: text/plain, Size: 2624 bytes --]

Owen Jacobson <ojacobson@heroku.com> wrote:

> Within the constraints that
> 
> * we cannot control which version of Git our users have installed, and
> * we run Git v1.9.1, obtained from the Ubuntu 14.04 LTS .deb repositories
> 
> what can we do in an update/pre-receive hook to detect that an
> incoming push originates from a shallow repository and reject it?

Junio C Hamano <gitster@pobox.com> wrote:

> Hmm, I would have suggested to set receive.fsckObjects which has
> been around since early 2008 (it is in v1.5.6, so it would likely be
> in v1.9.1 as well).
> 
> But even without that configuration set, "push" shouldn't leave the
> receiving repository in an inconsistent state (e.g. incomplete
> repository, gaps in history) in the first place.
> 
> Does anybody recall us having such a bug in the distant past in
> 1.9.1 and fixing it?  I do not offhand recall but I wouldn't be
> surprised.

It’s possible that that would fix the problem.

We do see pushes get rejected occasionally with output similar to

    To git@heroku.com:example.git
     ! [remote rejected] master -> master (missing necessary objects)

(We rely on customers reporting it to us, since right now we have no good way to spot this ourselves — we don’t get the output.)

However, this happens -after- the pre-receive and update hooks have already run. We’ve historically assumed that the hooks are the last step prior to accepting a push, so we assume that if the hook passes, then it’s safe to take side-effect-ful actions like deploying the newly-pushed branch tip to servers.

Git’s native error message also provides very little guidance on how users should resolve the problem. While I don’t think that’s Git’s responsibility in this case, it’d be nice to be able to insert our own messaging here, or to detect the problem earlier.

Owen Jacobson <ojacobson@heroku.com> wrote:

> Right now, the best strategy we have is to observe whether
> 
>    git rev-list OLD NEW
> 
> fails, and if it does fail, whether the stderr output includes the
> phrase "revision walk setup failed”. This feels like a fairly
> weak fix …

Junio C Hamano <gitster@pobox.com> wrote:

> Actually, that (with "--objects" option) is essentially the test the
> receiving end does internally to detect the "gaps in history" when
> receive.fsckObjects configuration is set.

Hm. Well, if it’s essentially the same check, then perhaps running it ourselves in our hooks is the right fix.

Thanks for the input! I’m also interested in any other thoughts the list might have.

-o


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 801 bytes --]