Fwd: git-daemon access-hook race condition

Fwd: git-daemon access-hook race condition

[Eugene Sajine]

Hi,


We are serving repos in closed netwrok via git protocol. We are using
git-daemon access hook (thank you very much for such a great feature)
in order to create push notifications for Jenkins.
I.e. upon the push the access-hook is called and then the curl command
is created and executed. As we have several instances of Jenkins, that
we need to notify (three), the execution of the access-hook can take
some time.

Sometimes we have a situation when the whole chain works fine but
Jenkins git plugin doesn't recognize the changes. I think it happens
because we hit a kind of race condition:

1. Incoming push triggers access-hook
2. notify jenkins 1
3. notify jenkins 2
4. jenkins 1 polls repo but sees no changes
5. notify Jenkins 3
6. the push data transfer finishes - consequent pushes will find
changes w/o any problem

The question is:

Is there a way to avoid that?
Is it possible to have access-hook to be executed after receive?
Is it possible to introduce a parameter that would specify if it needs
to be executed before receive or after?

Thanks,
Eugene

Re: Fwd: git-daemon access-hook race condition

[Junio C Hamano]

Eugene Sajine <euguess@gmail.com> writes:

> Is it possible to have access-hook to be executed after receive?

The whole point of access-hook is to allow it to decide whether the
access is allowed or not, so that is a non-starter.

A notification _after_ successful push update is usually done via
the post-receive hook in the receiving repository, I think.

Fwd: Fwd: git-daemon access-hook race condition

[Eugene Sajine]

On Thu, Sep 12, 2013 at 3:15 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Eugene Sajine <euguess@gmail.com> writes:
>
>> Is it possible to have access-hook to be executed after receive?
>
> The whole point of access-hook is to allow it to decide whether the
> access is allowed or not, so that is a non-starter.
>
> A notification _after_ successful push update is usually done via
> the post-receive hook in the receiving repository, I think.


Junio,

Thanks for the reply!

This is interesting: i always thought about the access-hook as
something to be executed when the repo is accessed, not just
verification if access is allowed - your definition is much more
limiting.

we have about 1400 bare repos - so i would like to avoid the
configuration of each one of them. I could probably find a way to
automate it, but already having access-hook in current implementation
makes me reluctant to go this way, because it is so much easier to use
centralized manager.

So are you really sure that it is a non-starter to have
--before-service/--after-service options for access-hook?

Thanks,
Eugene

Re: Fwd: Fwd: git-daemon access-hook race condition

[Junio C Hamano]

Eugene Sajine <euguess@gmail.com> writes:

> So are you really sure that it is a non-starter to have
> --before-service/--after-service options for access-hook?

Given the definition of "--access-hook" in "git help daemon":

    --access-hook=<path>::
            Every time a client connects, first run an external command
            specified by the <path> ... The external command can decide
            to decline the service by exiting with a non-zero status (or
            to allow it by exiting with a zero status)....

There is *NO* way in anywhere --after-service makes any sense (and
by definition --before-service is redundant).

What you _could_ propose is to define a *new* hook that is run when
the spawned service has returned, with the same information that is
fed to the access hook (possibly with its exit status).

I do not offhand know if we retain the original service information
that long after the main daemon process has spawned the service
process, though.  With the current system, the only thing it needs
to know is the PID of the service processes that are to be culled by
calls to waitpid().  So you may have to extend existing bookkeeping
data structures a bit to keep those pieces of information around if
you wanted to add such a new hook.

Re: Fwd: Fwd: git-daemon access-hook race condition

[Eugene Sajine]

Junio,

Thanks for the clarification! Your solution does look better.

For now though i think i will have to delay the notification somehow
and let the service finish first then notify the server.

Thanks again!

Eugene


On Thu, Sep 12, 2013 at 5:08 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Eugene Sajine <euguess@gmail.com> writes:
>
>> So are you really sure that it is a non-starter to have
>> --before-service/--after-service options for access-hook?
>
> Given the definition of "--access-hook" in "git help daemon":
>
>     --access-hook=<path>::
>             Every time a client connects, first run an external command
>             specified by the <path> ... The external command can decide
>             to decline the service by exiting with a non-zero status (or
>             to allow it by exiting with a zero status)....
>
> There is *NO* way in anywhere --after-service makes any sense (and
> by definition --before-service is redundant).
>
> What you _could_ propose is to define a *new* hook that is run when
> the spawned service has returned, with the same information that is
> fed to the access hook (possibly with its exit status).
>
> I do not offhand know if we retain the original service information
> that long after the main daemon process has spawned the service
> process, though.  With the current system, the only thing it needs
> to know is the PID of the service processes that are to be culled by
> calls to waitpid().  So you may have to extend existing bookkeeping
> data structures a bit to keep those pieces of information around if
> you wanted to add such a new hook.
>
>

Re: Fwd: Fwd: git-daemon access-hook race condition

[Eugene Sajine]

On Thu, Sep 12, 2013 at 5:16 PM, Eugene Sajine <euguess@gmail.com> wrote:
> Junio,
>
> Thanks for the clarification! Your solution does look better.
>
> For now though i think i will have to delay the notification somehow
> and let the service finish first then notify the server.
>
> Thanks again!
>
> Eugene
>
>
> On Thu, Sep 12, 2013 at 5:08 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Eugene Sajine <euguess@gmail.com> writes:
>>
>>> So are you really sure that it is a non-starter to have
>>> --before-service/--after-service options for access-hook?
>>
>> Given the definition of "--access-hook" in "git help daemon":
>>
>>     --access-hook=<path>::
>>             Every time a client connects, first run an external command
>>             specified by the <path> ... The external command can decide
>>             to decline the service by exiting with a non-zero status (or
>>             to allow it by exiting with a zero status)....
>>
>> There is *NO* way in anywhere --after-service makes any sense (and
>> by definition --before-service is redundant).
>>
>> What you _could_ propose is to define a *new* hook that is run when
>> the spawned service has returned, with the same information that is
>> fed to the access hook (possibly with its exit status).
>>
>> I do not offhand know if we retain the original service information
>> that long after the main daemon process has spawned the service
>> process, though.  With the current system, the only thing it needs
>> to know is the PID of the service processes that are to be culled by
>> calls to waitpid().  So you may have to extend existing bookkeeping
>> data structures a bit to keep those pieces of information around if
>> you wanted to add such a new hook.
>>
>>

For now I'm trying to do the following:

access-hook.bash has:

delayed-notify.bash $@ &

delayed-notify.bash has:

sleep 10
...
curl ...

I'm expecting access-hook to spawn new process and return without
waiting for it to finish to let the service to do its job. But when i
do push - it sleeps for 10 seconds anyway. Am i missing something
obvious here?

Any help is much appreciated!

Thanks,
Eugene

Re: Fwd: Fwd: git-daemon access-hook race condition

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Eugene Sajine <euguess@gmail.com> writes:
>
>> So are you really sure that it is a non-starter to have
>> --before-service/--after-service options for access-hook?
>
> Given the definition of "--access-hook" in "git help daemon":
>
>     --access-hook=<path>::
>             Every time a client connects, first run an external command
>             specified by the <path> ... The external command can decide
>             to decline the service by exiting with a non-zero status (or
>             to allow it by exiting with a zero status)....
>
> There is *NO* way in anywhere --after-service makes any sense (and
> by definition --before-service is redundant).
>
> What you _could_ propose is to define a *new* hook that is run when
> the spawned service has returned, with the same information that is
> fed to the access hook (possibly with its exit status).

Scratch that "exit status" part, as I do not think it is useful.

To a receive-pack and a send-pack that is talking to it, if a push
results in a failure, it is a failure.  Likewise for upload-pack and
fetch-pack for a transfer in the reverse direction.

And the way that failure is communicated from the receive-pack to
the end-user via the send-pack is for the receive-pack to send a
protocol message that tells the send-pack about the failure, and the
send-pack showing the error message and signalling the failure with
its exit status.  Likewise for upload-pack and fetch-pack (hence
"fetch", which is conceptually a thin wrapper around it).

Between the deamon and the receive-pack (or the fetch-pack) process,
however, such a failed push (or fetch) is still a success.  "I
correctly diagnosed and successfully sent a rejection notice to the
other end" is signalled by receive-pack to the daemon by exiting
with success (i.e. 0) exit status.

So even if we feed the exit status of the service process to the
hook script specified by the --post-service-hook, it does not tell
the script if the service "succeeded" in that sense.

Re: Fwd: Fwd: git-daemon access-hook race condition

[Eugene Sajine]

On Thu, Sep 12, 2013 at 6:20 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Junio C Hamano <gitster@pobox.com> writes:
>
>> Eugene Sajine <euguess@gmail.com> writes:
>>
>>> So are you really sure that it is a non-starter to have
>>> --before-service/--after-service options for access-hook?
>>
>> Given the definition of "--access-hook" in "git help daemon":
>>
>>     --access-hook=<path>::
>>             Every time a client connects, first run an external command
>>             specified by the <path> ... The external command can decide
>>             to decline the service by exiting with a non-zero status (or
>>             to allow it by exiting with a zero status)....
>>
>> There is *NO* way in anywhere --after-service makes any sense (and
>> by definition --before-service is redundant).
>>
>> What you _could_ propose is to define a *new* hook that is run when
>> the spawned service has returned, with the same information that is
>> fed to the access hook (possibly with its exit status).
>
> Scratch that "exit status" part, as I do not think it is useful.
>
> To a receive-pack and a send-pack that is talking to it, if a push
> results in a failure, it is a failure.  Likewise for upload-pack and
> fetch-pack for a transfer in the reverse direction.
>
> And the way that failure is communicated from the receive-pack to
> the end-user via the send-pack is for the receive-pack to send a
> protocol message that tells the send-pack about the failure, and the
> send-pack showing the error message and signalling the failure with
> its exit status.  Likewise for upload-pack and fetch-pack (hence
> "fetch", which is conceptually a thin wrapper around it).
>
> Between the deamon and the receive-pack (or the fetch-pack) process,
> however, such a failed push (or fetch) is still a success.  "I
> correctly diagnosed and successfully sent a rejection notice to the
> other end" is signalled by receive-pack to the daemon by exiting
> with success (i.e. 0) exit status.
>
> So even if we feed the exit status of the service process to the
> hook script specified by the --post-service-hook, it does not tell
> the script if the service "succeeded" in that sense.


I see what you're saying.
In my particular use case I can work around that service status
because even if it failed it will just trigger Jenkins to poll and in
case of failure to transfer data there will be no new changes for
Jenkins to work with. If we would want the --post-service-hook to know
that data transfer succeeded or failed, then may be there should be
some difference between "service status" and "service process status"?
In this case the existing logic works with "service process status"
while the --post-service-hook is fed with the "service status" (or
name it "data transfer status")

Do i make any sense?

Re: Fwd: Fwd: git-daemon access-hook race condition

[Junio C Hamano]

Eugene Sajine <euguess@gmail.com> writes:

>> So even if we feed the exit status of the service process to the
>> hook script specified by the --post-service-hook, it does not tell
>> the script if the service "succeeded" in that sense.
>
> I see what you're saying.
> In my particular use case I can work around that service status
> because even if it failed it will just trigger Jenkins to poll and in
> case of failure to transfer data there will be no new changes for
> Jenkins to work with. If we would want the --post-service-hook to know
> that data transfer succeeded or failed, then may be there should be
> some difference between "service status" and "service process status"?
> In this case the existing logic works with "service process status"
> while the --post-service-hook is fed with the "service status" (or
> name it "data transfer status")
>
> Do i make any sense?

Almost; you missed that there is no channel to pass "data transfer
status" from the service back to the daemon.

Re: Fwd: Fwd: git-daemon access-hook race condition

[Eugene Sajine]

>
> For now I'm trying to do the following:
>
> access-hook.bash has:
>
> delayed-notify.bash $@ &
>
> delayed-notify.bash has:
>
> sleep 10
> ...
> curl ...
>
> I'm expecting access-hook to spawn new process and return without
> waiting for it to finish to let the service to do its job. But when i
> do push - it sleeps for 10 seconds anyway. Am i missing something
> obvious here?
>
> Any help is much appreciated!
>
> Thanks,
> Eugene


I found a following solution to make it happen while waiting for
somebody to be generous enough to take on the --post-service-hook
(unfortunately i'm not a C guy):

It is using 'at' command. The access-hook script has:

echo "delayed-notify.bash $@" | at now

while delayed-notify.bash has:

sleep 10
curl ...

This is not perfect and in certain situations can still fail because
the delay is not long enough but this will at least resolve 90% of
issues.

I hope that might be helpful for someone.

Thanks,
Eugene