git-retry tool or git.retry config (built-in implementation)?

git-retry tool or git.retry config (built-in implementation)?

[Yaroslav Halchenko]

Dear Git Gurus,

In DataLad (https://datalad.org) we are doing lots of automated cloning,
fetching etc as part of our CI etc jobs.  Once in a while git operations
fail [see e.g. 1], and beg us to retry but we need to know when to
do so, and not do it upon every failed git invocation since some
failures could be legit (repository is gone). While looking how others
solve it we found
https://stackoverflow.com/questions/35014012/git-retry-if-http-request-failed
which pointed to tools like git-retry and later part of
https://chromium.googlesource.com/infra/infra/+/HEAD/go/src/infra/tools/git/retry_regexp.go
which serve as a collection of regexes to be on lookout for to retry.

Would that be the "best" strategy currently?  

As regex matching might eventually break whenever `git` changes
anything in the output messages, I wondered if there could be a more
robust internal implementation in git itself?  Similarly git-annex has
annex.retry config setting which sets the count of retries for
"retriable" operations.  

Do you think something like that could be implemented in git in some
foreseable future?

[1] https://github.com/datalad/datalad/issues/7485

Thank you in advance!
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
WWW:   http://www.linkedin.com/in/yarik        

Re: git-retry tool or git.retry config (built-in implementation)?

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 1808 bytes --]

On Tue, Sep 26, 2023 at 12:47:51PM -0400, Yaroslav Halchenko wrote:
> Dear Git Gurus,
> 
> In DataLad (https://datalad.org) we are doing lots of automated cloning,
> fetching etc as part of our CI etc jobs.  Once in a while git operations
> fail [see e.g. 1], and beg us to retry but we need to know when to
> do so, and not do it upon every failed git invocation since some
> failures could be legit (repository is gone). While looking how others
> solve it we found
> https://stackoverflow.com/questions/35014012/git-retry-if-http-request-failed
> which pointed to tools like git-retry and later part of
> https://chromium.googlesource.com/infra/infra/+/HEAD/go/src/infra/tools/git/retry_regexp.go
> which serve as a collection of regexes to be on lookout for to retry.
> 
> Would that be the "best" strategy currently?  

Looking at the actual git_retry.py script [1], it really just wraps
actual Git commands. IMO, git-retry(1) shell script as you mentioned
only calls the python version, which adds another level of indirection
(why not doing it in pure shell?).

AFAIK, to solve the retrying problem, we need to have a way to tell
transport backend (curl/ssh) to resume transfer from the faulty point.

> 
> As regex matching might eventually break whenever `git` changes
> anything in the output messages, I wondered if there could be a more
> robust internal implementation in git itself?  Similarly git-annex has
> annex.retry config setting which sets the count of retries for
> "retriable" operations.  

Do you use porcelain interfaces instead of plumbing ones?

Thanks.

[1]: https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/01d2cde990f22d409e74e239de7e4d347102d6f6/git_retry.py

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: git-retry tool or git.retry config (built-in implementation)?

[Yaroslav Halchenko]

On Wed, 27 Sep 2023, Bagas Sanjaya wrote:

> On Tue, Sep 26, 2023 at 12:47:51PM -0400, Yaroslav Halchenko wrote:
> > Dear Git Gurus,

> > In DataLad (https://datalad.org) we are doing lots of automated cloning,
> > fetching etc as part of our CI etc jobs.  Once in a while git operations
> > fail [see e.g. 1], and beg us to retry but we need to know when to
> > do so, and not do it upon every failed git invocation since some
> > failures could be legit (repository is gone). While looking how others
> > solve it we found
> > https://stackoverflow.com/questions/35014012/git-retry-if-http-request-failed
> > which pointed to tools like git-retry and later part of
> > https://chromium.googlesource.com/infra/infra/+/HEAD/go/src/infra/tools/git/retry_regexp.go
> > which serve as a collection of regexes to be on lookout for to retry.

> > Would that be the "best" strategy currently?  

> Looking at the actual git_retry.py script [1], it really just wraps
> actual Git commands. IMO, git-retry(1) shell script as you mentioned
> only calls the python version, which adds another level of indirection
> (why not doing it in pure shell?).

My guess would be that it is just easier to code in Python usually for
such cases with a "registry" of hits etc.  But why not just to strip .py
from python script which has shebang already and not require bash
wrapper at all? ;)

> AFAIK, to solve the retrying problem, we need to have a way to tell
> transport backend (curl/ssh) to resume transfer from the faulty point.

some times it seems not even getting connected (https) entirely and that
(?) leading to error in the caller above, e.g. from
https://github.com/datalad/datalad/issues/7485#issuecomment-1735619755

error: Failed to connect to datasets-tests.datalad.org port 443 after 8291 ms: Couldn't connect to server (curl_result = 28, http_code = 0, sha1 = 3980af8de56946a10ff5c48879e5d6025965d936)\nerror: Unable to find 3980af8de56946a10ff5c48879e5d6025965d936 under ...


> > As regex matching might eventually break whenever `git` changes
> > anything in the output messages, I wondered if there could be a more
> > robust internal implementation in git itself?  Similarly git-annex has
> > annex.retry config setting which sets the count of retries for
> > "retriable" operations.  

> Do you use porcelain interfaces instead of plumbing ones?

I would say -- a "mix". 

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
WWW:   http://www.linkedin.com/in/yarik        

Re: git-retry tool or git.retry config (built-in implementation)?

[Bagas Sanjaya]

On 27/09/2023 19:42, Yaroslav Halchenko wrote:
> 
> On Wed, 27 Sep 2023, Bagas Sanjaya wrote:
> 
>> On Tue, Sep 26, 2023 at 12:47:51PM -0400, Yaroslav Halchenko wrote:
>>> Dear Git Gurus,
> 
>>> In DataLad (https://datalad.org) we are doing lots of automated cloning,
>>> fetching etc as part of our CI etc jobs.  Once in a while git operations
>>> fail [see e.g. 1], and beg us to retry but we need to know when to
>>> do so, and not do it upon every failed git invocation since some
>>> failures could be legit (repository is gone). While looking how others
>>> solve it we found
>>> https://stackoverflow.com/questions/35014012/git-retry-if-http-request-failed
>>> which pointed to tools like git-retry and later part of
>>> https://chromium.googlesource.com/infra/infra/+/HEAD/go/src/infra/tools/git/retry_regexp.go
>>> which serve as a collection of regexes to be on lookout for to retry.
> 
>>> Would that be the "best" strategy currently?  
> 
>> Looking at the actual git_retry.py script [1], it really just wraps
>> actual Git commands. IMO, git-retry(1) shell script as you mentioned
>> only calls the python version, which adds another level of indirection
>> (why not doing it in pure shell?).
> 
> My guess would be that it is just easier to code in Python usually for
> such cases with a "registry" of hits etc.  But why not just to strip .py
> from python script which has shebang already and not require bash
> wrapper at all? ;)
> 
>> AFAIK, to solve the retrying problem, we need to have a way to tell
>> transport backend (curl/ssh) to resume transfer from the faulty point.
> 
> some times it seems not even getting connected (https) entirely and that
> (?) leading to error in the caller above, e.g. from
> https://github.com/datalad/datalad/issues/7485#issuecomment-1735619755
> 
> error: Failed to connect to datasets-tests.datalad.org port 443 after 8291 ms: Couldn't connect to server (curl_result = 28, http_code = 0, sha1 = 3980af8de56946a10ff5c48879e5d6025965d936)\nerror: Unable to find 3980af8de56946a10ff5c48879e5d6025965d936 under ...
> 

Then the question is how can the server cache generated on-the-fly bundles
in hope that future clients can resume the transfer.
 
> 
>>> As regex matching might eventually break whenever `git` changes
>>> anything in the output messages, I wondered if there could be a more
>>> robust internal implementation in git itself?  Similarly git-annex has
>>> annex.retry config setting which sets the count of retries for
>>> "retriable" operations.  
> 
>> Do you use porcelain interfaces instead of plumbing ones?
> 
> I would say -- a "mix". 
> 

Junio, what do you think of above ideas from Yaroslav?

-- 
An old man doll... just what I always wanted! - Clara

Re: git-retry tool or git.retry config (built-in implementation)?

[Meiswinkel, Jan SF/HZA-ZC3S]

(let's try this again - had html enabled)

Hi,

I think this thread might have died - a retry-logic for git commands would be great. IMO it is not even needed to continue the transfer from the faulty point, it would be a great benefit already if it dumps the failed request and simply retries the request X times with a backoff factor (just as the user would do manually if his request failed).

Any hopes?

Best
Jan