Fwd: Git credentials not working

Fwd: Git credentials not working

[Dimitri Kopriwa]

Dear Git list,


I have tried to used git credentials within Gitlab-CI runners. I have 4 
instance of GitLab and discovered a weird bug with Git credentials when 
use within a CI process.

Please note before all that the time spend allowed me multiple time to 
check that my credentials are valid for the repository. And calling git 
fetch --tags with the full remote url that include the credentials 
always succeeded.

Tested with Git 2.11, 2.19

Git credentials in ~/.git-credentials and ~/.config/git/credentials are 
being removed by git upon reading.

This happen randomly accross my CI runner, and change that make them 
work on not related.


{ Error: Command failed: git fetch --tags 
https://git.example.com/example/some-project.git
18:25:52.554903 git.c:415               trace: built-in: git fetch 
--tags https://git.example.com/example/some-project.git
18:25:52.555234 run-command.c:637       trace: run_command: GIT_DIR=.git 
git-remote-https https://git.example.com/example/some-project.git 
https://git.example.com/example/some-project.git
18:25:52.692741 run-command.c:637       trace: run_command: 'git 
credential-store get'
18:25:52.697314 git.c:659               trace: exec: 
git-credential-store get
18:25:52.697372 run-command.c:637       trace: run_command: 
git-credential-store get
18:25:52.936024 run-command.c:637       trace: run_command: 'git 
credential-store erase'
18:25:52.940307 git.c:659               trace: exec: 
git-credential-store erase
18:25:52.940365 run-command.c:637       trace: run_command: 
git-credential-store erase
remote: HTTP Basic: Access denied
fatal: Authentication failed for 
'https://git.example.com/example/some-project.git/'


See the full question here: 
https://stackoverflow.com/questions/52614467/why-does-git-credential-store-call-git-credential-erase-and-make-my-credential-f


Can you please help me found why is git credential-store erase called ?


Best regards,

Re: Git credentials not working

[Christian Couder]

(removing git-security from CC)

On Wed, Oct 3, 2018 at 4:09 PM Dimitri Kopriwa <d.kopriwa@gmail.com> wrote:

> Git credentials in ~/.git-credentials and ~/.config/git/credentials are
> being removed by git upon reading.

https://git-scm.com/docs/git-credential says:

"If the action is reject, git-credential will send the description to
any configured credential helpers, which may erase any stored
credential matching the description."

So maybe this is expected.

Another possibility is that your .gitlab-ci.yml might launch scripts
writing into those files, like the before_script.sh script that is
described on:

https://stackoverflow.com/questions/50553049/is-it-possible-to-do-a-git-push-within-a-gitlab-ci-without-ssh

Could you also check which credential helper and which options are
used? For example with commands like:

$ git config -l --show-origin | grep -i cred
$ git config -l --show-origin | grep -i http
$ git config -l --show-origin | grep -i askpass
$ env | grep -i askpass

Re: Git credentials not working

[Dimitri Kopriwa]

On 10/3/18 11:03 PM, Christian Couder wrote:
> (removing git-security from CC)
>
> On Wed, Oct 3, 2018 at 4:09 PM Dimitri Kopriwa <d.kopriwa@gmail.com> wrote:
>
>> Git credentials in ~/.git-credentials and ~/.config/git/credentials are
>> being removed by git upon reading.
> https://git-scm.com/docs/git-credential says:
>
> "If the action is reject, git-credential will send the description to
> any configured credential helpers, which may erase any stored
> credential matching the description."
>
> So maybe this is expected.
I am using this script to create my credential file, how am I supposed 
to do in a non tty environment? Is there a prefered way?
>
> Another possibility is that your .gitlab-ci.yml might launch scripts
> writing into those files, like the before_script.sh script that is
> described on:
>
> https://stackoverflow.com/questions/50553049/is-it-possible-to-do-a-git-push-within-a-gitlab-ci-without-ssh
>
> Could you also check which credential helper and which options are
> used? For example with commands like:
>
> $ git config -l --show-origin | grep -i cred
> $ git config -l --show-origin | grep -i http
> $ git config -l --show-origin | grep -i askpass
> $ env | grep -i askpass
  * branch            HEAD       -> FETCH_HEAD
17:15:36.175966 run-command.c:637       trace: run_command: git gc --auto
17:15:36.177688 git.c:415               trace: built-in: git gc --auto
^[[32;1m$ git config -l --show-origin | grep -i cred^[[0;m
17:15:36.180191 git.c:415               trace: built-in: git config -l 
--show-origin
file:/root/.gitconfig    credential.helper=store
file:.git/config    credential.helper=store
^[[32;1m$ git config -l --show-origin | grep -i http^[[0;m
17:15:36.182768 git.c:415               trace: built-in: git config -l 
--show-origin
file:.git/config 
remote.origin.url=https://git.example.com/example/sample-project.git
^[[32;1m$ git config -l --show-origin | grep -i askpass || echo nothing 
to do^[[0;m
17:15:36.185306 git.c:415               trace: built-in: git config -l 
--show-origin
nothing to do
^[[32;1m$ env | grep -i askpass || echo nothing to do^[[0;m
nothing to do

Re: Fwd: Git credentials not working

[Jeff King]

On Wed, Oct 03, 2018 at 09:06:38PM +0700, Dimitri Kopriwa wrote:

> 18:25:52.940307 git.c:659               trace: exec: git-credential-store erase
> 18:25:52.940365 run-command.c:637       trace: run_command: git-credential-store erase
> remote: HTTP Basic: Access denied
> fatal: Authentication failed for
> 'https://git.example.com/example/some-project.git/'
> [...]
> 
> Can you please help me found why is git credential-store erase called ?

This is expected. We tried to use a credential that was rejected by the
server, so we told all of the helpers it was invalid. You can try
running GIT_TRACE_CURL=1 to see the HTTP conversation. There will be an
HTTP 401 with the authentication failure, though it may not tell you
anything more useful than that.

git-credential-store is meant to be used interactively, to insert and
erase credentials as they're grabbed from the terminal.

It sounds more like you want to just have a stored credential that you
try to use. You could do that with a custom helper. E.g., something like
this in your ~/.gitconfig:

  [credential "https://example.com"]
  helper = "!f() { test $1 = get && echo password=$(cat /path/with/password); }; f"

-Peff

Re: Fwd: Git credentials not working

[Dimitri Kopriwa]

Thanks for your reply. I have activated GIT_TRACE_CURL=1 and I can see 
that the request is failing 401.

I can't see which token is used and using what header ?

The log say:

17:50:26.414654 http.c:657              => Send header: Authorization: Basic <redacted>

I have retested the token locally and it work when used in the url or 
using `Private-Token: <token>` as stated in the Gitlab documentation 
https://docs.gitlab.com/ee/api/README.html#personal-access-tokens

Peff, what would be the appropriate way to input my git credential in a 
100% success way in a CI?

Is this good:

git credential approve <<EOF
protocol=https
host=example.com
username=bob
password=secr3t
OEF

?

I would use the custom helper after I can understand how to properly use 
the git credential store in a CI environment. The fact that I am using a 
generated file is simply because this is what the documentation told me 
to do. I did not found anywhere in the doc how I should create that file 
in a non tty terminal.

Thanks again for your help.

On 10/4/18 12:11 AM, Jeff King wrote:
> On Wed, Oct 03, 2018 at 09:06:38PM +0700, Dimitri Kopriwa wrote:
>
>> 18:25:52.940307 git.c:659               trace: exec: git-credential-store erase
>> 18:25:52.940365 run-command.c:637       trace: run_command: git-credential-store erase
>> remote: HTTP Basic: Access denied
>> fatal: Authentication failed for
>> 'https://git.example.com/example/some-project.git/'
>> [...]
>>
>> Can you please help me found why is git credential-store erase called ?
> This is expected. We tried to use a credential that was rejected by the
> server, so we told all of the helpers it was invalid. You can try
> running GIT_TRACE_CURL=1 to see the HTTP conversation. There will be an
> HTTP 401 with the authentication failure, though it may not tell you
> anything more useful than that.
>
> git-credential-store is meant to be used interactively, to insert and
> erase credentials as they're grabbed from the terminal.
>
> It sounds more like you want to just have a stored credential that you
> try to use. You could do that with a custom helper. E.g., something like
> this in your ~/.gitconfig:
>
>    [credential "https://example.com"]
>    helper = "!f() { test $1 = get && echo password=$(cat /path/with/password); }; f"
>
> -Peff

Re: Fwd: Git credentials not working

[Jeff King]

On Thu, Oct 04, 2018 at 01:12:11AM +0700, Dimitri Kopriwa wrote:

> Thanks for your reply. I have activated GIT_TRACE_CURL=1 and I can see that
> the request is failing 401.
> 
> I can't see which token is used and using what header ?
> 
> The log say:
> 
> 17:50:26.414654 http.c:657              => Send header: Authorization: Basic <redacted>

Yeah, we redact the auth information so people don't accidentally share
it publicly. If you use the older GIT_CURL_VERBOSE=1, it will include
the credential (I think it may be base64 encoded, though, so you'll have
to decipher it).

> I have retested the token locally and it work when used in the url or using
> `Private-Token: <token>` as stated in the Gitlab documentation
> https://docs.gitlab.com/ee/api/README.html#personal-access-tokens

I don't think Git will ever send your token in either of those ways. It
will always some as an Authorization header.

> Peff, what would be the appropriate way to input my git credential in a 100%
> success way in a CI?

I don't know the details of what GitLab would want, but...

> Is this good:
> 
> git credential approve <<EOF
> protocol=https
> host=example.com
> username=bob
> password=secr3t
> OEF

Yes, that would work to preload a token into any configured helpers.

-Peff

Re: Fwd: Git credentials not working

[Dimitri Kopriwa]

I have replaced the way I fill the git credentials store, I have verify 
~/.git-credentials and information are there, the ~/.gitconfig look fine 
too.

I still have 401 error when reading from that file.

This is the paste log : https://paste.gnome.org/pmntlkdw0

Now that I use git approve, I dont think that I need a custom helper.

Any idea why I still can't log in using git-credential?

Thanks in advance,

On 10/4/18 1:24 AM, Jeff King wrote:
> On Thu, Oct 04, 2018 at 01:12:11AM +0700, Dimitri Kopriwa wrote:
>
>> Thanks for your reply. I have activated GIT_TRACE_CURL=1 and I can see that
>> the request is failing 401.
>>
>> I can't see which token is used and using what header ?
>>
>> The log say:
>>
>> 17:50:26.414654 http.c:657              => Send header: Authorization: Basic <redacted>
> Yeah, we redact the auth information so people don't accidentally share
> it publicly. If you use the older GIT_CURL_VERBOSE=1, it will include
> the credential (I think it may be base64 encoded, though, so you'll have
> to decipher it).
>
>> I have retested the token locally and it work when used in the url or using
>> `Private-Token: <token>` as stated in the Gitlab documentation
>> https://docs.gitlab.com/ee/api/README.html#personal-access-tokens
> I don't think Git will ever send your token in either of those ways. It
> will always some as an Authorization header.
>
>> Peff, what would be the appropriate way to input my git credential in a 100%
>> success way in a CI?
> I don't know the details of what GitLab would want, but...
>
>> Is this good:
>>
>> git credential approve <<EOF
>> protocol=https
>> host=example.com
>> username=bob
>> password=secr3t
>> OEF
> Yes, that would work to preload a token into any configured helpers.
>
> -Peff

Re: Fwd: Git credentials not working

[Bryan Turner]

On Wed, Oct 3, 2018 at 12:34 PM Dimitri Kopriwa <d.kopriwa@gmail.com> wrote:
>
> I have replaced the way I fill the git credentials store, I have verify
> ~/.git-credentials and information are there, the ~/.gitconfig look fine
> too.
>
> I still have 401 error when reading from that file.
>
> This is the paste log : https://paste.gnome.org/pmntlkdw0
>
> Now that I use git approve, I dont think that I need a custom helper.
>
> Any idea why I still can't log in using git-credential?

I'm pretty sure Peff touched on this in his reply. When it works,
you're either sending a "Private-Token" header or including it in the
URL, but, as Peff said, Git will never do either of those things. It
sends an "Authorization" header, and, based on their documentation, it
doesn't appear Gitlab accepts access tokens in that header.

It looks like you're either going to need to include it in the URL
(like what happens earlier in the posted trace), or adjust your git
config with a "http.extraHeader" set to "Private-Token: <token>" to
include the "Private-Token" header (or you could pass it on the
command line, like `git -c http.extraHeader="Private-Token: <token>"
clone ...`.

Hope this helps!
Bryan

>
> Thanks in advance,
>
> On 10/4/18 1:24 AM, Jeff King wrote:
> > On Thu, Oct 04, 2018 at 01:12:11AM +0700, Dimitri Kopriwa wrote:
> >
> >> Thanks for your reply. I have activated GIT_TRACE_CURL=1 and I can see that
> >> the request is failing 401.
> >>
> >> I can't see which token is used and using what header ?
> >>
> >> The log say:
> >>
> >> 17:50:26.414654 http.c:657              => Send header: Authorization: Basic <redacted>
> > Yeah, we redact the auth information so people don't accidentally share
> > it publicly. If you use the older GIT_CURL_VERBOSE=1, it will include
> > the credential (I think it may be base64 encoded, though, so you'll have
> > to decipher it).
> >
> >> I have retested the token locally and it work when used in the url or using
> >> `Private-Token: <token>` as stated in the Gitlab documentation
> >> https://docs.gitlab.com/ee/api/README.html#personal-access-tokens
> > I don't think Git will ever send your token in either of those ways. It
> > will always some as an Authorization header.
> >
> >> Peff, what would be the appropriate way to input my git credential in a 100%
> >> success way in a CI?
> > I don't know the details of what GitLab would want, but...
> >
> >> Is this good:
> >>
> >> git credential approve <<EOF
> >> protocol=https
> >> host=example.com
> >> username=bob
> >> password=secr3t
> >> OEF
> > Yes, that would work to preload a token into any configured helpers.
> >
> > -Peff

Re: Fwd: Git credentials not working

[Jeff King]

On Thu, Oct 04, 2018 at 02:34:17AM +0700, Dimitri Kopriwa wrote:

> I have replaced the way I fill the git credentials store, I have verify
> ~/.git-credentials and information are there, the ~/.gitconfig look fine
> too.
> 
> I still have 401 error when reading from that file.
> 
> This is the paste log : https://paste.gnome.org/pmntlkdw0
> 
> Now that I use git approve, I dont think that I need a custom helper.
> 
> Any idea why I still can't log in using git-credential?

Looking at your pastebin, it looks like the server sometimes takes it
and sometimes not. E.g., piping the log through:

  egrep '(Send|Recv) header:' |
  perl -lpe 's/^.*?(=>|<=) //'

I see:

  Send header: GET /example-keys/sample-project.git/info/refs?service=git-upload-pack HTTP/1.1
  Send header: User-Agent: git/2.19.0
  ...
  Recv header: HTTP/1.1 401 Unauthorized
  Recv header: WWW-Authenticate: Basic realm="GitLab"
  ...
  Send header: GET /example-keys/sample-project.git/info/refs?service=git-upload-pack HTTP/1.1
  Send header: Authorization: Basic <redacted>
  Send header: User-Agent: git/2.19.0
  ...
  Recv header: HTTP/1.1 200 OK

So that works. But then later we get:

  Send header: GET /example-keys/sample-project.git/info/refs?service=git-upload-pack HTTP/1.1
  Send header: User-Agent: git/2.19.0
  ...
  Recv header: HTTP/1.1 401 Unauthorized
  Recv header: WWW-Authenticate: Basic realm="GitLab"
  ...
  Send header: GET /example-keys/sample-project.git/info/refs?service=git-upload-pack HTTP/1.1
  Send header: Authorization: Basic <redacted>
  Send header: User-Agent: git/2.19.0
  ...
  Recv header: HTTP/1.1 401 Unauthorized

And then that causes credential-store to delete the non-working entry,
after which all of them must fail (because you have no working
credential, and presumably no terminal to prompt the user).

I have no idea why the same request would sometimes be allowed and
sometimes not. It's possible the <redacted> data is different in those
two times, but I don't know why that would be. It's also possible you're
hitting different load-balancing servers that behave differently.

-Peff

Re: Fwd: Git credentials not working

[Dimitri Kopriwa]

Thanks everyone.

All your answers helped. I found out that the issue was not related to git.

I am using semantic-release to perform a release, apparently 
git-credentials is not working with semantic-release.

I did also setup the double authentication and every fix applied on 
git-credentials were simply useless.

Read more here : 
https://github.com/semantic-release/semantic-release/issues/941#issuecomment-426691824

Thanks a lot for your help and git is the best software ever made thanks!

Dimitri Kopriwa


On 10/4/18 3:43 AM, Jeff King wrote:
> On Thu, Oct 04, 2018 at 02:34:17AM +0700, Dimitri Kopriwa wrote:
>
>> I have replaced the way I fill the git credentials store, I have verify
>> ~/.git-credentials and information are there, the ~/.gitconfig look fine
>> too.
>>
>> I still have 401 error when reading from that file.
>>
>> This is the paste log : https://paste.gnome.org/pmntlkdw0
>>
>> Now that I use git approve, I dont think that I need a custom helper.
>>
>> Any idea why I still can't log in using git-credential?
> Looking at your pastebin, it looks like the server sometimes takes it
> and sometimes not. E.g., piping the log through:
>
>    egrep '(Send|Recv) header:' |
>    perl -lpe 's/^.*?(=>|<=) //'
>
> I see:
>
>    Send header: GET /example-keys/sample-project.git/info/refs?service=git-upload-pack HTTP/1.1
>    Send header: User-Agent: git/2.19.0
>    ...
>    Recv header: HTTP/1.1 401 Unauthorized
>    Recv header: WWW-Authenticate: Basic realm="GitLab"
>    ...
>    Send header: GET /example-keys/sample-project.git/info/refs?service=git-upload-pack HTTP/1.1
>    Send header: Authorization: Basic <redacted>
>    Send header: User-Agent: git/2.19.0
>    ...
>    Recv header: HTTP/1.1 200 OK
>
> So that works. But then later we get:
>
>    Send header: GET /example-keys/sample-project.git/info/refs?service=git-upload-pack HTTP/1.1
>    Send header: User-Agent: git/2.19.0
>    ...
>    Recv header: HTTP/1.1 401 Unauthorized
>    Recv header: WWW-Authenticate: Basic realm="GitLab"
>    ...
>    Send header: GET /example-keys/sample-project.git/info/refs?service=git-upload-pack HTTP/1.1
>    Send header: Authorization: Basic <redacted>
>    Send header: User-Agent: git/2.19.0
>    ...
>    Recv header: HTTP/1.1 401 Unauthorized
>
> And then that causes credential-store to delete the non-working entry,
> after which all of them must fail (because you have no working
> credential, and presumably no terminal to prompt the user).
>
> I have no idea why the same request would sometimes be allowed and
> sometimes not. It's possible the <redacted> data is different in those
> two times, but I don't know why that would be. It's also possible you're
> hitting different load-balancing servers that behave differently.
>
> -Peff