gitsm shared DL_DIR and race conditions

gitsm shared DL_DIR and race conditions

[Stefan Agner]

Hi,

We came across race conditions while fetching a repository using gitsm:
ERROR: aktualizr-native-1.0+gitAUTOINC+d00d1a04cc-7 do_fetch: Fetcher
failure: Fetch command export PSEUDO_DISABLED=1; export
PATH="/workdir/oe/layers/openembedded-core/scripts/native-intercept:/workdir/oe/layers/openembedded-core/scripts:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/usr/bin/x86_64-linux:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/usr/bin:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/usr/sbin:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/usr/bin:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/sbin:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/bin:/workdir/oe/bitbake/bin:/workdir/oe/tmp/hosttools";
export HOME="/home/yocto"; git -c core.fsyncobjectfiles=0 config
submodule.tests/tuf-test-vectors.url
/workdir/downloads/git2/github.com.advancedtelematic.tuf-test-vectors.
failed with exit code 255, output:
error: could not lock config file config: File exists

We share the same DL_DIR located on a NFS across multiple builders. We
are using latest state of the 1.40 branch  (thud) of bitbake.

It seems that two git config invocations raced in this case. Is there
locking required in the current gitsm implementation?

--
Stefan

Re: gitsm shared DL_DIR and race conditions

[Mark Hatle]

On 1/9/19 7:48 AM, Stefan Agner wrote:
> Hi,
> 
> We came across race conditions while fetching a repository using gitsm:
> ERROR: aktualizr-native-1.0+gitAUTOINC+d00d1a04cc-7 do_fetch: Fetcher
> failure: Fetch command export PSEUDO_DISABLED=1; export
> PATH="/workdir/oe/layers/openembedded-core/scripts/native-intercept:/workdir/oe/layers/openembedded-core/scripts:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/usr/bin/x86_64-linux:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/usr/bin:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/usr/sbin:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/usr/bin:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/sbin:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/bin:/workdir/oe/bitbake/bin:/workdir/oe/tmp/hosttools";
> export HOME="/home/yocto"; git -c core.fsyncobjectfiles=0 config
> submodule.tests/tuf-test-vectors.url
> /workdir/downloads/git2/github.com.advancedtelematic.tuf-test-vectors.
> failed with exit code 255, output:
> error: could not lock config file config: File exists
> 
> We share the same DL_DIR located on a NFS across multiple builders. We
> are using latest state of the 1.40 branch  (thud) of bitbake.
> 
> It seems that two git config invocations raced in this case. Is there
> locking required in the current gitsm implementation?

Did not expect git config to have locking issue... but it is showing that two
fetches appear to have happened together... gitsm uses the regular git fetcher
for that bit, and then after the fetch is complete updates the config (like git
submodule init would) to point to downloaded components.  It's this
configuration step that appears to have failed.

It should be fairly trivial to catch this issue and retry.

--Mark

> --
> Stefan
> 

Re: gitsm shared DL_DIR and race conditions

[Stefan Agner]

On 09.01.2019 16:38, Mark Hatle wrote:
> On 1/9/19 7:48 AM, Stefan Agner wrote:
>> Hi,
>>
>> We came across race conditions while fetching a repository using gitsm:
>> ERROR: aktualizr-native-1.0+gitAUTOINC+d00d1a04cc-7 do_fetch: Fetcher
>> failure: Fetch command export PSEUDO_DISABLED=1; export
>> PATH="/workdir/oe/layers/openembedded-core/scripts/native-intercept:/workdir/oe/layers/openembedded-core/scripts:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/usr/bin/x86_64-linux:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/usr/bin:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/usr/sbin:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/usr/bin:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/sbin:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/bin:/workdir/oe/bitbake/bin:/workdir/oe/tmp/hosttools";
>> export HOME="/home/yocto"; git -c core.fsyncobjectfiles=0 config
>> submodule.tests/tuf-test-vectors.url
>> /workdir/downloads/git2/github.com.advancedtelematic.tuf-test-vectors.
>> failed with exit code 255, output:
>> error: could not lock config file config: File exists
>>
>> We share the same DL_DIR located on a NFS across multiple builders. We
>> are using latest state of the 1.40 branch  (thud) of bitbake.
>>
>> It seems that two git config invocations raced in this case. Is there
>> locking required in the current gitsm implementation?
> 
> Did not expect git config to have locking issue... but it is showing that two
> fetches appear to have happened together... gitsm uses the regular git fetcher
> for that bit, and then after the fetch is complete updates the config (like git
> submodule init would) to point to downloaded components.  It's this
> configuration step that appears to have failed.
> 
> It should be fairly trivial to catch this issue and retry.

It does not happen often. It happened only once in the last ~100 builds
or so...

Since this is in the CI environment I need to check whether I can jump
into that environment.

As far as I can see git holds a lock to write its config file:
https://github.com/git/git/blob/master/config.c#L2715

Shouldn't we use a blocking lock on bitbake level to avoid running into
the git lock and erroring out?

--
Stefan

Re: gitsm shared DL_DIR and race conditions

[Mark Hatle]

On 1/9/19 10:21 AM, Stefan Agner wrote:
> On 09.01.2019 16:38, Mark Hatle wrote:
>> On 1/9/19 7:48 AM, Stefan Agner wrote:
>>> Hi,
>>>
>>> We came across race conditions while fetching a repository using gitsm:
>>> ERROR: aktualizr-native-1.0+gitAUTOINC+d00d1a04cc-7 do_fetch: Fetcher
>>> failure: Fetch command export PSEUDO_DISABLED=1; export
>>> PATH="/workdir/oe/layers/openembedded-core/scripts/native-intercept:/workdir/oe/layers/openembedded-core/scripts:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/usr/bin/x86_64-linux:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/usr/bin:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/usr/sbin:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/usr/bin:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/sbin:/workdir/oe/tmp/work/x86_64-linux/aktualizr-native/1.0+gitAUTOINC+d00d1a04cc-7/recipe-sysroot-native/bin:/workdir/oe/bitbake/bin:/workdir/oe/tmp/hosttools";
>>> export HOME="/home/yocto"; git -c core.fsyncobjectfiles=0 config
>>> submodule.tests/tuf-test-vectors.url
>>> /workdir/downloads/git2/github.com.advancedtelematic.tuf-test-vectors.
>>> failed with exit code 255, output:
>>> error: could not lock config file config: File exists
>>>
>>> We share the same DL_DIR located on a NFS across multiple builders. We
>>> are using latest state of the 1.40 branch  (thud) of bitbake.
>>>
>>> It seems that two git config invocations raced in this case. Is there
>>> locking required in the current gitsm implementation?
>>
>> Did not expect git config to have locking issue... but it is showing that two
>> fetches appear to have happened together... gitsm uses the regular git fetcher
>> for that bit, and then after the fetch is complete updates the config (like git
>> submodule init would) to point to downloaded components.  It's this
>> configuration step that appears to have failed.
>>
>> It should be fairly trivial to catch this issue and retry.
> 
> It does not happen often. It happened only once in the last ~100 builds
> or so...
> 
> Since this is in the CI environment I need to check whether I can jump
> into that environment.
> 
> As far as I can see git holds a lock to write its config file:
> https://github.com/git/git/blob/master/config.c#L2715
> 
> Shouldn't we use a blocking lock on bitbake level to avoid running into
> the git lock and erroring out?

Any locking that occurs would need to be synchronized between the git and gitsm
fetchers.  I'm not sure how easy or hard this would be to do.  If we can
coordinate the locking there, that would be best... but baring that catch and
retry should be a reasonable alternative as it is highly unlikely that this file
will be locked for an extended period of time.

--Mark

> --
> Stefan
>