Lock external repository in custom FSTYPE

Lock external repository in custom FSTYPE

[Stefan Agner]

Hi,

We use meta-updater, which has a custom FSTYPE to build a OSTree
repository. We share that repository across multiple bitbake executions.
The underlying OSTree tools lock the OSTree repository before trying to
interact, and if it fails ("error: Locking repo exclusive failed:
Resource temporarily unavailable") then the complete build fails (see
also https://github.com/advancedtelematic/meta-updater/issues/412).

Now I'd rather prefer that two bitbake tasks would serialize the access
to the OSTree repository. Is there a mechanism in bitbake to lock (and
wait) for the repository to be not in use?

We tried using bb.utils.lockfile, but the task is written in shell. Also
inline Python would not work since locking/unlocking need to be done
within one Python script as far as I understand.

--
Stefan

Re: Lock external repository in custom FSTYPE

[Khem Raj]

On Mon, Oct 29, 2018 at 5:31 AM Stefan Agner <stefan@agner.ch> wrote:
>
> Hi,
>
> We use meta-updater, which has a custom FSTYPE to build a OSTree
> repository. We share that repository across multiple bitbake executions.
> The underlying OSTree tools lock the OSTree repository before trying to
> interact, and if it fails ("error: Locking repo exclusive failed:
> Resource temporarily unavailable") then the complete build fails (see
> also https://github.com/advancedtelematic/meta-updater/issues/412).
>
> Now I'd rather prefer that two bitbake tasks would serialize the access
> to the OSTree repository. Is there a mechanism in bitbake to lock (and
> wait) for the repository to be not in use?
>
> We tried using bb.utils.lockfile, but the task is written in shell. Also
> inline Python would not work since locking/unlocking need to be done
> within one Python script as far as I understand.
>

perhaps you could use lockfiles something like

do_foo[lockfiles] = ...

> --
> Stefan
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: Lock external repository in custom FSTYPE

[Stefan Agner]

On 29.10.2018 14:01, Khem Raj wrote:
> On Mon, Oct 29, 2018 at 5:31 AM Stefan Agner <stefan@agner.ch> wrote:
>>
>> Hi,
>>
>> We use meta-updater, which has a custom FSTYPE to build a OSTree
>> repository. We share that repository across multiple bitbake executions.
>> The underlying OSTree tools lock the OSTree repository before trying to
>> interact, and if it fails ("error: Locking repo exclusive failed:
>> Resource temporarily unavailable") then the complete build fails (see
>> also https://github.com/advancedtelematic/meta-updater/issues/412).
>>
>> Now I'd rather prefer that two bitbake tasks would serialize the access
>> to the OSTree repository. Is there a mechanism in bitbake to lock (and
>> wait) for the repository to be not in use?
>>
>> We tried using bb.utils.lockfile, but the task is written in shell. Also
>> inline Python would not work since locking/unlocking need to be done
>> within one Python script as far as I understand.
>>
> 
> perhaps you could use lockfiles something like
> 
> do_foo[lockfiles] = ...

Wasn't aware of the lockfiles varflag, also seems not to appear in any
documentation. Reading lib/bb/build.py suggests that it really is
locking the complete task, exactly what we need.

Added the lockfiles varflag to the do_image_ostree task in
image_types_ostree.bbclass:
do_image_ostree[lockfiles] += "${OSTREE_REPO}/ostree.lock"

And run two builds, the OSTree lock issues seem to be gone. Thanks Khem!

--
Stefan

> 
>> --
>> Stefan
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: Lock external repository in custom FSTYPE

[Stefan Agner]

On 01.11.2018 23:27, Stefan Agner wrote:
> On 29.10.2018 14:01, Khem Raj wrote:
>> On Mon, Oct 29, 2018 at 5:31 AM Stefan Agner <stefan@agner.ch> wrote:
>>>
>>> Hi,
>>>
>>> We use meta-updater, which has a custom FSTYPE to build a OSTree
>>> repository. We share that repository across multiple bitbake executions.
>>> The underlying OSTree tools lock the OSTree repository before trying to
>>> interact, and if it fails ("error: Locking repo exclusive failed:
>>> Resource temporarily unavailable") then the complete build fails (see
>>> also https://github.com/advancedtelematic/meta-updater/issues/412).
>>>
>>> Now I'd rather prefer that two bitbake tasks would serialize the access
>>> to the OSTree repository. Is there a mechanism in bitbake to lock (and
>>> wait) for the repository to be not in use?
>>>
>>> We tried using bb.utils.lockfile, but the task is written in shell. Also
>>> inline Python would not work since locking/unlocking need to be done
>>> within one Python script as far as I understand.
>>>
>>
>> perhaps you could use lockfiles something like
>>
>> do_foo[lockfiles] = ...
> 
> Wasn't aware of the lockfiles varflag, also seems not to appear in any
> documentation. Reading lib/bb/build.py suggests that it really is

I take that back:

https://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user-manual.html#variable-flags

--
Stefan

> locking the complete task, exactly what we need.
> 
> Added the lockfiles varflag to the do_image_ostree task in
> image_types_ostree.bbclass:
> do_image_ostree[lockfiles] += "${OSTREE_REPO}/ostree.lock"
> 
> And run two builds, the OSTree lock issues seem to be gone. Thanks Khem!
> 
> --
> Stefan
> 
>>
>>> --
>>> Stefan
>>> --
>>> _______________________________________________
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core