[PATCH] ref-manual: tasks: do_cleanall: mention it is not allowed with a shared DL_DIR

[PATCH] ref-manual: tasks: do_cleanall: mention it is not allowed with a shared DL_DIR

[Luca Ceresoli]

do_cleanall can produce failures if using a shared download directory. This
is why it is forbidden when writing tests that will run on the autobuilders
(https://docs.yoctoproject.org/test-manual/intro.html?highlight=cleanall#considerations-when-writing-tests).

Reported-by: Sam Liddicott
Link: https://bootlin.com/blog/yocto-sharing-the-sstate-cache-and-download-directories/#comment-2650335
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
---
 documentation/ref-manual/tasks.rst | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/documentation/ref-manual/tasks.rst b/documentation/ref-manual/tasks.rst
index 0db960b22f80..c754eb15514b 100644
--- a/documentation/ref-manual/tasks.rst
+++ b/documentation/ref-manual/tasks.rst
@@ -474,6 +474,19 @@ Typically, you would not normally use the :ref:`ref-tasks-cleanall` task. Do so
 if you want to start fresh with the :ref:`ref-tasks-fetch`
 task.
 
+.. note::
+
+   Never run the :ref:`ref-tasks-cleanall` task if your :term:`DL_DIR` is
+   shared. This would break in this scenario::
+
+      dir1$ bitbake <recipe> -c fetch
+      dir2$ bitbake <recipe> -c cleanall
+      dir1$ bitbake <recipe> -c unpack
+
+   because before step 3 in dir1 there is a stamp file for the
+   :ref:`ref-tasks-fetch` task, so the unpack task will try to extract the
+   downloaded archive and fail as it has been deleted in step 2.
+
 .. _ref-tasks-cleansstate:
 
 ``do_cleansstate``
-- 
2.34.1

Re: [docs] [PATCH] ref-manual: tasks: do_cleanall: mention it is not allowed with a shared DL_DIR

[Quentin Schulz]

Hi Luca,

On 2/19/24 16:55, Luca Ceresoli via lists.yoctoproject.org wrote:
> [You don't often get email from luca.ceresoli=bootlin.com@lists.yoctoproject.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> 
> do_cleanall can produce failures if using a shared download directory. This
> is why it is forbidden when writing tests that will run on the autobuilders
> (https://docs.yoctoproject.org/test-manual/intro.html?highlight=cleanall#considerations-when-writing-tests).
> 
> Reported-by: Sam Liddicott
> Link: https://bootlin.com/blog/yocto-sharing-the-sstate-cache-and-download-directories/#comment-2650335
> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
> ---
>   documentation/ref-manual/tasks.rst | 13 +++++++++++++
>   1 file changed, 13 insertions(+)
> 
> diff --git a/documentation/ref-manual/tasks.rst b/documentation/ref-manual/tasks.rst
> index 0db960b22f80..c754eb15514b 100644
> --- a/documentation/ref-manual/tasks.rst
> +++ b/documentation/ref-manual/tasks.rst
> @@ -474,6 +474,19 @@ Typically, you would not normally use the :ref:`ref-tasks-cleanall` task. Do so
>   if you want to start fresh with the :ref:`ref-tasks-fetch`
>   task.
> 
> +.. note::
> +
> +   Never run the :ref:`ref-tasks-cleanall` task if your :term:`DL_DIR` is

This also applies if SSTATE_DIR is shared I believe?

> +   shared. This would break in this scenario::
> +
> +      dir1$ bitbake <recipe> -c fetch
> +      dir2$ bitbake <recipe> -c cleanall
> +      dir1$ bitbake <recipe> -c unpack
> +
> +   because before step 3 in dir1 there is a stamp file for the
> +   :ref:`ref-tasks-fetch` task, so the unpack task will try to extract the
> +   downloaded archive and fail as it has been deleted in step 2.
> +
>   .. _ref-tasks-cleansstate:
> 
>   ``do_cleansstate``

We would need the same warning here for a shared SSTATE_DIR I believe.

Cheers,
Quentin

Re: [docs] [PATCH] ref-manual: tasks: do_cleanall: mention it is not allowed with a shared DL_DIR

[Luca Ceresoli]

Hi Quentin,

On Mon, 19 Feb 2024 17:12:46 +0100
Quentin Schulz <quentin.schulz@theobroma-systems.com> wrote:

> Hi Luca,
> 
> On 2/19/24 16:55, Luca Ceresoli via lists.yoctoproject.org wrote:
> > [You don't often get email from luca.ceresoli=bootlin.com@lists.yoctoproject.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> > 
> > do_cleanall can produce failures if using a shared download directory. This
> > is why it is forbidden when writing tests that will run on the autobuilders
> > (https://docs.yoctoproject.org/test-manual/intro.html?highlight=cleanall#considerations-when-writing-tests).
> > 
> > Reported-by: Sam Liddicott
> > Link: https://bootlin.com/blog/yocto-sharing-the-sstate-cache-and-download-directories/#comment-2650335
> > Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
> > ---
> >   documentation/ref-manual/tasks.rst | 13 +++++++++++++
> >   1 file changed, 13 insertions(+)
> > 
> > diff --git a/documentation/ref-manual/tasks.rst b/documentation/ref-manual/tasks.rst
> > index 0db960b22f80..c754eb15514b 100644
> > --- a/documentation/ref-manual/tasks.rst
> > +++ b/documentation/ref-manual/tasks.rst
> > @@ -474,6 +474,19 @@ Typically, you would not normally use the :ref:`ref-tasks-cleanall` task. Do so
> >   if you want to start fresh with the :ref:`ref-tasks-fetch`
> >   task.
> > 
> > +.. note::
> > +
> > +   Never run the :ref:`ref-tasks-cleanall` task if your :term:`DL_DIR` is  
> 
> This also applies if SSTATE_DIR is shared I believe?

I don't think so. As far as I understand the sstate cache, a cache
entry is either found or not found, while for the downloads there is
the strange state of having a do_fetch stamp but no downloaded archive.

However I may be wrong. Do you have an example command sequence where
do_cleansstate can create a build failure?

Luca

-- 
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH] ref-manual: tasks: do_cleanall: mention it is not allowed with a shared DL_DIR

[Richard Purdie]

On Tue, 2024-02-20 at 08:58 +0100, Luca Ceresoli via
lists.yoctoproject.org wrote:
> Hi Quentin,
> 
> On Mon, 19 Feb 2024 17:12:46 +0100
> Quentin Schulz <quentin.schulz@theobroma-systems.com> wrote:
> 
> > Hi Luca,
> > 
> > On 2/19/24 16:55, Luca Ceresoli via lists.yoctoproject.org wrote:
> > > [You don't often get email from
> > > luca.ceresoli=bootlin.com@lists.yoctoproject.org. Learn why this
> > > is important at https://aka.ms/LearnAboutSenderIdentification ]
> > > 
> > > do_cleanall can produce failures if using a shared download
> > > directory. This
> > > is why it is forbidden when writing tests that will run on the
> > > autobuilders
> > > (
> > > https://docs.yoctoproject.org/test-manual/intro.html?highlight=cle
> > > anall#considerations-when-writing-tests).
> > > 
> > > Reported-by: Sam Liddicott
> > > Link:
> > > https://bootlin.com/blog/yocto-sharing-the-sstate-cache-and-download-directories/#comment-2650335
> > > Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
> > > ---
> > >   documentation/ref-manual/tasks.rst | 13 +++++++++++++
> > >   1 file changed, 13 insertions(+)
> > > 
> > > diff --git a/documentation/ref-manual/tasks.rst
> > > b/documentation/ref-manual/tasks.rst
> > > index 0db960b22f80..c754eb15514b 100644
> > > --- a/documentation/ref-manual/tasks.rst
> > > +++ b/documentation/ref-manual/tasks.rst
> > > @@ -474,6 +474,19 @@ Typically, you would not normally use the
> > > :ref:`ref-tasks-cleanall` task. Do so
> > >   if you want to start fresh with the :ref:`ref-tasks-fetch`
> > >   task.
> > > 
> > > +.. note::
> > > +
> > > +   Never run the :ref:`ref-tasks-cleanall` task if your
> > > :term:`DL_DIR` is  
> > 
> > This also applies if SSTATE_DIR is shared I believe?
> 
> I don't think so. As far as I understand the sstate cache, a cache
> entry is either found or not found, while for the downloads there is
> the strange state of having a do_fetch stamp but no downloaded
> archive.
> 
> However I may be wrong. Do you have an example command sequence where
> do_cleansstate can create a build failure?

The builds check sstate "up front" then download later so it if is
deleted in the meantime, it will cause an error but not a total failure
as it will rebuild it.

We therefore don't recommend cleansstate on a shared sstate directory.
Instead you can use -f to invalidate a task.

Cheers,

Richard

Re: [docs] [PATCH] ref-manual: tasks: do_cleanall: mention it is not allowed with a shared DL_DIR

[Quentin Schulz]

Hi all,

On 2/20/24 10:01, Richard Purdie wrote:
> On Tue, 2024-02-20 at 08:58 +0100, Luca Ceresoli via
> lists.yoctoproject.org wrote:
>> Hi Quentin,
>>
>> On Mon, 19 Feb 2024 17:12:46 +0100
>> Quentin Schulz <quentin.schulz@theobroma-systems.com> wrote:
>>
>>> Hi Luca,
>>>
>>> On 2/19/24 16:55, Luca Ceresoli via lists.yoctoproject.org wrote:
>>>> [You don't often get email from
>>>> luca.ceresoli=bootlin.com@lists.yoctoproject.org. Learn why this
>>>> is important at https://aka.ms/LearnAboutSenderIdentification ]
>>>>
>>>> do_cleanall can produce failures if using a shared download
>>>> directory. This
>>>> is why it is forbidden when writing tests that will run on the
>>>> autobuilders
>>>> (
>>>> https://docs.yoctoproject.org/test-manual/intro.html?highlight=cle
>>>> anall#considerations-when-writing-tests).
>>>>
>>>> Reported-by: Sam Liddicott
>>>> Link:
>>>> https://bootlin.com/blog/yocto-sharing-the-sstate-cache-and-download-directories/#comment-2650335
>>>> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
>>>> ---
>>>>    documentation/ref-manual/tasks.rst | 13 +++++++++++++
>>>>    1 file changed, 13 insertions(+)
>>>>
>>>> diff --git a/documentation/ref-manual/tasks.rst
>>>> b/documentation/ref-manual/tasks.rst
>>>> index 0db960b22f80..c754eb15514b 100644
>>>> --- a/documentation/ref-manual/tasks.rst
>>>> +++ b/documentation/ref-manual/tasks.rst
>>>> @@ -474,6 +474,19 @@ Typically, you would not normally use the
>>>> :ref:`ref-tasks-cleanall` task. Do so
>>>>    if you want to start fresh with the :ref:`ref-tasks-fetch`
>>>>    task.
>>>>
>>>> +.. note::
>>>> +
>>>> +   Never run the :ref:`ref-tasks-cleanall` task if your
>>>> :term:`DL_DIR` is
>>>
>>> This also applies if SSTATE_DIR is shared I believe?
>>
>> I don't think so. As far as I understand the sstate cache, a cache
>> entry is either found or not found, while for the downloads there is
>> the strange state of having a do_fetch stamp but no downloaded
>> archive.
>>
>> However I may be wrong. Do you have an example command sequence where
>> do_cleansstate can create a build failure?
> 
> The builds check sstate "up front" then download later so it if is
> deleted in the meantime, it will cause an error but not a total failure
> as it will rebuild it.
> 
> We therefore don't recommend cleansstate on a shared sstate directory.
> Instead you can use -f to invalidate a task.
> 

And considering that cleanall also calls cleansstate (or at least does 
the same), it applies and we should document it :)

Cheers,
Quentin