cleansstate

cleansstate

[Joe Slater]

[-- Attachment #1: Type: text/plain, Size: 337 bytes --]

This might be specific to the distro I am typically using (take a guess) but I notice that cleansstate usually pauses at the 33% mark for a significant amount of time.  I don't think this used to be quite as bad as it is now, but it is hard to remember.  Just an observation that someone might be able to confirm/deny/explain.

Joe


[-- Attachment #2: Type: text/html, Size: 1914 bytes --]

Re: [OE-core] cleansstate

[Richard Purdie]

On Wed, 2020-08-19 at 23:28 +0000, Joe Slater wrote:
> This might be specific to the distro I am typically using (take a
> guess) but I notice that cleansstate usually pauses at the 33% mark
> for a significant amount of time.  I don’t think this used to be
> quite as bad as it is now, but it is hard to remember.  Just an
> observation that someone might be able to confirm/deny/explain.

sstateclean iterates through all the sstate directories globbing for
potential things to remove. If your sstate cache is large or remote
(e.g. NFS), this will be slow.

Note it doesn't remove just the current hash but any sstate for the
given recipe.

In normal use you should never be needing sstateclean.

Cheers,

Richard

Re: [OE-core] cleansstate

[tom king]

[-- Attachment #1: Type: text/plain, Size: 927 bytes --]

What would be a use case for cleanstate?

Tom

On Fri, Aug 21, 2020 at 8:14 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Wed, 2020-08-19 at 23:28 +0000, Joe Slater wrote:
> > This might be specific to the distro I am typically using (take a
> > guess) but I notice that cleansstate usually pauses at the 33% mark
> > for a significant amount of time.  I don’t think this used to be
> > quite as bad as it is now, but it is hard to remember.  Just an
> > observation that someone might be able to confirm/deny/explain.
>
> sstateclean iterates through all the sstate directories globbing for
> potential things to remove. If your sstate cache is large or remote
> (e.g. NFS), this will be slow.
>
> Note it doesn't remove just the current hash but any sstate for the
> given recipe.
>
> In normal use you should never be needing sstateclean.
>
> Cheers,
>
> Richard
>
> 
>

[-- Attachment #2: Type: text/html, Size: 1304 bytes --]

Re: [OE-core] cleansstate

[Richard Purdie]

On Fri, 2020-08-21 at 17:32 -0700, Tom King wrote:
> What would be a use case for cleanstate?

I never really wanted to add it at all. There is/was some case for
wanting to remove sstate objects and ensure something really does
rebuild from source. When sstate was new and had more issues, people
really wanted it and I was pushed into having it exist.

I understand why people want it although it should never really need to
be used.

There are some users who do use it regularly and they really need to
think about why and whether they need to. Personally, I don't use it.

Cheers,

Richard

Re: [OE-core] cleansstate

[Joe Slater]

I use it via cleanall to be sure I'm building something as if it had never been built before.  I share sstate between a number of projects (I'm calling where you run bitbake from a project), and sometimes just want to be sure there's no "contamination".

Joe

-----Original Message-----
From: Richard Purdie <richard.purdie@linuxfoundation.org> 
Sent: Saturday, August 22, 2020 12:28 AM
To: Tom King <ka6sox@gmail.com>
Cc: Slater, Joseph <joe.slater@windriver.com>; openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] cleansstate

On Fri, 2020-08-21 at 17:32 -0700, Tom King wrote:
> What would be a use case for cleanstate?

I never really wanted to add it at all. There is/was some case for wanting to remove sstate objects and ensure something really does rebuild from source. When sstate was new and had more issues, people really wanted it and I was pushed into having it exist.

I understand why people want it although it should never really need to be used.

There are some users who do use it regularly and they really need to think about why and whether they need to. Personally, I don't use it.

Cheers,

Richard

Re: [OE-core] cleansstate

[Rasmus Villemoes]

On 22/08/2020 09.27, Richard Purdie via lists.openembedded.org wrote:
> On Fri, 2020-08-21 at 17:32 -0700, Tom King wrote:
>> What would be a use case for cleanstate?
> 
> I never really wanted to add it at all. There is/was some case for
> wanting to remove sstate objects and ensure something really does
> rebuild from source. When sstate was new and had more issues, people
> really wanted it and I was pushed into having it exist.
> 
> I understand why people want it although it should never really need to
> be used.

Well, except when you discover that your sstate is effectively
corrupted/polluted by objects that happen to have the same hashes as
those computed from current recipes, but the build artifacts
would/should be different. Such as when using a file://foo/* glob
pattern in SRC_URI.

Richard, I'm still hoping you'd respond in the
https://lists.openembedded.org/g/openembedded-core/topic/75235594
thread. As I see it, half-way supporting glob patterns in file:// URIs
is a giant footgun.

Rasmus

Re: [OE-core] cleansstate

[Richard Purdie]

On Mon, 2020-08-24 at 23:05 +0200, Rasmus Villemoes wrote:
> On 22/08/2020 09.27, Richard Purdie via lists.openembedded.org wrote:
> > On Fri, 2020-08-21 at 17:32 -0700, Tom King wrote:
> > > What would be a use case for cleanstate?
> > 
> > I never really wanted to add it at all. There is/was some case for
> > wanting to remove sstate objects and ensure something really does
> > rebuild from source. When sstate was new and had more issues,
> > people
> > really wanted it and I was pushed into having it exist.
> > 
> > I understand why people want it although it should never really
> > need to
> > be used.
> 
> Well, except when you discover that your sstate is effectively
> corrupted/polluted by objects that happen to have the same hashes as
> those computed from current recipes, but the build artifacts
> would/should be different. Such as when using a file://foo/* glob
> pattern in SRC_URI.

Where such issues are reported, we do try and fix them. I hadn't quite
connected that the globbing issue was actually breaking sstate hashes,
that is more worrying :(

> 
> Richard, I'm still hoping you'd respond in the
> https://lists.openembedded.org/g/openembedded-core/topic/75235594
> thread. As I see it, half-way supporting glob patterns in file://
> URIs is a giant footgun.

My leaning is to remove them. You can just do file://foo and I think
that actually works?

The globs are horrible for a lot of different reasons.

Cheers,

Richard

Re: [OE-core] cleansstate

[Rasmus Villemoes]

On 24/08/2020 23.08, Richard Purdie wrote:
> On Mon, 2020-08-24 at 23:05 +0200, Rasmus Villemoes wrote:
>> On 22/08/2020 09.27, Richard Purdie via lists.openembedded.org wrote:
>>> On Fri, 2020-08-21 at 17:32 -0700, Tom King wrote:
>>>> What would be a use case for cleanstate?
>>>

>>> I understand why people want it although it should never really
>>> need to
>>> be used.
>>
>> Well, except when you discover that your sstate is effectively
>> corrupted/polluted by objects that happen to have the same hashes as
>> those computed from current recipes, but the build artifacts
>> would/should be different. Such as when using a file://foo/* glob
>> pattern in SRC_URI.
> 
> Where such issues are reported, we do try and fix them. I hadn't quite
> connected that the globbing issue was actually breaking sstate hashes,
> that is more worrying :(

It does, and you can reproduce yourself using upstream core recipes. Try
building connman-gnome. Then alter one of the png files under images/,
and build again. The recipe does not get rebuilt. Then cleansstate,
build, and yes, then the modified png does get picked up.

Of course, those pngs don't change often, if ever. But in our case the
recipe was for including some firmware blobs for some I/O modules. We
found out when the CI builds that used sstate cache failed to produce
BSPs containing the updated firmware, while the nightly builds that
build from scratch did.

>>
>> Richard, I'm still hoping you'd respond in the
>> https://lists.openembedded.org/g/openembedded-core/topic/75235594
>> thread. As I see it, half-way supporting glob patterns in file://
>> URIs is a giant footgun.
> 
> My leaning is to remove them. You can just do file://foo and I think
> that actually works?

In some of the cases where we had recipes using globs, yes, that was the
solution. In a few other cases we had more complicated patterns (e.g.
*.sh or something else that wasn't just including a whole dir).

Rasmus