[RFC] One shared state reuse solution

[RFC] One shared state reuse solution

[Chris Larson]

Greetings,

Over the past day, I've implemented a solution for the shared state
reuse issues Mentor has seen with our poky-based product. This
solution is similar in concept to what we had in our Mentor Embedded
Linux 4 (non-yocto-based) product.

This implementation restructures SSTATE_DIR such that non-target
sstate archives are placed in a directory specific to the host we're
running on, and allows fallback to sstates from compatible hosts. In
addition, there is a hook in place for modifying the returned build
host identifier string. Using these capabilities, configured as you'll
see below at the gist, I can populate sstate-cache from a Centos5
machine and fully reuse that shared state on a u1004 machine, but if I
take the u1004 sstate-cache and pull it over to Centos5, all the
non-target recipes will be rebuilt.

This has a list of "compatible hosts", which are used as fallback
regardless of what distro you're running on, so assumes you won't be
running on a host older than the ones you're using as your
compatibility baseline. I think this will satisfy the needs of most,
and as you'll see when you look at the implementation, is entirely
opt-in currently, so does no harm to anyone who chooses not to utilize
it.

https://gist.github.com/2253903 - shows an example of how to make use
of this functionality
https://github.com/kergoth/oe-core/compare/sstate-structure - the implementation

Regarding the implementation, I realize it isn't as clean as it could
be, but the only way to resolve it in a cleaner way would be to modify
bitbake, which I wasn't prepared to do at this juncture. The
fundamental issue adding complexity is that SSTATE_DIR is pulled from
the configuration metadata, not cached per-recipe, so one can't
manipulate where individual recipes get their archives stored. As a
workaround, I set the global SSTATE_DIR to the host-bound location,
then when writing the archives for target recipes, moves the archive
up to the parent directory (to the root of the original SSTATE_DIR)
and symlinks it back.

Richard had proposed modifying the filename rather than the directory
structure (e.g. via the sstate package arch), but this would make
things far more complex. In order to implement fallback, one would
have to mangle the filename, and one wouldn't be able to simply
leverage SSTATE_MIRRORS to fetch the variants in a simple way, as it
would have to attempt to fetch multiple filenames, which would require
invasive changes to sstate.bbclass. I think using the directory
structure is the easiest and cleanest route to the goal without
invasive changes, and given it's opt-in nature, I'd like to see this
go upstream, at a minimum as a temporary measure until/if a longer
term more invasive solution occurs.

I'm looking for questions, comments, testers, and in particular,
thoughts on whether this will meet the needs of others with similar
requirements (e.g. others shipping metadata with associated shared
state).
-- 
Christopher Larson

Re: [RFC] One shared state reuse solution

[Chris Larson]

On Fri, Mar 30, 2012 at 2:54 PM, Chris Larson <clarson@kergoth.com> wrote:
> Over the past day, I've implemented a solution for the shared state
> reuse issues Mentor has seen with our poky-based product. This
> solution is similar in concept to what we had in our Mentor Embedded
> Linux 4 (non-yocto-based) product.
>
> This implementation restructures SSTATE_DIR such that non-target
> sstate archives are placed in a directory specific to the host we're
> running on, and allows fallback to sstates from compatible hosts. In
> addition, there is a hook in place for modifying the returned build
> host identifier string. Using these capabilities, configured as you'll
> see below at the gist, I can populate sstate-cache from a Centos5
> machine and fully reuse that shared state on a u1004 machine, but if I
> take the u1004 sstate-cache and pull it over to Centos5, all the
> non-target recipes will be rebuilt.

I forgot to mention, it allows fallback both within the SSTATE_DIR and
for mirrors in SSTATE_MIRRORS, as it iterates over each mirror and
adjusts the mirror uri to include each compatible host (including the
current one).
-- 
Christopher Larson

Re: [RFC] One shared state reuse solution

[Koen Kooi]

Op 30 mrt. 2012, om 14:54 heeft Chris Larson het volgende geschreven:

> Greetings,
> 
> Over the past day, I've implemented a solution for the shared state
> reuse issues Mentor has seen with our poky-based product. This
> solution is similar in concept to what we had in our Mentor Embedded
> Linux 4 (non-yocto-based) product.
> 
> This implementation restructures SSTATE_DIR such that non-target
> sstate archives are placed in a directory specific to the host we're
> running on, and allows fallback to sstates from compatible hosts. In
> addition, there is a hook in place for modifying the returned build
> host identifier string. Using these capabilities, configured as you'll
> see below at the gist, I can populate sstate-cache from a Centos5
> machine and fully reuse that shared state on a u1004 machine, but if I
> take the u1004 sstate-cache and pull it over to Centos5, all the
> non-target recipes will be rebuilt.
> 
> This has a list of "compatible hosts", which are used as fallback
> regardless of what distro you're running on, so assumes you won't be
> running on a host older than the ones you're using as your
> compatibility baseline. I think this will satisfy the needs of most,
> and as you'll see when you look at the implementation, is entirely
> opt-in currently, so does no harm to anyone who chooses not to utilize
> it.
> 
> https://gist.github.com/2253903 - shows an example of how to make use
> of this functionality
> https://github.com/kergoth/oe-core/compare/sstate-structure - the implementation
> 
> Regarding the implementation, I realize it isn't as clean as it could
> be, but the only way to resolve it in a cleaner way would be to modify
> bitbake, which I wasn't prepared to do at this juncture. The
> fundamental issue adding complexity is that SSTATE_DIR is pulled from
> the configuration metadata, not cached per-recipe, so one can't
> manipulate where individual recipes get their archives stored. As a
> workaround, I set the global SSTATE_DIR to the host-bound location,
> then when writing the archives for target recipes, moves the archive
> up to the parent directory (to the root of the original SSTATE_DIR)
> and symlinks it back.
> 
> Richard had proposed modifying the filename rather than the directory
> structure (e.g. via the sstate package arch), but this would make
> things far more complex. In order to implement fallback, one would
> have to mangle the filename, and one wouldn't be able to simply
> leverage SSTATE_MIRRORS to fetch the variants in a simple way, as it
> would have to attempt to fetch multiple filenames, which would require
> invasive changes to sstate.bbclass. I think using the directory
> structure is the easiest and cleanest route to the goal without
> invasive changes, and given it's opt-in nature, I'd like to see this
> go upstream, at a minimum as a temporary measure until/if a longer
> term more invasive solution occurs.
> 
> I'm looking for questions, comments, testers, and in particular,
> thoughts on whether this will meet the needs of others with similar
> requirements (e.g. others shipping metadata with associated shared
> state).

This is not strictly related to your patchset, but has anyone thought about license based blacklisting of sstate? I can imagine that an autobuilder will build everything, includes things like evil 3d drivers, but no want anyone to access the sstate for those builds.

regards,

Koen

Re: [RFC] One shared state reuse solution

[Chris Larson]

On Fri, Mar 30, 2012 at 3:02 PM, Koen Kooi <koen@dominion.thruhere.net> wrote:
> Op 30 mrt. 2012, om 14:54 heeft Chris Larson het volgende geschreven:

>> I'm looking for questions, comments, testers, and in particular,
>> thoughts on whether this will meet the needs of others with similar
>> requirements (e.g. others shipping metadata with associated shared
>> state).
>
> This is not strictly related to your patchset, but has anyone thought about license based blacklisting of sstate? I can imagine that an autobuilder will build everything, includes things like evil 3d drivers, but no want anyone to access the sstate for those builds.

Oh, that's a very good point. I hadn't considered that aspect. I
haven't seen any implementation in that regard (actually, I have no
idea how one goes about disabling sstate period -- it'd be nice if
there was an easy variable rather than having to go deleting tasks and
the like).
-- 
Christopher Larson

Re: [RFC] One shared state reuse solution

[Chris Larson]

On Fri, Mar 30, 2012 at 3:02 PM, Koen Kooi <koen@dominion.thruhere.net> wrote:
> Op 30 mrt. 2012, om 14:54 heeft Chris Larson het volgende geschreven:
>
>> Greetings,
>>
>> Over the past day, I've implemented a solution for the shared state
>> reuse issues Mentor has seen with our poky-based product. This
>> solution is similar in concept to what we had in our Mentor Embedded
>> Linux 4 (non-yocto-based) product.
>>
>> This implementation restructures SSTATE_DIR such that non-target
>> sstate archives are placed in a directory specific to the host we're
>> running on, and allows fallback to sstates from compatible hosts. In
>> addition, there is a hook in place for modifying the returned build
>> host identifier string. Using these capabilities, configured as you'll
>> see below at the gist, I can populate sstate-cache from a Centos5
>> machine and fully reuse that shared state on a u1004 machine, but if I
>> take the u1004 sstate-cache and pull it over to Centos5, all the
>> non-target recipes will be rebuilt.
>>
>> This has a list of "compatible hosts", which are used as fallback
>> regardless of what distro you're running on, so assumes you won't be
>> running on a host older than the ones you're using as your
>> compatibility baseline. I think this will satisfy the needs of most,
>> and as you'll see when you look at the implementation, is entirely
>> opt-in currently, so does no harm to anyone who chooses not to utilize
>> it.
>>
>> https://gist.github.com/2253903 - shows an example of how to make use
>> of this functionality
>> https://github.com/kergoth/oe-core/compare/sstate-structure - the implementation
>>
>> Regarding the implementation, I realize it isn't as clean as it could
>> be, but the only way to resolve it in a cleaner way would be to modify
>> bitbake, which I wasn't prepared to do at this juncture. The
>> fundamental issue adding complexity is that SSTATE_DIR is pulled from
>> the configuration metadata, not cached per-recipe, so one can't
>> manipulate where individual recipes get their archives stored. As a
>> workaround, I set the global SSTATE_DIR to the host-bound location,
>> then when writing the archives for target recipes, moves the archive
>> up to the parent directory (to the root of the original SSTATE_DIR)
>> and symlinks it back.
>>
>> Richard had proposed modifying the filename rather than the directory
>> structure (e.g. via the sstate package arch), but this would make
>> things far more complex. In order to implement fallback, one would
>> have to mangle the filename, and one wouldn't be able to simply
>> leverage SSTATE_MIRRORS to fetch the variants in a simple way, as it
>> would have to attempt to fetch multiple filenames, which would require
>> invasive changes to sstate.bbclass. I think using the directory
>> structure is the easiest and cleanest route to the goal without
>> invasive changes, and given it's opt-in nature, I'd like to see this
>> go upstream, at a minimum as a temporary measure until/if a longer
>> term more invasive solution occurs.
>>
>> I'm looking for questions, comments, testers, and in particular,
>> thoughts on whether this will meet the needs of others with similar
>> requirements (e.g. others shipping metadata with associated shared
>> state).
>
> This is not strictly related to your patchset, but has anyone thought about license based blacklisting of sstate? I can imagine that an autobuilder will build everything, includes things like evil 3d drivers, but no want anyone to access the sstate for those builds.

I'd think that this would best be handled as a part of population of
the shared state mirror. That is, we could create a class like
copyleft_compliance, but for population of a shared state repository,
obeying licensing for distribution constraints.
-- 
Christopher Larson

Re: [RFC] One shared state reuse solution

[Koen Kooi]

Op 30 mrt. 2012, om 16:23 heeft Chris Larson het volgende geschreven:

> On Fri, Mar 30, 2012 at 3:02 PM, Koen Kooi <koen@dominion.thruhere.net> wrote:
>> Op 30 mrt. 2012, om 14:54 heeft Chris Larson het volgende geschreven:
>> 
>>> Greetings,
>>> 
>>> Over the past day, I've implemented a solution for the shared state
>>> reuse issues Mentor has seen with our poky-based product. This
>>> solution is similar in concept to what we had in our Mentor Embedded
>>> Linux 4 (non-yocto-based) product.
>>> 
>>> This implementation restructures SSTATE_DIR such that non-target
>>> sstate archives are placed in a directory specific to the host we're
>>> running on, and allows fallback to sstates from compatible hosts. In
>>> addition, there is a hook in place for modifying the returned build
>>> host identifier string. Using these capabilities, configured as you'll
>>> see below at the gist, I can populate sstate-cache from a Centos5
>>> machine and fully reuse that shared state on a u1004 machine, but if I
>>> take the u1004 sstate-cache and pull it over to Centos5, all the
>>> non-target recipes will be rebuilt.
>>> 
>>> This has a list of "compatible hosts", which are used as fallback
>>> regardless of what distro you're running on, so assumes you won't be
>>> running on a host older than the ones you're using as your
>>> compatibility baseline. I think this will satisfy the needs of most,
>>> and as you'll see when you look at the implementation, is entirely
>>> opt-in currently, so does no harm to anyone who chooses not to utilize
>>> it.
>>> 
>>> https://gist.github.com/2253903 - shows an example of how to make use
>>> of this functionality
>>> https://github.com/kergoth/oe-core/compare/sstate-structure - the implementation
>>> 
>>> Regarding the implementation, I realize it isn't as clean as it could
>>> be, but the only way to resolve it in a cleaner way would be to modify
>>> bitbake, which I wasn't prepared to do at this juncture. The
>>> fundamental issue adding complexity is that SSTATE_DIR is pulled from
>>> the configuration metadata, not cached per-recipe, so one can't
>>> manipulate where individual recipes get their archives stored. As a
>>> workaround, I set the global SSTATE_DIR to the host-bound location,
>>> then when writing the archives for target recipes, moves the archive
>>> up to the parent directory (to the root of the original SSTATE_DIR)
>>> and symlinks it back.
>>> 
>>> Richard had proposed modifying the filename rather than the directory
>>> structure (e.g. via the sstate package arch), but this would make
>>> things far more complex. In order to implement fallback, one would
>>> have to mangle the filename, and one wouldn't be able to simply
>>> leverage SSTATE_MIRRORS to fetch the variants in a simple way, as it
>>> would have to attempt to fetch multiple filenames, which would require
>>> invasive changes to sstate.bbclass. I think using the directory
>>> structure is the easiest and cleanest route to the goal without
>>> invasive changes, and given it's opt-in nature, I'd like to see this
>>> go upstream, at a minimum as a temporary measure until/if a longer
>>> term more invasive solution occurs.
>>> 
>>> I'm looking for questions, comments, testers, and in particular,
>>> thoughts on whether this will meet the needs of others with similar
>>> requirements (e.g. others shipping metadata with associated shared
>>> state).
>> 
>> This is not strictly related to your patchset, but has anyone thought about license based blacklisting of sstate? I can imagine that an autobuilder will build everything, includes things like evil 3d drivers, but no want anyone to access the sstate for those builds.
> 
> I'd think that this would best be handled as a part of population of
> the shared state mirror. That is, we could create a class like
> copyleft_compliance, but for population of a shared state repository,
> obeying licensing for distribution constraints.

That was my idea as well, but I worded my question too vague :)

regards,

Koen

Re: [RFC] One shared state reuse solution

[Mark Hatle]

(top posting, ya I know...)

We've worked on a similar problem in the past.  In specific cases we've added a 
checksum of the host's glibc to the mix.  The problem we were solving was in 
glibc uprevs during the RHEL 4 world.. new APIs would arrive and things would 
break -- or workaround would break.

Have you considered adding that to the mix in an attempt to help determine 
compatible host distributions?

I do like the idea of directory based sstate cache.  It can more easily identify 
what the components belong to.  (I wouldn't mind some type of hierarchy for the 
target stuff as well, but so far I'm not sure exactly it would look like or 
trigger off of.)

However, shy of directory based.. adjusting the generated checksum in some way 
should be enough -- the downside is how do you detect compatible hosts or not.. 
  You may have to generate multiple checksums and look for best matches, which I 
suspect is also new code.

--Mark

On 3/30/12 4:54 PM, Chris Larson wrote:
> Greetings,
>
> Over the past day, I've implemented a solution for the shared state
> reuse issues Mentor has seen with our poky-based product. This
> solution is similar in concept to what we had in our Mentor Embedded
> Linux 4 (non-yocto-based) product.
>
> This implementation restructures SSTATE_DIR such that non-target
> sstate archives are placed in a directory specific to the host we're
> running on, and allows fallback to sstates from compatible hosts. In
> addition, there is a hook in place for modifying the returned build
> host identifier string. Using these capabilities, configured as you'll
> see below at the gist, I can populate sstate-cache from a Centos5
> machine and fully reuse that shared state on a u1004 machine, but if I
> take the u1004 sstate-cache and pull it over to Centos5, all the
> non-target recipes will be rebuilt.
>
> This has a list of "compatible hosts", which are used as fallback
> regardless of what distro you're running on, so assumes you won't be
> running on a host older than the ones you're using as your
> compatibility baseline. I think this will satisfy the needs of most,
> and as you'll see when you look at the implementation, is entirely
> opt-in currently, so does no harm to anyone who chooses not to utilize
> it.
>
> https://gist.github.com/2253903 - shows an example of how to make use
> of this functionality
> https://github.com/kergoth/oe-core/compare/sstate-structure - the implementation
>
> Regarding the implementation, I realize it isn't as clean as it could
> be, but the only way to resolve it in a cleaner way would be to modify
> bitbake, which I wasn't prepared to do at this juncture. The
> fundamental issue adding complexity is that SSTATE_DIR is pulled from
> the configuration metadata, not cached per-recipe, so one can't
> manipulate where individual recipes get their archives stored. As a
> workaround, I set the global SSTATE_DIR to the host-bound location,
> then when writing the archives for target recipes, moves the archive
> up to the parent directory (to the root of the original SSTATE_DIR)
> and symlinks it back.
>
> Richard had proposed modifying the filename rather than the directory
> structure (e.g. via the sstate package arch), but this would make
> things far more complex. In order to implement fallback, one would
> have to mangle the filename, and one wouldn't be able to simply
> leverage SSTATE_MIRRORS to fetch the variants in a simple way, as it
> would have to attempt to fetch multiple filenames, which would require
> invasive changes to sstate.bbclass. I think using the directory
> structure is the easiest and cleanest route to the goal without
> invasive changes, and given it's opt-in nature, I'd like to see this
> go upstream, at a minimum as a temporary measure until/if a longer
> term more invasive solution occurs.
>
> I'm looking for questions, comments, testers, and in particular,
> thoughts on whether this will meet the needs of others with similar
> requirements (e.g. others shipping metadata with associated shared
> state).

Re: [RFC] One shared state reuse solution

[Chris Larson]

On Fri, Mar 30, 2012 at 3:04 PM, Mark Hatle <mark.hatle@windriver.com> wrote:
>
> We've worked on a similar problem in the past.  In specific cases we've
> added a checksum of the host's glibc to the mix.  The problem we were
> solving was in glibc uprevs during the RHEL 4 world.. new APIs would arrive
> and things would break -- or workaround would break.
>
> Have you considered adding that to the mix in an attempt to help determine
> compatible host distributions?

It's been discussed in the past, but the problem is, as far as I know,
glibc isn't the only build host dependency we have. It's the most
visible, since glibc versioned symbols are the first place one
generally sees this sort of failure, but I think the safe bet is to
operate based upon the distro name/version, to avoid any potential
other compatibility issues with other host libraries that get linked
against. It'd be nice to ensure that glibc is the only dependency, at
which point that sort of approach would be more reliable, but I don't
think we're there yet (please correct me if I'm wrong here).

> I do like the idea of directory based sstate cache.  It can more easily
> identify what the components belong to.  (I wouldn't mind some type of
> hierarchy for the target stuff as well, but so far I'm not sure exactly it
> would look like or trigger off of.)
>
> However, shy of directory based.. adjusting the generated checksum in some
> way should be enough -- the downside is how do you detect compatible hosts
> or not..  You may have to generate multiple checksums and look for best
> matches, which I suspect is also new code.

What we had before this was just injection of the lsb_release
identifier/release into the native/cross signatures via vardeps, but
as you say, dealing with compatible hosts is non-trivial. I'm
certainly open to that sort of approach, but as you say, I think
there's other value to this approach, and seems a bit less confusing
:) Thanks for the comments, it's certainly a non-trivial problem to
solve.
-- 
Christopher Larson

Re: [RFC] One shared state reuse solution

[Mark Hatle]

On 3/30/12 5:09 PM, Chris Larson wrote:
> On Fri, Mar 30, 2012 at 3:04 PM, Mark Hatle<mark.hatle@windriver.com>  wrote:
>>
>> We've worked on a similar problem in the past.  In specific cases we've
>> added a checksum of the host's glibc to the mix.  The problem we were
>> solving was in glibc uprevs during the RHEL 4 world.. new APIs would arrive
>> and things would break -- or workaround would break.
>>
>> Have you considered adding that to the mix in an attempt to help determine
>> compatible host distributions?
>
> It's been discussed in the past, but the problem is, as far as I know,
> glibc isn't the only build host dependency we have. It's the most
> visible, since glibc versioned symbols are the first place one
> generally sees this sort of failure, but I think the safe bet is to
> operate based upon the distro name/version, to avoid any potential
> other compatibility issues with other host libraries that get linked
> against. It'd be nice to ensure that glibc is the only dependency, at
> which point that sort of approach would be more reliable, but I don't
> think we're there yet (please correct me if I'm wrong here).

It would be nice if the host libc was the only dependency.  ;)

Have you run any type of scan on the host dependencies to see where we actually 
sit?  I know a while back I ran a scan and was surprised at how few host 
dependencies there was in a fairly standard oe-core build.

--Mark

>> I do like the idea of directory based sstate cache.  It can more easily
>> identify what the components belong to.  (I wouldn't mind some type of
>> hierarchy for the target stuff as well, but so far I'm not sure exactly it
>> would look like or trigger off of.)
>>
>> However, shy of directory based.. adjusting the generated checksum in some
>> way should be enough -- the downside is how do you detect compatible hosts
>> or not..  You may have to generate multiple checksums and look for best
>> matches, which I suspect is also new code.
>
> What we had before this was just injection of the lsb_release
> identifier/release into the native/cross signatures via vardeps, but
> as you say, dealing with compatible hosts is non-trivial. I'm
> certainly open to that sort of approach, but as you say, I think
> there's other value to this approach, and seems a bit less confusing
> :) Thanks for the comments, it's certainly a non-trivial problem to
> solve.

Re: [RFC] One shared state reuse solution

[Chris Larson]

On Fri, Mar 30, 2012 at 3:17 PM, Mark Hatle <mark.hatle@windriver.com> wrote:
> On 3/30/12 5:09 PM, Chris Larson wrote:
>>
>> On Fri, Mar 30, 2012 at 3:04 PM, Mark Hatle<mark.hatle@windriver.com>
>>  wrote:
>>>
>>>
>>> We've worked on a similar problem in the past.  In specific cases we've
>>> added a checksum of the host's glibc to the mix.  The problem we were
>>> solving was in glibc uprevs during the RHEL 4 world.. new APIs would
>>> arrive
>>> and things would break -- or workaround would break.
>>>
>>> Have you considered adding that to the mix in an attempt to help
>>> determine
>>> compatible host distributions?
>>
>>
>> It's been discussed in the past, but the problem is, as far as I know,
>> glibc isn't the only build host dependency we have. It's the most
>> visible, since glibc versioned symbols are the first place one
>> generally sees this sort of failure, but I think the safe bet is to
>> operate based upon the distro name/version, to avoid any potential
>> other compatibility issues with other host libraries that get linked
>> against. It'd be nice to ensure that glibc is the only dependency, at
>> which point that sort of approach would be more reliable, but I don't
>> think we're there yet (please correct me if I'm wrong here).
>
>
> It would be nice if the host libc was the only dependency.  ;)
>
> Have you run any type of scan on the host dependencies to see where we
> actually sit?  I know a while back I ran a scan and was surprised at how few
> host dependencies there was in a fairly standard oe-core build.

I haven't -- that's a very good idea though. In fact, I hate to admit
it, but I've never even run swabber before :) It's on the 'when I find
the time' list..
-- 
Christopher Larson