Yocto & long-term reproducibility of rebuilds

Yocto & long-term reproducibility of rebuilds

[paul_nathan]

[-- Attachment #1: Type: text/plain, Size: 851 bytes --]

Hi,

I am investigating Yocto for a build system here. Part of my requirements 
for a build system is auditable/traceable builds that can be replicated 
long into the future (our company has a 10 year warranty on our products, 
and we build products for the multidecade term). Initial examination of 
Yocto shows that it builds packages from a number of different domains 
online, which will not meet our requirements - we'll have to store these 
packages to ensure these packages exist in the correct version long into 
the future.  I know we can manually edit the SRC_URI setting in .bb files, 
but the more general problem of package archiving exists. 

I am interested in any information or recommendations other users of the 
Yocto system have on how they have solved this sort of design constraint.

- - -
Regards,
Paul Nathan

[-- Attachment #2: Type: text/html, Size: 1033 bytes --]

Re: Yocto & long-term reproducibility of rebuilds

[Khem Raj]

On Mon, Jan 30, 2012 at 8:34 AM,  <paul_nathan@selinc.com> wrote:
> Hi,
>
> I am investigating Yocto for a build system here. Part of my requirements
> for a build system is auditable/traceable builds that can be replicated long
> into the future (our company has a 10 year warranty on our products, and we
> build products for the multidecade term). Initial examination of Yocto shows
> that it builds packages from a number of different domains online, which
> will not meet our requirements - we'll have to store these packages to
> ensure these packages exist in the correct version long into the future.  I
> know we can manually edit the SRC_URI setting in .bb files, but the more
> general problem of package archiving exists.
>
> I am interested in any information or recommendations other users of the
> Yocto system have on how they have solved this sort of design constraint.

Hi Paul

You can cache all the source tars as you mentioned and then
store them internally and use PREMIRROR facility which should
check premirrors before it hits src_uri so provided you cache
all the sources it should always get it from your internal
mirror

>
> - - -
> Regards,
> Paul Nathan
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

Re: Yocto & long-term reproducibility of rebuilds

[Gary Thomas]

On 2012-01-30 10:37, Khem Raj wrote:
> On Mon, Jan 30, 2012 at 8:34 AM,<paul_nathan@selinc.com>  wrote:
>> Hi,
>>
>> I am investigating Yocto for a build system here. Part of my requirements
>> for a build system is auditable/traceable builds that can be replicated long
>> into the future (our company has a 10 year warranty on our products, and we
>> build products for the multidecade term). Initial examination of Yocto shows
>> that it builds packages from a number of different domains online, which
>> will not meet our requirements - we'll have to store these packages to
>> ensure these packages exist in the correct version long into the future.  I
>> know we can manually edit the SRC_URI setting in .bb files, but the more
>> general problem of package archiving exists.
>>
>> I am interested in any information or recommendations other users of the
>> Yocto system have on how they have solved this sort of design constraint.
>
> Hi Paul
>
> You can cache all the source tars as you mentioned and then
> store them internally and use PREMIRROR facility which should
> check premirrors before it hits src_uri so provided you cache
> all the sources it should always get it from your internal
> mirror

Check the archives - there was an extensive discussion of this last week
   '[yocto] tar ball vs. git development questions'

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Yocto & long-term reproducibility of rebuilds

[Frans Meulenbroeks]

[-- Attachment #1: Type: text/plain, Size: 1707 bytes --]

2012/1/30 <paul_nathan@selinc.com>

> Hi,
>
> I am investigating Yocto for a build system here. Part of my requirements
> for a build system is auditable/traceable builds that can be replicated
> long into the future (our company has a 10 year warranty on our products,
> and we build products for the multidecade term). Initial examination of
> Yocto shows that it builds packages from a number of different domains
> online, which will not meet our requirements - we'll have to store these
> packages to ensure these packages exist in the correct version long into
> the future.  I know we can manually edit the SRC_URI setting in .bb files,
> but the more general problem of package archiving exists.
>
> I am interested in any information or recommendations other users of the
> Yocto system have on how they have solved this sort of design constraint.
>
> - - -
> Regards,
> Paul Nathan
>
> Paul,

My corrent project is still on oe-classic not on yocto, but we also aim for
reproducible builds.
Basically we perform the following steps.

checkout the opembedded repo. We use a specific hash not head as we do not
want unexpected updates
create a tarball of the git tree
build your product
save all files in the downloads dir.
put that on a local ftp mirror, add a premirror for that ftp mirror
rebuild only using that mirror (e.g. disallow your build host to go to
another external system than your ftp mirror.
archive your git tarball and the saved downloads dir

That's it.
Actually you might then also want to put the stuff on your local ftp mirror
as well as the oe tarball on a public ftp site in order to fulfill your GPL
obligations

Good luck!
Frans

[-- Attachment #2: Type: text/html, Size: 2130 bytes --]

Re: Yocto & long-term reproducibility of rebuilds

[Flanagan, Elizabeth]

On Mon, Jan 30, 2012 at 8:34 AM,  <paul_nathan@selinc.com> wrote:
> Hi,
>
> I am investigating Yocto for a build system here. Part of my requirements
> for a build system is auditable/traceable builds that can be replicated long
> into the future (our company has a 10 year warranty on our products, and we
> build products for the multidecade term). Initial examination of Yocto shows
> that it builds packages from a number of different domains online, which
> will not meet our requirements - we'll have to store these packages to
> ensure these packages exist in the correct version long into the future.  I
> know we can manually edit the SRC_URI setting in .bb files, but the more
> general problem of package archiving exists.
>

I would avoid manually editing SRC_URI.

Set up PREMIRRORS and BB_FETCH_PREMIRRORONLY. That would ensure that
you avoided any upstream contamination.

-b

> I am interested in any information or recommendations other users of the
> Yocto system have on how they have solved this sort of design constraint.
>
> - - -
> Regards,
> Paul Nathan
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>



-- 
Elizabeth Flanagan
Yocto Project
Build and Release

Re: Yocto & long-term reproducibility of rebuilds

[Chris Tapp]

On 30 Jan 2012, at 18:34, Flanagan, Elizabeth wrote:

> On Mon, Jan 30, 2012 at 8:34 AM,  <paul_nathan@selinc.com> wrote:
>> Hi,
>>
>> I am investigating Yocto for a build system here. Part of my  
>> requirements
>> for a build system is auditable/traceable builds that can be  
>> replicated long
>> into the future (our company has a 10 year warranty on our  
>> products, and we
>> build products for the multidecade term). Initial examination of  
>> Yocto shows
>> that it builds packages from a number of different domains online,  
>> which
>> will not meet our requirements - we'll have to store these packages  
>> to
>> ensure these packages exist in the correct version long into the  
>> future.  I
>> know we can manually edit the SRC_URI setting in .bb files, but the  
>> more
>> general problem of package archiving exists.
>>
>
> I would avoid manually editing SRC_URI.
>
> Set up PREMIRRORS and BB_FETCH_PREMIRRORONLY. That would ensure that
> you avoided any upstream contamination.
>
> I am interested in any information or recommendations other users of  
> the
>> Yocto system have on how they have solved this sort of design  
>> constraint.

This sounds like a good a good article for the 'How Do I...' section  
on the wiki. Or is there already something similar on there?

Chris Tapp

opensource@keylevel.com
www.keylevel.com

Re: Yocto & long-term reproducibility of rebuilds

[Richard Purdie]

On Mon, 2012-01-30 at 08:34 -0800, paul_nathan@selinc.com wrote:
> I am investigating Yocto for a build system here. Part of my
> requirements for a build system is auditable/traceable builds that can
> be replicated long into the future (our company has a 10 year warranty
> on our products, and we build products for the multidecade term).
> Initial examination of Yocto shows that it builds packages from a
> number of different domains online, which will not meet our
> requirements - we'll have to store these packages to ensure these
> packages exist in the correct version long into the future.  I know we
> can manually edit the SRC_URI setting in .bb files, but the more
> general problem of package archiving exists. 
>
> I am interested in any information or recommendations other users of
> the Yocto system have on how they have solved this sort of design
> constraint. 

This is something we've worked to ensure works since we understand the
importance of long term reproducibility. 

Others have mentioned how a local source cache can be created and how
MIRRORS and PREMIRRORS can be used to override or supplement to the
SRC_URI. In addition I'd mention we have a variable BB_NO_NETWORK which
disables all network access so you can ensure only local files are used
in a given build.

Cheers,

Richard