Best way to "freeze" Yocto for production

Best way to "freeze" Yocto for production

[Marlon Smith]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]

Hi everyone,

We've decided to use Yocto in our company's product!  And I have a
question:

Since Yocto is updated pretty frequently, at some point we'll need to
freeze everything and keep a permanent copy so that we can make small
changes/bug fixes without having to worry about anything being changed
outside of our control.  My current theory is that we'll do a bitbake -c
fetchall, then zip up the entire Yocto directory and save it somewhere
so we'll always have a static copy.  We'll place our application and
custom bsp layer under a separate Git repository, and then to do a build
we'll just combine everything together and run bitbake on the whole
thing.

Is there a better or recommended way of doing this?

Thanks again, and I'm excited to start working with Yocto!

Marlon

[-- Attachment #2: Type: text/html, Size: 1088 bytes --]

Re: Best way to "freeze" Yocto for production

[Philip Balister]

On 06/10/2014 10:31 AM, Marlon Smith wrote:
> Hi everyone,
> 
> We've decided to use Yocto in our company's product!  And I have a
> question:
> 
> Since Yocto is updated pretty frequently, at some point we'll need to
> freeze everything and keep a permanent copy so that we can make small
> changes/bug fixes without having to worry about anything being changed
> outside of our control.  My current theory is that we'll do a bitbake -c
> fetchall, then zip up the entire Yocto directory and save it somewhere
> so we'll always have a static copy.  We'll place our application and
> custom bsp layer under a separate Git repository, and then to do a build
> we'll just combine everything together and run bitbake on the whole
> thing.
> 
> Is there a better or recommended way of doing this?

This is how I provide a stable base for people doing SDR work with OE.

https://github.com/balister/oe-gnuradio-manifest/tree/daisy

I'm sure you'll get several other suggestions also.

Pick what works best for your team.

Philip

> 
> Thanks again, and I'm excited to start working with Yocto!
> 
> Marlon
> 
> 
> 

Re: Best way to "freeze" Yocto for production

[Burton, Ross]

On 10 June 2014 18:31, Marlon Smith <marlon.smith10@gmail.com> wrote:
> Since Yocto is updated pretty frequently, at some point we'll need to freeze
> everything and keep a permanent copy so that we can make small changes/bug
> fixes without having to worry about anything being changed outside of our
> control.  My current theory is that we'll do a bitbake -c fetchall, then zip
> up the entire Yocto directory and save it somewhere so we'll always have a
> static copy.  We'll place our application and custom bsp layer under a
> separate Git repository, and then to do a build we'll just combine
> everything together and run bitbake on the whole thing.

The source archiver can be used to keep a "known good" copy of the
sources used, or you can just backup the DL_DIR (put it on a server
and use it as a source mirror for good measure). For your layers, I
recommend keeping them in git and document what oe-core/poky/bitbake
versions your own code has been tested against.  For any changes you
have against oe-core/bitbake/poky them in git clones of the relevant
repository, branched off the relevant release branches (daisy, dylan,
etc).

Ross

Re: Best way to "freeze" Yocto for production

[Flanagan, Elizabeth]

On Tue, Jun 10, 2014 at 10:31 AM, Marlon Smith <marlon.smith10@gmail.com> wrote:
> Hi everyone,
>
> We've decided to use Yocto in our company's product!  And I have a question:
>
> Since Yocto is updated pretty frequently, at some point we'll need to freeze
> everything and keep a permanent copy so that we can make small changes/bug
> fixes without having to worry about anything being changed outside of our
> control.  My current theory is that we'll do a bitbake -c fetchall, then zip
> up the entire Yocto directory and save it somewhere so we'll always have a
> static copy.

You shouldn't really need to worry about anything changing outside of
your control if you're doing this right. The SRC_URI[md5sums] will
yell if something changes out from under you. That said....

> We'll place our application and custom bsp layer under a
> separate Git repository, and then to do a build we'll just combine
> everything together and run bitbake on the whole thing.

That sounds about right and is how I would do it (and do do it via the yocto-ab)

>
> Is there a better or recommended way of doing this?

The git strategy sounds ok. I wouldn't do the zip everything up necessarily.

I would suggest something similar to what we're doing on the autobuilder.

DL_DIR is at nas:/mount/dl_dir

nightly does a bitbake -c fetchall. It dumps into DL_DIR. a cron
rsyncs DL_DIR to http://internal.yourorg.com/src_mirror

When you're ready to lock down your base OS, set PREMIRRORS to your
internal mirror.

I would not expose that mirror externally if you have any proprietary
IP that may be in DL_DIR; for GPL compliance, you'll want to utilize
the source archiver.

>
> Thanks again, and I'm excited to start working with Yocto!
>
> Marlon
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

-b

-- 
Elizabeth Flanagan
Yocto Project
Build and Release

Re: Best way to "freeze" Yocto for production

[Mark Hatle]

On 6/10/14, 12:31 PM, Marlon Smith wrote:
> Hi everyone,
>
> We've decided to use Yocto in our company's product!  And I have a question:
>
> Since Yocto is updated pretty frequently, at some point we'll need to freeze
> everything and keep a permanent copy so that we can make small changes/bug fixes
> without having to worry about anything being changed outside of our control.  My
> current theory is that we'll do a bitbake -c fetchall, then zip up the entire
> Yocto directory and save it somewhere so we'll always have a static copy.  We'll

This is really the first step.  Take you Yocto Project sources, and related 
downloads and store them for retrieval later.  You can use the:

    BB_NO_NETWORK = "1"

to ensure that the system will use your local copy and not attempt to go to the 
network for downloads.

> place our application and custom bsp layer under a separate Git repository, and
> then to do a build we'll just combine everything together and run bitbake on the
> whole thing.

This pretty much is the standard way to do it.  If your BSP is based off of 
another git tree, you can enable networking just for that one item if necessary.

> Is there a better or recommended way of doing this?

I think you have the basics down.  When it comes to upgrading and bug fixing, 
this is where doing very management resources come into play.

One way to do it is just track the Yocto Project until your product release. 
This way you get all of the latest fixes for the branch you choose.

Another mechanism (what I'd recommend for after product release, but some do it 
during development) is to watch the changes from your starting location, to the 
current state of development (either in the branch, or even master).  Using this 
information, you can determine if any of the items are fixes that you believe 
your product will require.  Use the git 'cherry-pick' option to pull down only 
the fixes you are interested in.  This dramatically slows the rate-of-change of 
your local tree.

One thing to keep in mind, you will -always- end up with some local fix that 
does not exist upstream.  You should limit these as much as possible.  For a 
single project, it's not all that painful to carry and support your changes, but 
when you start working on the next product, it's nice to be able to start with a 
newer version of the Yocto Project.  If you have a lot of local changes that 
have not gone back to the community, you'll then have to abandon them -- or 
carry them forward.  This can be time consuming, and the usual reaction is to 
stick with the older version to avoid this additional work.

You should always budget the time to get the fixes ready and send them back to 
the community, it will save you (and others) time when getting ready for the 
next project.

> Thanks again, and I'm excited to start working with Yocto!
>
> Marlon
>
>

Re: Best way to "freeze" Yocto for production

[Cliff Brake]

On Tue, Jun 10, 2014 at 1:31 PM, Marlon Smith <marlon.smith10@gmail.com> wrote:
> Hi everyone,
>
> We've decided to use Yocto in our company's product!  And I have a question:
>
> Since Yocto is updated pretty frequently, at some point we'll need to freeze
> everything and keep a permanent copy so that we can make small changes/bug
> fixes without having to worry about anything being changed outside of our
> control.  My current theory is that we'll do a bitbake -c fetchall, then zip
> up the entire Yocto directory and save it somewhere so we'll always have a
> static copy.  We'll place our application and custom bsp layer under a
> separate Git repository, and then to do a build we'll just combine
> everything together and run bitbake on the whole thing.

Lately, I've been implementing projects based on yocto releases (dora,
daisy, etc).  Most layers have corresponding branches.  You can use
git submodules or repo to lock everything down.  Philip provided a
repo example.  Here are a couple more:

https://github.com/cbrake/oe-build (uses submodules)
https://github.com/kraj/angstrom-manifest (uses repo)

Custom bits for a project goes in a new layer.  If anything needs to
change in an upstream layer, I typically put a bbappend in the custom
layer, rather than modifying an upstream layer, and then needing to
maintain my own copies of the upstream repos.

I'm sure there are many other ways ...

Cliff