My thoughts on the future of OE?

My thoughts on the future of OE?

[Richard Purdie]

I was asked what I thought were things that needed discussion at OEDAM.
Sadly I won't be there but I thought it might help to write down my
thoughts in a few areas.

Developer Workflow
------------------

Firstly, I think the big piece we need to address as a project is
"developer workflow" as this is where people are struggling using it.

Unfortunately "developer workflow" means different things to different
people? Which one do I mean then? I actually mean all of them. As some
examples:

-----------------------------------------------------------------------

* A kernel developer wanting to rebuild a kernel 
  [on/off target, with the ADT/SDK or a recipe]
* A kernel developer wanting to build a kernel module
  [on/off target, with the ADT/SDK or a recipe]
* An application developer wanting to build a single App
  [on/off target, with the ADT/SDK or a recipe]
* An application developer wanting to (re)build a library, linking an 
  App to it
  [on/off target, with the ADT/SDK or a recipe]
* A user wanting to rebuild an image with a package added
  [on and off target - feeds or a build]
* A user wanting to rebuild an image with more advanced changes

The user may want to skip the image creation step and deploy data
straight onto an existing running target using rsync.

Their application/kernel may be in an external directory already under
SCM control.

There are requests for "real time" viewing of the build logs

There are requests for the python devshell to better be able to poke
around the datastore

There are requests for a shell like environment with commands
interacting with a memory resident bitbake.

In a team environment how should patch review work? Which tools would we
recommend to help people? Where does and autobuilder fit into this?
Gerrit? How to handle bugs/regressions/features?

Also, when something fails, how do they get help and fix it? My dream
here is an error reporting server and over time, we can attach "fixes"
to those failures showing people how to potentially fix their problems.


-----------------------------------------------------------------------

So my first ask is that we actually try and write down all these
different cases which is no small task in itself. I've a start of a list
above, we should probably put this into the wiki and have people add
their own use cases (or use cases of those around them in their company
etc.). The trouble is there are some many different variants!

Once we have some idea of the cases, we can start to put together some
kind of plan about how we intend to help the given use cases and to try
and prioritise them. Perhaps we should put some kind of weighting
against them in the wiki and people can increase the numbers of say
their top three desires.

Whilst this looks like an impossible problem, the good news is that I
believe we can solve it and we actually already have a rather powerful
toolbox to help work on it. I have a rough idea of a roadmap that can
move us forward:

a) locked sstate - I know its not in master yet but I believe this is
key to allowing some of the use cases where you want to change some
number of things but keep the rest of the system the same.

b) Rework the ADT so it consists of a build of prebuilt objects and
locked sstate and bitbake is hiding behind the scenes. There would be
some kind of setup command (which would extract the toolchain and core
libs for example) and then they could use it like the existing ADT. 

c) We could then document workflows where you could build an extra app
into an image and so on.

d) In parallel with the above, we need to look at things like
externalsrc and see if there are ways we can better integrate it. Maybe
some helper scripts? Can it work with the new ADT/SDK?

Hopefully the above is enough to seed some discussion! :)


Automated XYZ
-------------

We've made great steps forward with the automation we have. I would like
to see us move forward in a few key areas:

a) Allow automated ptest running as part of our qemu image tests. This 
   is harder since it means:

     i) installing ptest packages
     ii) collecting up the results in some wya that can be parsed
     iii) allow comparison of existing results with previous runs
     iv) report regressions

b) Implement automated recipe upgrade detection. We have proof of 
   concept code for this and there are some logical steps for how to 
   proceed:

     i) I'd like to see it integrated into the fetcher codebase. There 
        are some proof of concept patches at:
        http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=46ba4664a8b1d2d59de5fba92e7e77929a0ee24d
        (and the preceding patches)
     ii) We will probably need to markup the recipes themselves with 
         some new data. Some example data:
         http://git.yoctoproject.org/cgit.cgi/poky/tree/meta-yocto/conf/distro/include/package_regex.inc

     iii) Feed the data from the above code into the package reporting system

   The parts are there and have been proven to roughly work. The next 
   step would be to bring them into the core and start using them 
   properly. Its not an easy thing to get right but I think the 
   potential benefits are worth it.

c) Implement code which attempts automated package upgrades and then 
   reports the success/failure. This depends on a) and b) for best 
   results.

d) Continue to move forward with automated testing on real hardware 
   too. The idea is that the above tests in a) should also be usable 
   here so we can test significantly more of the software stack than 
   we've ever been able to before.

e) Continue to add test cases for:

     i) the images (ptest and others)
     ii) oe-selftest
     iii) bitbake-selftest
     iv) add new tests like tests for the toaster UI

The basic idea here is that existing work done manually isn't going to
scale. We have all the foundations to be able to do something few
projects have ever done before and automate this on a scale not
previously seen. This means it would stop people from having to do
"boring" work and concentrate on the interesting pieces where the
problems are. It also means we should be able to track down regressions
much earlier and deal with them more quickly.

How soon will all of this work? Probably not overnight but I believe its
something to aim for.


Contributors to the Project
---------------------------

This is an open question. How do we attract more developers to work on
the project? This applies both to layers like OE-Core and outside that,
we could all use some help in the maintainership, the development and
the testing. What blocks people from joining us? How can we encourage
more people to participate?

Unfortunately when things are working for people they tend to think that
ok, they'll go and do something else (like work on their product!) but
is there a way we could get some involvement from them? Lots of small
pieces of help could build to a large whole...


Conclusion
----------

I've probably rambled enough. The project is I believe in a strong
position but the above are areas I think we need to work on if we want
to truly fulfil the project's potential.

Cheers,

Richard

Re: My thoughts on the future of OE?

[Stewart, David C]

davest throws in his two cents below...

On 5/1/14, 10:02 AM, "Richard Purdie" <richard.purdie@linuxfoundation.org>
wrote:

>I was asked what I thought were things that needed discussion at OEDAM.
>Sadly I won't be there but I thought it might help to write down my
>thoughts in a few areas.
>
>Developer Workflow
>------------------
>
>Firstly, I think the big piece we need to address as a project is
>"developer workflow" as this is where people are struggling using it.
>
>Unfortunately "developer workflow" means different things to different
>people? Which one do I mean then? I actually mean all of them. As some
>examples:
>
>-----------------------------------------------------------------------
>
>* A kernel developer wanting to rebuild a kernel
>  [on/off target, with the ADT/SDK or a recipe]
>* A kernel developer wanting to build a kernel module
>  [on/off target, with the ADT/SDK or a recipe]
>* An application developer wanting to build a single App
>  [on/off target, with the ADT/SDK or a recipe]
>* An application developer wanting to (re)build a library, linking an
>  App to it
>  [on/off target, with the ADT/SDK or a recipe]
>* A user wanting to rebuild an image with a package added
>  [on and off target - feeds or a build]
>* A user wanting to rebuild an image with more advanced changes
>
>The user may want to skip the image creation step and deploy data
>straight onto an existing running target using rsync.
>
>Their application/kernel may be in an external directory already under
>SCM control.
>
>There are requests for "real time" viewing of the build logs
>
>There are requests for the python devshell to better be able to poke
>around the datastore
>
>There are requests for a shell like environment with commands
>interacting with a memory resident bitbake.
>
>In a team environment how should patch review work? Which tools would we
>recommend to help people? Where does and autobuilder fit into this?
>Gerrit? How to handle bugs/regressions/features?
>
>Also, when something fails, how do they get help and fix it? My dream
>here is an error reporting server and over time, we can attach "fixes"
>to those failures showing people how to potentially fix their problems.

If I were managing engineers creating an embedded device (which I'm not),
and let's say I had a team of 10 engineers working on some application /
middleware / service which goes into a line of devices, I might add:

* If I need to add a recipe to my standard image builds or patch one, how
do I communicate that fact to the maintainer of my product's metadata?

* If a developer is ready to merge their code with the device's master
branch, how can this trigger an automated rebuild?

I have a feeling that OE today is well suited for a single developer or a
really small team, but it doesn't scale up particularly well to larger
teams. At least, not without a lot of work. Is there anything we can do to
help that?

>
>
>-----------------------------------------------------------------------
>
>So my first ask is that we actually try and write down all these
>different cases which is no small task in itself. I've a start of a list
>above, we should probably put this into the wiki and have people add
>their own use cases (or use cases of those around them in their company
>etc.). The trouble is there are some many different variants!
>
>Once we have some idea of the cases, we can start to put together some
>kind of plan about how we intend to help the given use cases and to try
>and prioritise them. Perhaps we should put some kind of weighting
>against them in the wiki and people can increase the numbers of say
>their top three desires.
>
>Whilst this looks like an impossible problem, the good news is that I
>believe we can solve it and we actually already have a rather powerful
>toolbox to help work on it. I have a rough idea of a roadmap that can
>move us forward:
>
>a) locked sstate - I know its not in master yet but I believe this is
>key to allowing some of the use cases where you want to change some
>number of things but keep the rest of the system the same.
>
>b) Rework the ADT so it consists of a build of prebuilt objects and
>locked sstate and bitbake is hiding behind the scenes. There would be
>some kind of setup command (which would extract the toolchain and core
>libs for example) and then they could use it like the existing ADT.
>
>c) We could then document workflows where you could build an extra app
>into an image and so on.
>
>d) In parallel with the above, we need to look at things like
>externalsrc and see if there are ways we can better integrate it. Maybe
>some helper scripts? Can it work with the new ADT/SDK?
>
>Hopefully the above is enough to seed some discussion! :)
>
>
>Automated XYZ
>-------------
>
>We've made great steps forward with the automation we have. I would like
>to see us move forward in a few key areas:
>
>a) Allow automated ptest running as part of our qemu image tests. This
>   is harder since it means:
>
>     i) installing ptest packages
>     ii) collecting up the results in some wya that can be parsed
>     iii) allow comparison of existing results with previous runs
>     iv) report regressions
>
>b) Implement automated recipe upgrade detection. We have proof of
>   concept code for this and there are some logical steps for how to
>   proceed:
>
>     i) I'd like to see it integrated into the fetcher codebase. There
>        are some proof of concept patches at:
>        
>http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&i
>d=46ba4664a8b1d2d59de5fba92e7e77929a0ee24d
>        (and the preceding patches)
>     ii) We will probably need to markup the recipes themselves with
>         some new data. Some example data:
>         
>http://git.yoctoproject.org/cgit.cgi/poky/tree/meta-yocto/conf/distro/incl
>ude/package_regex.inc
>
>     iii) Feed the data from the above code into the package reporting
>system
>
>   The parts are there and have been proven to roughly work. The next
>   step would be to bring them into the core and start using them
>   properly. Its not an easy thing to get right but I think the
>   potential benefits are worth it.
>
>c) Implement code which attempts automated package upgrades and then
>   reports the success/failure. This depends on a) and b) for best
>   results.
>
>d) Continue to move forward with automated testing on real hardware
>   too. The idea is that the above tests in a) should also be usable
>   here so we can test significantly more of the software stack than
>   we've ever been able to before.
>
>e) Continue to add test cases for:
>
>     i) the images (ptest and others)
>     ii) oe-selftest
>     iii) bitbake-selftest
>     iv) add new tests like tests for the toaster UI
>
>The basic idea here is that existing work done manually isn't going to
>scale. We have all the foundations to be able to do something few
>projects have ever done before and automate this on a scale not
>previously seen. This means it would stop people from having to do
>"boring" work and concentrate on the interesting pieces where the
>problems are. It also means we should be able to track down regressions
>much earlier and deal with them more quickly.
>
>How soon will all of this work? Probably not overnight but I believe its
>something to aim for.
>
>
>Contributors to the Project
>---------------------------
>
>This is an open question. How do we attract more developers to work on
>the project? This applies both to layers like OE-Core and outside that,
>we could all use some help in the maintainership, the development and
>the testing. What blocks people from joining us? How can we encourage
>more people to participate?
>
>Unfortunately when things are working for people they tend to think that
>ok, they'll go and do something else (like work on their product!) but
>is there a way we could get some involvement from them? Lots of small
>pieces of help could build to a large whole...
>
>
>Conclusion
>----------
>
>I've probably rambled enough. The project is I believe in a strong
>position but the above are areas I think we need to work on if we want
>to truly fulfil the project's potential.
>
>Cheers,
>
>Richard
>
>
>
>
>
>
>
>
>-- 
>_______________________________________________
>Openembedded-core mailing list
>Openembedded-core@lists.openembedded.org
>http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: My thoughts on the future of OE?

[Mike Looijmans]

On 05/01/2014 07:02 PM, Richard Purdie wrote:
> I was asked what I thought were things that needed discussion at OEDAM.
> Sadly I won't be there but I thought it might help to write down my
> thoughts in a few areas.
>
> Developer Workflow
> ------------------
>
> Firstly, I think the big piece we need to address as a project is
> "developer workflow" as this is where people are struggling using it.
>
> Unfortunately "developer workflow" means different things to different
> people? Which one do I mean then? I actually mean all of them. As some
> examples:

Just about fifteen minutes ago I was asked the question of "Yay, I could 
build a complete image for the board from scratch. Now how do I create 
and run a HelloWorld application on it?"

The answer I was forced to give him was still horribly complicated - 
create a repository, create a recipe, add the recipe, build the image, 
and loop until it works. That's the method I usually follow myself, but 
there are probably a few dozen others.

I expect him back in about an hour asking: "How can I quickly recompile 
and run after changing the code?"

My answer again is going to be disappointing to him again. I'll be 
forced to say he'll have to commit the change and build the image again.

After the weekend he'll ask me how to debug his code on target. The 
answer to that? Well, I have the whole weekend to think about that.


This is basically what scares away new developers. It's easy to compile 
an image, but it there isn't an obvious way to start developing your own 
application.


Don't get me wrong. OE is still by far the best cross-compile 
environment in existence.


Mike.

Re: My thoughts on the future of OE?

[Burton, Ross]

On 2 May 2014 14:00, Mike Looijmans <mike.looijmans@topic.nl> wrote:
> Just about fifteen minutes ago I was asked the question of "Yay, I could
> build a complete image for the board from scratch. Now how do I create and
> run a HelloWorld application on it?"

Depending on context simply adding the "tools-sdk dev-pkgs"
IMAGE_FEATURES is sufficient to build applications on the target, as
long as it's for the hello world or iterative development approaches.
Obviously a recipe would need to be written at some point but getting
a toolchain in the image for development on the target is trivial.

Ross

Re: My thoughts on the future of OE?

[Mike Looijmans]

On 05/02/2014 03:47 PM, Burton, Ross wrote:
> On 2 May 2014 14:00, Mike Looijmans <mike.looijmans@topic.nl> wrote:
>> Just about fifteen minutes ago I was asked the question of "Yay, I could
>> build a complete image for the board from scratch. Now how do I create and
>> run a HelloWorld application on it?"
>
> Depending on context simply adding the "tools-sdk dev-pkgs"
> IMAGE_FEATURES is sufficient to build applications on the target, as
> long as it's for the hello world or iterative development approaches.
> Obviously a recipe would need to be written at some point but getting
> a toolchain in the image for development on the target is trivial.
>
> Ross
>

That would be for "how do I create and run a HelloWorld application on 
target without using my PC at all?".

I guess I'm still a bit stuck in the era where the target had very 
limited resources. For first time users, this is much easier, and for a 
workshop, I've indeed supplied a dozen boards that had editors, 
compilers and library headers pre-installed for immediate development.

That will work for hello-world style apps, but for more complex projects 
that draw in a few dozen complex libraries (gstreamer comes to mind) 
this is not going to be very satisfactory.


What he was asking for was "How do I use the cross-compile environment 
to develop my own application and run it on the target machine?"


-- 
Mike Looijmans

Re: My thoughts on the future of OE?

[Philip Balister]

On 05/02/2014 07:10 AM, Mike Looijmans wrote:
> On 05/02/2014 03:47 PM, Burton, Ross wrote:
>> On 2 May 2014 14:00, Mike Looijmans <mike.looijmans@topic.nl> wrote:
>>> Just about fifteen minutes ago I was asked the question of "Yay, I could
>>> build a complete image for the board from scratch. Now how do I
>>> create and
>>> run a HelloWorld application on it?"
>>
>> Depending on context simply adding the "tools-sdk dev-pkgs"
>> IMAGE_FEATURES is sufficient to build applications on the target, as
>> long as it's for the hello world or iterative development approaches.
>> Obviously a recipe would need to be written at some point but getting
>> a toolchain in the image for development on the target is trivial.
>>
>> Ross
>>
> 
> That would be for "how do I create and run a HelloWorld application on
> target without using my PC at all?".
> 
> I guess I'm still a bit stuck in the era where the target had very
> limited resources. For first time users, this is much easier, and for a
> workshop, I've indeed supplied a dozen boards that had editors,
> compilers and library headers pre-installed for immediate development.
> 
> That will work for hello-world style apps, but for more complex projects
> that draw in a few dozen complex libraries (gstreamer comes to mind)
> this is not going to be very satisfactory.
> 
> 
> What he was asking for was "How do I use the cross-compile environment
> to develop my own application and run it on the target machine?"
> 
> 

https://github.com/balister/meta-sdr/wiki/CrossCompile

IS what I tell people for GNU Radio. The generic method should be OK for
other cases. I need to collect techniques to move output onto the target.

Philip

Re: My thoughts on the future of OE?

[Jack Mitchell]

On 02/05/14 14:47, Burton, Ross wrote:
> On 2 May 2014 14:00, Mike Looijmans <mike.looijmans@topic.nl> wrote:
>> Just about fifteen minutes ago I was asked the question of "Yay, I could
>> build a complete image for the board from scratch. Now how do I create and
>> run a HelloWorld application on it?"
> 
> Depending on context simply adding the "tools-sdk dev-pkgs"
> IMAGE_FEATURES is sufficient to build applications on the target, as
> long as it's for the hello world or iterative development approaches.
> Obviously a recipe would need to be written at some point but getting
> a toolchain in the image for development on the target is trivial.
> 
> Ross
> 

Or generating and installing the SDK, then develop -> cross compile ->
copy to target -> run -> rinse and repeat.

Then when working, update your applications recipe to point to the new
revision and have a "clean" production image pop out.

The pain points with this come when you want to use new libraries etc
and you have to do a rootfs + sdk rebuild and upgrade but in reality
these are few and far between.

Cheers,

-- 
  Jack Mitchell (jack@embed.me.uk)
  Embedded Systems Engineer
  Cambridgeshire, UK
  http://www.embed.me.uk
-- 

Re: My thoughts on the future of OE?

[Koen Kooi]

Op 1 mei 2014, om 19:02 heeft Richard Purdie <richard.purdie@linuxfoundation.org> het volgende geschreven:

> I was asked what I thought were things that needed discussion at OEDAM.
> Sadly I won't be there but I thought it might help to write down my
> thoughts in a few areas.
> 
> Developer Workflow
> ------------------
> 
> Firstly, I think the big piece we need to address as a project is
> "developer workflow" as this is where people are struggling using it.
> 
> Unfortunately "developer workflow" means different things to different
> people? Which one do I mean then? I actually mean all of them. As some
> examples:

[snip use cases]

I notice none of the use case mention distro developers, but the paragraph above is vague enough to be interpreted as "we rock at distro stuff, we suck at app/product development". To clear up my confusion, could you say something about distro development? Something like "bitbake-prserv-tool import won't be glacially slow in the future" :)

After 5 OE-core based angstrom releases I can say that the 'Core OS' quality has improved a lot and I've made peace with the version update policy, but the overall experience for a downstream distro is painful. A lot of small, stupid and preventable mistakes combined are tedious to deal with. The most recent example I hit was xinput-calibrator losing XDG autolaunch support in the move to OE-core without mentioning it in the commit log. It took me a week (admittingly a spare time week, so maybe a single workday) to track it down and come up with a patch.

And there's a *big* stupid and preventable issue that is not really OE-cores fault, but more the yocto-projects fault: BSPs using linux-yocto.bbappend and doing:

	SRCREV = "githash"
	LINUX_VERSION = "3.12"

BSP maintainers assume only their BSP will be affected, but it affects *all* BSPs using linux-yocto.bbappends. This applied to all bbappends, but I've seen all meta-handheld machines jump up and down in PV due to other BSPs setting global vars. Ross deserves a big thank you for obsoleting mesa bbappends, that issues prevented angstrom from having a working GL version for >1 ARM silicon vendor.


> So my first ask is that we actually try and write down all these
> different cases which is no small task in itself. I've a start of a list
> above, we should probably put this into the wiki and have people add
> their own use cases (or use cases of those around them in their company
> etc.). The trouble is there are some many different variants!
> 
> Once we have some idea of the cases, we can start to put together some
> kind of plan about how we intend to help the given use cases and to try
> and prioritise them. Perhaps we should put some kind of weighting
> against them in the wiki and people can increase the numbers of say
> their top three desires.
> 
> Whilst this looks like an impossible problem, the good news is that I
> believe we can solve it and we actually already have a rather powerful
> toolbox to help work on it. I have a rough idea of a roadmap that can
> move us forward:
> 
> a) locked sstate - I know its not in master yet but I believe this is
> key to allowing some of the use cases where you want to change some
> number of things but keep the rest of the system the same.

This is something I would very much like to see happening. I've noticed at least 3 patches in the 'dora' stable branch that triggered pretty much a complete rebuild due to sstate+prserv. And when I apply a buildfix for recent host distros to e.g. gcc to meta-linaro-toolchain I trigger a complete rebuild as well. It would be a lot less churn in the package feeds if I could lock down everything in console-image. That way the 'core OS' packages won't have as much spurious updates as they have now.

regards,

Koen

Re: My thoughts on the future of OE?

[Richard Purdie]

On Fri, 2014-05-02 at 21:16 +0200, Koen Kooi wrote:
> Op 1 mei 2014, om 19:02 heeft Richard Purdie <richard.purdie@linuxfoundation.org> het volgende geschreven:
> 
> > I was asked what I thought were things that needed discussion at OEDAM.
> > Sadly I won't be there but I thought it might help to write down my
> > thoughts in a few areas.
> > 
> > Developer Workflow
> > ------------------
> > 
> > Firstly, I think the big piece we need to address as a project is
> > "developer workflow" as this is where people are struggling using it.
> > 
> > Unfortunately "developer workflow" means different things to different
> > people? Which one do I mean then? I actually mean all of them. As some
> > examples:
> 
> [snip use cases]
>
> I notice none of the use case mention distro developers, but the
> paragraph above is vague enough to be interpreted as "we rock at
> distro stuff, we suck at app/product development". To clear up my
> confusion, could you say something about distro development? Something
> like "bitbake-prserv-tool import won't be glacially slow in the
> future" :)

Is there an open bug for that?

I didn't really think too much about distro developers since I believe
they are comparatively well covered and they know where/how to shout if
they're having problems.

Some of the other use cases are massively under represented amongst the
regular contributors to the project which is more why I was focusing on
them. That doesn't mean the distro development cases aren't important, I
just think they can look after themselves to an extend :).

> After 5 OE-core based angstrom releases I can say that the 'Core OS'
> quality has improved a lot and I've made peace with the version update
> policy, but the overall experience for a downstream distro is painful.
> A lot of small, stupid and preventable mistakes combined are tedious
> to deal with. The most recent example I hit was xinput-calibrator
> losing XDG autolaunch support in the move to OE-core without
> mentioning it in the commit log. It took me a week (admittingly a
> spare time week, so maybe a single workday) to track it down and come
> up with a patch.

I hope that kind of thing doesn't happen often. When/where it does,
we'll figure it out and warn the people who let the issue through so it
doesn't happen again. Are there other examples of the kind of
preventable issues we could look for? I do my best to screen the
incoming commits and we have seen more people helping with review
recently but its a tough job. If I know what to look for...

> And there's a *big* stupid and preventable issue that is not really
> OE-cores fault, but more the yocto-projects fault: BSPs using
> linux-yocto.bbappend and doing:
> 
> 	SRCREV = "githash"
> 	LINUX_VERSION = "3.12"
> 
> BSP maintainers assume only their BSP will be affected, but it affects
> *all* BSPs using linux-yocto.bbappends. This applied to all bbappends,
> but I've seen all meta-handheld machines jump up and down in PV due to
> other BSPs setting global vars.

Hmm, not good. Perhaps we should try and detect this?

We could do:

LINUXYOCTO_VERSION = "x"
LINUX_VERSION = "${LINUXYOCTO_VERSION}"

and then check in anonymous python if d.getVar("LINUX_VERSION", False)
is still what we think it should be?

I'm open to ways of improving this if its a major problem.

>  Ross deserves a big thank you for obsoleting mesa bbappends, that
> issues prevented angstrom from having a working GL version for >1 ARM
> silicon vendor.

Yes, that is nice to see. In general I'd like to significantly reduce
the number of bbappends we need. I'd also like to introduce sstate
checksum checks on layers and maybe tie that to Yocto Project
Compatible. If you add a layer and it changes core checksums, and its
not a policy/distro layer, its not compatible. Nice and easy to check
programmatically.

> 
> > So my first ask is that we actually try and write down all these
> > different cases which is no small task in itself. I've a start of a list
> > above, we should probably put this into the wiki and have people add
> > their own use cases (or use cases of those around them in their company
> > etc.). The trouble is there are some many different variants!
> > 
> > Once we have some idea of the cases, we can start to put together some
> > kind of plan about how we intend to help the given use cases and to try
> > and prioritise them. Perhaps we should put some kind of weighting
> > against them in the wiki and people can increase the numbers of say
> > their top three desires.
> > 
> > Whilst this looks like an impossible problem, the good news is that I
> > believe we can solve it and we actually already have a rather powerful
> > toolbox to help work on it. I have a rough idea of a roadmap that can
> > move us forward:
> > 
> > a) locked sstate - I know its not in master yet but I believe this is
> > key to allowing some of the use cases where you want to change some
> > number of things but keep the rest of the system the same.
> 
> This is something I would very much like to see happening. I've
> noticed at least 3 patches in the 'dora' stable branch that triggered
> pretty much a complete rebuild due to sstate+prserv. And when I apply
> a buildfix for recent host distros to e.g. gcc to
> meta-linaro-toolchain I trigger a complete rebuild as well. It would
> be a lot less churn in the package feeds if I could lock down
> everything in console-image. That way the 'core OS' packages won't
> have as much spurious updates as they have now.

Agreed. I think this will take a little bit of bedding in as we find the
right use cases and the right configuration but its something I want to
spend time on in 1.7 and get working well...

Cheers,

Richard

Re: My thoughts on the future of OE?

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2348 bytes --]

On Thu, May 01, 2014 at 06:02:41PM +0100, Richard Purdie wrote:
> I was asked what I thought were things that needed discussion at OEDAM.
> Sadly I won't be there but I thought it might help to write down my
> thoughts in a few areas.
> 
> Developer Workflow
> ------------------
> 
> Firstly, I think the big piece we need to address as a project is
> "developer workflow" as this is where people are struggling using it.
> 
> Unfortunately "developer workflow" means different things to different
> people? Which one do I mean then? I actually mean all of them. As some
> examples:
> 
> -----------------------------------------------------------------------
> 
> * A kernel developer wanting to rebuild a kernel 
>   [on/off target, with the ADT/SDK or a recipe]
> * A kernel developer wanting to build a kernel module
>   [on/off target, with the ADT/SDK or a recipe]
> * An application developer wanting to build a single App
>   [on/off target, with the ADT/SDK or a recipe]
> * An application developer wanting to (re)build a library, linking an 
>   App to it
>   [on/off target, with the ADT/SDK or a recipe]
> * A user wanting to rebuild an image with a package added
>   [on and off target - feeds or a build]
> * A user wanting to rebuild an image with more advanced changes

One more workflow I see quite often and really isn't supported well is:

* An application developer wanting to (re)build just one library and one
  test application, verify on target, iterate as long sa needed and then
  do "clean" build which will rebuild everything depending on it (not
  just the test app)

Real world example is newer eglibc breaking audio from qtmultimedia, so
the test app is depending on all qt* crap which takes forever to build
and every developer change to eglibc will basically trigger rebuild of
everything.

We need an easy way to do such "simplified" builds where you don't care
much about reproducibility. Currently most developers I know will do
devshell/source run.do_foo  for eglibc and the test-app + rsync/scp of these 2 changed
binaries to your target device to verify if the fix works.

The bitbake -S improvements help to find "why" the build rebuilt so many
components, but doesn't help with preventing it (see 
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: My thoughts on the future of OE?

[David Nyström]

On 2014-05-01 19:02, Richard Purdie wrote:
> I was asked what I thought were things that needed discussion at OEDAM.
> Sadly I won't be there but I thought it might help to write down my
> thoughts in a few areas.
>
> Developer Workflow
> ------------------
>
> Firstly, I think the big piece we need to address as a project is
> "developer workflow" as this is where people are struggling using it.

Good initiative.

> Unfortunately "developer workflow" means different things to different
> people? Which one do I mean then? I actually mean all of them. As some
> examples:

Whilst OE supplies a superior interface towards distribution engineers, 
and one-man-army developers who usually work top-down on the entire 
software stack, its currently supplies a suboptimal user experience for 
bigger teams, where each team tend to care only about their app, and its 
dependencies.

IMHO, the biggest issue is maintainable interfaces between different 
participants within the larger projects.
The package repository should be able to facilitate a good user facing 
interface for Application developers and kernel developers.

I fear that an item in a locked sstate might not be a small enough 
cross-section for reporting bugs against. How do I trace a buggy package 
installed on my target rootfs, back to an item in the locked sstate?

> -----------------------------------------------------------------------
>
> * A kernel developer wanting to rebuild a kernel
>    [on/off target, with the ADT/SDK or a recipe]
> * A kernel developer wanting to build a kernel module
>    [on/off target, with the ADT/SDK or a recipe]
> * An application developer wanting to build a single App
>    [on/off target, with the ADT/SDK or a recipe]
> * An application developer wanting to (re)build a library, linking an
>    App to it
>    [on/off target, with the ADT/SDK or a recipe]
> * A user wanting to rebuild an image with a package added
>    [on and off target - feeds or a build]
> * A user wanting to rebuild an image with more advanced changes

An example of a low-overhead scenario for a bigger project:

** Distribution Engineers
* Works in the bitbake/OE environment and against the upstream Yocto 
community.
* Makes sure the distribution package repository is kept up to date..
* Syncs releses, upgrades and bugfixes with the upstream community.
* Recieves patches and configuration from Developers and integrate them 
into the distribution, either by SRCREV:s or patches.

** Kernel Developers
* Uses on SDK/ADT tarball provided by the SI.
* Upstreams patches to Upstream.
* Backports patches, sends new SCM revisions, and kernel configuration 
shards to the SI.
* Creates the rootfs image by method X[1] from the package repository.
* Expands the rootfs image by method X[1] from the package repository.
* Bugreports on other ADs or KDs packages using the package versions.

** Application Developers
* Uses on SDK/ADT tarball provided by the SI.
* Determines application dependencies/configuration and notifies SI of 
requirements.
* Maintains the applications recipe.
* Creates the rootfs image by method X[1] from the package repository.
* Expands the rootfs image by method X[1] from the package repository.
* Upstreams recipes, patches or SCM revisions to the SI for his 
application/s for distribution.
* Bugreports on other ADs or KDs packages using the package versions.


Ref:[1]
The current options are:

1. Static core-image-best-guess with package-management delivered by DE.
--
Works OK with static storage. This is a pain with volatile storage 
though, or read-only rootfs:s.
And it is bloaty with bigger repos, and limited storage size.
Different teams images will differ over time.

2. BUILD_IMAGES_FROM_FEEDS
--
Works only for ipk. Still requires bitbake/OE env for rather lightweight 
use-case.

Possible Future for [1]:
Step 1:
Let AD/KD create rootfs images from the package repository with a 
lightweight tool in the SDK tarball. Dependencies needed by postinstall 
hooks needs to be available in the nativesdk sysroot.
This would also allow customization of the SDK targets sysroot, and SDK 
nativesdk sysroots on a per-package level.
i.e. If I'm cross compiling and missing libcurl from the target sysroot, 
I should be able to expand it easily with the distributions version of 
libcurl.

Re: My thoughts on the future of OE?

[Barros Pena, Belen]

On 05/05/2014 04:39, "David Nyström" <david.nystrom@enea.com> wrote:

>How do I trace a buggy package
>installed on my target rootfs, back to an item in the locked sstate?

FWIW, you should be able to do that with Toaster, as long as you know
which one is the buggy package :) With Toaster you can navigate the build
process, from the package, to the recipe, to the tasks executed by the
build system for that recipe. As part of the tasks information, Toaster
will show you which tasks reused which sstate objects. So effectively you
can trace any package to the corresponding sstate object(s).

Cheers

Belén