Impressions of using Poky to create a BSP

Impressions of using Poky to create a BSP

[Holger Freyther]

Hi all,

I hope these impressions are useful. In the last two days I added a BSP for a
TI Davinci based system and I am afraid it was a bit harder than I think it
should be. This is a summary of the underlying issues (without them being
debugged or completely understood).

meta-skeleton and BBFILES:
   - Maybe it would be nice if the file glob would list *.bbappend as I think
     many people want to use it.

BBLAYERs and dependencies:
    - I had changed BBFILES to list *.bbappend but the change didn't take
      effect. I had to touch conf/local.conf to force a re-parse. So somehow
      the layers config didn't end up in dependencies.

fetch vs. fetch2:
    - I wanted to fetch my kernel from my internal git/ssh repository. I was
      not sure where to put the username (git://foo@bar or git://bar;user=foo)
      so I was experimenting a bit. This triggered me being banned from my
      server (fail2ban) so the git process on the buildhost got stuck and I
      backgrounded bitbake with CTRL+Z.

    - Then I tried bitbake -cfetch virtual/kernel again and it didn't do anything,
      it took me a while to think that it might wait on a lock. I think it would be
      great if this could be printed somewhere (if my understanding of this is
      correct).

    - Also it was not obvious to find out which of the two fetchers are used..


Adding a kernel:
     - So by default the linuxdummy is used, but I wanted to build my kernel,
       I started off by copying the yocto kernel recipes. This creates a very funny
       bug. My kernel didn't have the meta branch, the task to test the branches
       executes git log for the branch and this is failing.
       The symptoms are not obvious at all. An empty log file is left behind.

     - I was using the yocto kernel recipes incorrectly and ended up with a logfile
       that pointed to itself for details but nothing else.

     - In the end I used a minimal kernel recipe, it would be nice to either have
       an easy way to 'produce' a yocto like kernel of a simple kernel recipe in
       the skeletons.

Python Exceptions:
     - Building something that is not provided gives a python exception
     - Adding a task that does not exist gives a python exception
     - I think it would be nice to have meaningful error messages. Specially
       the laters runs into something like:
      if not Data:
           print "BLA BLA"
      use(data)

Machine Extra Depends:
     - I was using the core-minimal-image but wanted to have some of my
       kernel modules installed. After finding the name that task-core-boot
       is using I tried to do bitbake -cclean task-core-boot; bitbake the-image
       but this didn't end up with a task-core-boot to carry my depends.
       I had to resort to using bitbake -b task-core-boot.bb to build  it, I am not
       sure why it failed when running from within the normal bitbake. IIRC it
       continued to be like this even after I removed tmp/cache.

I hope this point of view is helpful.

Re: Impressions of using Poky to create a BSP

[Bruce Ashfield]

On Fri, May 27, 2011 at 1:42 PM, Holger Freyther <holger@freyther.de> wrote:
> Hi all,
>
> I hope these impressions are useful. In the last two days I added a BSP for a
> TI Davinci based system and I am afraid it was a bit harder than I think it
> should be. This is a summary of the underlying issues (without them being
> debugged or completely understood).
>
> meta-skeleton and BBFILES:
>   - Maybe it would be nice if the file glob would list *.bbappend as I think
>     many people want to use it.
>
> BBLAYERs and dependencies:
>    - I had changed BBFILES to list *.bbappend but the change didn't take
>      effect. I had to touch conf/local.conf to force a re-parse. So somehow
>      the layers config didn't end up in dependencies.
>
> fetch vs. fetch2:
>    - I wanted to fetch my kernel from my internal git/ssh repository. I was
>      not sure where to put the username (git://foo@bar or git://bar;user=foo)
>      so I was experimenting a bit. This triggered me being banned from my
>      server (fail2ban) so the git process on the buildhost got stuck and I
>      backgrounded bitbake with CTRL+Z.
>
>    - Then I tried bitbake -cfetch virtual/kernel again and it didn't do anything,
>      it took me a while to think that it might wait on a lock. I think it would be
>      great if this could be printed somewhere (if my understanding of this is
>      correct).
>
>    - Also it was not obvious to find out which of the two fetchers are used..
>
>
> Adding a kernel:
>     - So by default the linuxdummy is used, but I wanted to build my kernel,
>       I started off by copying the yocto kernel recipes. This creates a very funny
>       bug. My kernel didn't have the meta branch, the task to test the branches
>       executes git log for the branch and this is failing.
>       The symptoms are not obvious at all. An empty log file is left behind.

Was this out of master ? There's been some commits and sanity checks
added just recently that emit clear messages if you don't have things like
the meta branch, or the right configuration. Auto creation of meta data is
also in the tree now.

This is also a failing of the docs if this wasn't clear, we can look into that
as well.

>
>     - I was using the yocto kernel recipes incorrectly and ended up with a logfile
>       that pointed to itself for details but nothing else.
>
>     - In the end I used a minimal kernel recipe, it would be nice to either have
>       an easy way to 'produce' a yocto like kernel of a simple kernel recipe in
>       the skeletons.

There's a 1.1 feature for this, and it will actually be sent out
shortly. There's
the ability to auto create meta data, using any upstream source, and the
ability to build from arbitrary upstream repos.

Both of these are supposed to be transitional aids though, since if you are
basing something on the linux-yocto kernel(s), it's best to actually use the
uptream kernel (branches and all) and create BSP branches on it.

>
> Python Exceptions:
>     - Building something that is not provided gives a python exception
>     - Adding a task that does not exist gives a python exception
>     - I think it would be nice to have meaningful error messages. Specially
>       the laters runs into something like:
>      if not Data:
>           print "BLA BLA"
>      use(data)
>
> Machine Extra Depends:
>     - I was using the core-minimal-image but wanted to have some of my
>       kernel modules installed. After finding the name that task-core-boot
>       is using I tried to do bitbake -cclean task-core-boot; bitbake the-image
>       but this didn't end up with a task-core-boot to carry my depends.
>       I had to resort to using bitbake -b task-core-boot.bb to build  it, I am not
>       sure why it failed when running from within the normal bitbake. IIRC it
>       continued to be like this even after I removed tmp/cache.
>
> I hope this point of view is helpful.

It' definitely helps me, I think a lot has already been addressed, and this
should be a seamless process from the kernel build point of view.

cheers,

Bruce

>
>
>
>
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky
>



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: Impressions of using Poky to create a BSP

[Darren Hart]

On 05/27/2011 10:42 AM, Holger Freyther wrote:
> Hi all,
> 
> I hope these impressions are useful. In the last two days I added a BSP for a
> TI Davinci based system and I am afraid it was a bit harder than I think it
> should be. This is a summary of the underlying issues (without them being
> debugged or completely understood).
> 
> meta-skeleton and BBFILES:
>    - Maybe it would be nice if the file glob would list *.bbappend as I think
>      many people want to use it.
> 
> BBLAYERs and dependencies:
>     - I had changed BBFILES to list *.bbappend but the change didn't take
>       effect. I had to touch conf/local.conf to force a re-parse. So somehow
>       the layers config didn't end up in dependencies.
> 
> fetch vs. fetch2:
>     - I wanted to fetch my kernel from my internal git/ssh repository. I was
>       not sure where to put the username (git://foo@bar or git://bar;user=foo)

I believe the correct options would have been:

ssh://foo@bar/path/to/repo
or
foo@bar:/path/to/repo

with protocol=git

(I haven't tested this)

>       so I was experimenting a bit. This triggered me being banned from my
>       server (fail2ban)

Ouch :(

> so the git process on the buildhost got stuck and I
>       backgrounded bitbake with CTRL+Z.
> 
>     - Then I tried bitbake -cfetch virtual/kernel again and it didn't do anything,
>       it took me a while to think that it might wait on a lock. I think it would be
>       great if this could be printed somewhere (if my understanding of this is
>       correct).
> 
>     - Also it was not obvious to find out which of the two fetchers are used..
> 
> 
> Adding a kernel:
>      - So by default the linuxdummy is used, but I wanted to build my kernel,
>        I started off by copying the yocto kernel recipes. This creates a very funny
>        bug. My kernel didn't have the meta branch, the task to test the branches
>        executes git log for the branch and this is failing.
>        The symptoms are not obvious at all. An empty log file is left behind.


This is probably best dealt with a kernel recipe writing manual. We have
a few things like this forming, but nothing tangible yet.


> 
>      - I was using the yocto kernel recipes incorrectly and ended up with a logfile
>        that pointed to itself for details but nothing else.


Better checkpoint logging in the kernel recipes and classes is
definitely needed. I have a branch with the start of that. Something
I'll continue to look into.


> 
>      - In the end I used a minimal kernel recipe, it would be nice to either have
>        an easy way to 'produce' a yocto like kernel of a simple kernel recipe in
>        the skeletons.


Checkout the meta-kernel-dev repository. We have (or will be very soon)
added a linux-korg_dev.bb which does exactly that. Bruce on CC is
driving that.


> 
> Python Exceptions:
>      - Building something that is not provided gives a python exception
>      - Adding a task that does not exist gives a python exception
>      - I think it would be nice to have meaningful error messages. Specially
>        the laters runs into something like:
>       if not Data:
>            print "BLA BLA"
>       use(data)

Not my area, but yes, there is room for improvement here.

> 
> Machine Extra Depends:
>      - I was using the core-minimal-image but wanted to have some of my
>        kernel modules installed. After finding the name that task-core-boot
>        is using I tried to do bitbake -cclean task-core-boot; bitbake the-image
>        but this didn't end up with a task-core-boot to carry my depends.
>        I had to resort to using bitbake -b task-core-boot.bb to build  it, I am not
>        sure why it failed when running from within the normal bitbake. IIRC it
>        continued to be like this even after I removed tmp/cache.


This is indeed a point where intuition has failed me as well. I remember
there was another machine specific task that I had to rebuild to trigger
getting the modules into the image. Unfortunatley I can't recall what it
was atm and "bitbake -g -u depexp core-image-minimal" isn't giving me
any hints either :/


> 
> I hope this point of view is helpful.
> 

Yes, thanks for sharing!

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: Impressions of using Poky to create a BSP

[Chris Larson]

On Fri, May 27, 2011 at 12:48 PM, Darren Hart <dvhart@linux.intel.com> wrote:
> On 05/27/2011 10:42 AM, Holger Freyther wrote:
>> Hi all,
>>
>> I hope these impressions are useful. In the last two days I added a BSP for a
>> TI Davinci based system and I am afraid it was a bit harder than I think it
>> should be. This is a summary of the underlying issues (without them being
>> debugged or completely understood).
>>
>> meta-skeleton and BBFILES:
>>    - Maybe it would be nice if the file glob would list *.bbappend as I think
>>      many people want to use it.
>>
>> BBLAYERs and dependencies:
>>     - I had changed BBFILES to list *.bbappend but the change didn't take
>>       effect. I had to touch conf/local.conf to force a re-parse. So somehow
>>       the layers config didn't end up in dependencies.
>>
>> fetch vs. fetch2:
>>     - I wanted to fetch my kernel from my internal git/ssh repository. I was
>>       not sure where to put the username (git://foo@bar or git://bar;user=foo)
>
> I believe the correct options would have been:
>
> ssh://foo@bar/path/to/repo
> or
> foo@bar:/path/to/repo
>
> with protocol=git
>

This is backwards. The url scheme determines the fetcher to use, the
protocol= determines the method that fetcher uses to obtain the
sources. Holger had it correct. git:// to associate with the git
fetcher, protocol= to ensure it uses ssh.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: Impressions of using Poky to create a BSP

[Darren Hart]

On 05/27/2011 01:06 PM, Chris Larson wrote:
> On Fri, May 27, 2011 at 12:48 PM, Darren Hart <dvhart@linux.intel.com> wrote:
>> On 05/27/2011 10:42 AM, Holger Freyther wrote:
>>> Hi all,
>>>
>>> I hope these impressions are useful. In the last two days I added a BSP for a
>>> TI Davinci based system and I am afraid it was a bit harder than I think it
>>> should be. This is a summary of the underlying issues (without them being
>>> debugged or completely understood).
>>>
>>> meta-skeleton and BBFILES:
>>>    - Maybe it would be nice if the file glob would list *.bbappend as I think
>>>      many people want to use it.
>>>
>>> BBLAYERs and dependencies:
>>>     - I had changed BBFILES to list *.bbappend but the change didn't take
>>>       effect. I had to touch conf/local.conf to force a re-parse. So somehow
>>>       the layers config didn't end up in dependencies.
>>>
>>> fetch vs. fetch2:
>>>     - I wanted to fetch my kernel from my internal git/ssh repository. I was
>>>       not sure where to put the username (git://foo@bar or git://bar;user=foo)
>>
>> I believe the correct options would have been:
>>
>> ssh://foo@bar/path/to/repo
>> or
>> foo@bar:/path/to/repo
>>
>> with protocol=git
>>
> 
> This is backwards. The url scheme determines the fetcher to use, the
> protocol= determines the method that fetcher uses to obtain the
> sources. Holger had it correct. git:// to associate with the git
> fetcher, protocol= to ensure it uses ssh.

Doh, thanks for correcting. That explains some things for me. I had
thought a real git url was appropriate here, good to know!

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: Impressions of using Poky to create a BSP

[Chris Larson]

On Fri, May 27, 2011 at 1:19 PM, Darren Hart <dvhart@linux.intel.com> wrote:
>> This is backwards. The url scheme determines the fetcher to use, the
>> protocol= determines the method that fetcher uses to obtain the
>> sources. Holger had it correct. git:// to associate with the git
>> fetcher, protocol= to ensure it uses ssh.
>
> Doh, thanks for correcting. That explains some things for me. I had
> thought a real git url was appropriate here, good to know!

It's not the first time I've heard it the other way around. Could be
the other way is more intuitive, don't really know.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: Impressions of using Poky to create a BSP

[Robert Berger]

Hi,

On 05/27/2011 10:48 PM, Darren Hart wrote:
>>
>> Adding a kernel:
>>      - So by default the linuxdummy is used, but I wanted to build my kernel,
>>        I started off by copying the yocto kernel recipes. This creates a very funny
>>        bug. My kernel didn't have the meta branch, the task to test the branches
>>        executes git log for the branch and this is failing.
>>        The symptoms are not obvious at all. An empty log file is left behind.
> 
> 
> This is probably best dealt with a kernel recipe writing manual. We have
> a few things like this forming, but nothing tangible yet.
> 
> 

I'm currently building a manufacturer supplied 3rd party kernel and
u-boot outside of yocto/poky to avoid all this trouble.

It's the classic case of a non mainline 3rd party kernel + patch + config

If it's easy enough to create a BSP this would be my preferred solution
and it would probably also be a good use case for such a manual.

If you have some documentation (even preliminary) I'm willing to work
with the people involved to come up with something more substantial. (me
being the one testing how well the instructions work for my 3rd party
kernel)

The goal would be to end up with something not much more difficult than
building the stuff outside of yocto/poky;)

Regards,

Robert

..."Experience is that marvelous thing that enables you recognize a
mistake when you make it again." - F. P. Jones

My public pgp key is available at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1

Re: Impressions of using Poky to create a BSP

[Bruce Ashfield]

On Sat, May 28, 2011 at 2:35 PM, Robert Berger
<gmane@reliableembeddedsystems.com> wrote:
> Hi,
>
> On 05/27/2011 10:48 PM, Darren Hart wrote:
>>>
>>> Adding a kernel:
>>>      - So by default the linuxdummy is used, but I wanted to build my kernel,
>>>        I started off by copying the yocto kernel recipes. This creates a very funny
>>>        bug. My kernel didn't have the meta branch, the task to test the branches
>>>        executes git log for the branch and this is failing.
>>>        The symptoms are not obvious at all. An empty log file is left behind.
>>
>>
>> This is probably best dealt with a kernel recipe writing manual. We have
>> a few things like this forming, but nothing tangible yet.
>>
>>
>
> I'm currently building a manufacturer supplied 3rd party kernel and
> u-boot outside of yocto/poky to avoid all this trouble.

There really shouldn't be any trouble now, in particular out of master.
In the past three weeks, I've helped a few people work through just
this sort of scenario.

>
> It's the classic case of a non mainline 3rd party kernel + patch + config
>
> If it's easy enough to create a BSP this would be my preferred solution
> and it would probably also be a good use case for such a manual.
>
> If you have some documentation (even preliminary) I'm willing to work
> with the people involved to come up with something more substantial. (me
> being the one testing how well the instructions work for my 3rd party
> kernel)

The released docs with yocto 1.0 still apply here. We need to start
there. If they are insufficient, or are missing information, then we'll
get it fixed.

>
> The goal would be to end up with something not much more difficult than
> building the stuff outside of yocto/poky;)

There's a few levels to this comment. The point is that if you use the
linux-yocto recipes, then hopefully someone want to use the tooling
and maintain some extra meta data with the kernel. master already
has changes to auto-create the minimum set of meta data for building
any git tree you throw at it.

But that's just the first step. That bootstrapped data can be used
as the starting point for maintaining and extending things.

Even better than having random kernel versions and potentially
unmaintained patches, is to try and sync up with a kernel version
that yocto (or others) are using as collecting points for embedded
features of interest. From there, we can leverage the work, re-use
it and ultimately get changes into the mainline kernel where they
belong.

So if you've got patches and changes that are of interest, I'd like
to see them from that point of view, just as much as making sure
that the yocto kernel recipes + BSP model work seamlessly.

Cheers,

Bruce



>
> Regards,
>
> Robert
>
> ..."Experience is that marvelous thing that enables you recognize a
> mistake when you make it again." - F. P. Jones
>
> My public pgp key is available at:
> http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1
>
>
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky
>



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: Impressions of using Poky to create a BSP

[Khem Raj]

On Fri, May 27, 2011 at 10:42 AM, Holger Freyther <holger@freyther.de> wrote:
>
> Machine Extra Depends:
>     - I was using the core-minimal-image but wanted to have some of my
>       kernel modules installed. After finding the name that task-core-boot
>       is using I tried to do bitbake -cclean task-core-boot; bitbake the-image
>       but this didn't end up with a task-core-boot to carry my depends.
>       I had to resort to using bitbake -b task-core-boot.bb to build  it, I am not
>       sure why it failed when running from within the normal bitbake. IIRC it
>       continued to be like this even after I removed tmp/cache.
>

generally -ccleanall helped me in such cases.