Documenting YP Development Environment in more detail

Documenting YP Development Environment in more detail

[Rifenbark, Scott M]

Hi, 

Recently, Paul, Ross, Richard and I had a video conference meeting where we had some initial discussion on how to satisfy YOCTO #2808 (https://bugzilla.yoctoproject.org/show_bug.cgi?id=2808), which calls for an illustration showing source and destination directories during a build using YP.  This bug originated from a discussion Dave Stewart and I had a while back around an idea of a more detailed "flow diagram" that would go into greater detail than the now famous and ubiquitous "The Yocto Project Development Environment" illustration, which appears in the YP Quick Start (http://www.yoctoproject.org/docs/1.4/yocto-project-qs/yocto-project-qs.html) and has been used in many other places and in many other forms (some quite elaborate).

We can't really create a completely accurate, all-encompassing illustration that breaks apart the entire build process.  It is not a static thing and it is quite complicated inside the BitBake process.  We can, however, show where metadata comes from, how it is provided, what it defines, where and how source files are found, what processes occur, what output is produced, and where it ends up.  We can also provide some sort of idea of how key BitBake and environment variables affect things along the way.  The idea here is to dig deeper into this conceptual figure and root out the details to a level that would be useful to the user but not impossible to maintain or develop. This type of information can be communicated through a mix of several illustrations with supporting text.

This first meeting started with some detailed discussion of the configuration inputs for a typical YP build but soon migrated to discussing the bigger picture and possible ways to provide more information.  It became obvious we were not going to dig the solution and all the needed information out in one short meeting. Consequently, I am sending out this email to help open up some discussion on the issue and to also solicit information for some basic blocks of information.

Two things are needed at this point:  1) a presentation solution for this new and more detailed information, and 2) a starting point on some of the information itself.  

PRESENTATION SOLUTION

Here are some thoughts on how to present this information.  There are disadvantages and advantages to each of these methods of which I will not list.  I would like to see what people think about them:

 * Manual - Create a section in the "Technical Details" Chapter of the YP Reference Manual that holds this information.  The section would be pretty much self-contained and would consist of several illustrations that would stem from an overall figure that is similar to the figure we now have in the YP Quick Start.  However, we would use a drill-down strategy that would progressively reveal more detail through subsequent figures.  This method is similar to how hardware devices used to be documented where functional blocks would be connected and described in one area and then subsequent areas would elaborate more deeply on a particular block.  Linking within the manual could connect up the various functional blocks (inputs, outputs, and tasks) that comprise the overall flow.

* Manual / Website Mix - Devise a mix between the YP Reference Manual and some pages in the YP Website that provide the information.  Create a section in the "Technical details" Chapter of the YP Reference Manual that covers this information at a high level.  For example, the overall flow of the system with its "big-block" inputs and outputs and tasks could be discussed at some length.  Links in the text could go to the YP website where more detail would be revealed.  This strategy effectively splits the content into "overview" and "details" between the manual and website, respectively.  

* Website - With this strategy, everything is pretty much in the YP website.  This area would exist in a stand-alone fashion.  However, you could link to the website from within the existing YP documentation set from existing areas that deal with the build process.  Several exit points from within the manual set already exist.  We would obviously add a primary one as well that would likely originate from the YP Reference Manual's "Technical Details" Chapter.

SOLICITATION FOR INFORMATION

We can get started now on this by starting to define details for some obvious points or large areas of the flow.  What would be nice to get would be some graphical breakdowns of these areas of concern:

* User-defined layers
* YP provided layers
* User configuration
* Machine Configuration
* Policy Configuration
* Patches
* YP provided recipes
* User provided source code
* YP provided source code
* SCMs
* Generated images
* Generated SDKs/toolchains
* Package Output
* Source fetching process
* Patch application process
* Configuration process (fragments, etc.)
* Key variable use and effects
* User-initiated commands along the way

Much of this list is directly from our existing "The Yocto Project Development Environment" illustration.  

Thanks, 
Scott R.



Scott Rifenbark
Intel Corporation
Yocto Project Documentation
503.712.2702
503.341.0418 (cell)

Re: Documenting YP Development Environment in more detail

[Bruce Ashfield]

On 13-06-20 04:40 AM, Rifenbark, Scott M wrote:
> Hi,
>
> Recently, Paul, Ross, Richard and I had a video conference meeting where we had some initial discussion on how to satisfy YOCTO #2808 (https://bugzilla.yoctoproject.org/show_bug.cgi?id=2808), which calls for an illustration showing source and destination directories during a build using YP.  This bug originated from a discussion Dave Stewart and I had a while back around an idea of a more detailed "flow diagram" that would go into greater detail than the now famous and ubiquitous "The Yocto Project Development Environment" illustration, which appears in the YP Quick Start (http://www.yoctoproject.org/docs/1.4/yocto-project-qs/yocto-project-qs.html) and has been used in many other places and in many other forms (some quite elaborate).
>
> We can't really create a completely accurate, all-encompassing illustration that breaks apart the entire build process.  It is not a static thing and it is quite complicated inside the BitBake process.  We can, however, show where metadata comes from, how it is provided, what it defines, where and how source files are found, what processes occur, what output is produced, and where it ends up.  We can also provide some sort of idea of how key BitBake and environment variables affect things along the way.  The idea here is to dig deeper into this conceptual figure and root out the details to a level that would be useful to the user but not impossible to maintain or develop. This type of information can be communicated through a mix of several illustrations with supporting text.
>
> This first meeting started with some detailed discussion of the configuration inputs for a typical YP build but soon migrated to discussing the bigger picture and possible ways to provide more information.  It became obvious we were not going to dig the solution and all the needed information out in one short meeting. Consequently, I am sending out this email to help open up some discussion on the issue and to also solicit information for some basic blocks of information.
>
> Two things are needed at this point:  1) a presentation solution for this new and more detailed information, and 2) a starting point on some of the information itself.
>
> PRESENTATION SOLUTION
>
> Here are some thoughts on how to present this information.  There are disadvantages and advantages to each of these methods of which I will not list.  I would like to see what people think about them:
>
>   * Manual - Create a section in the "Technical Details" Chapter of the YP Reference Manual that holds this information.  The section would be pretty much self-contained and would consist of several illustrations that would stem from an overall figure that is similar to the figure we now have in the YP Quick Start.  However, we would use a drill-down strategy that would progressively reveal more detail through subsequent figures.  This method is similar to how hardware devices used to be documented where functional blocks would be connected and described in one area and then subsequent areas would elaborate more deeply on a particular block.  Linking within the manual could connect up the various functional blocks (inputs, outputs, and tasks) that comprise the overall flow.
>
> * Manual / Website Mix - Devise a mix between the YP Reference Manual and some pages in the YP Website that provide the information.  Create a section in the "Technical details" Chapter of the YP Reference Manual that covers this information at a high level.  For example, the overall flow of the system with its "big-block" inputs and outputs and tasks could be discussed at some length.  Links in the text could go to the YP website where more detail would be revealed.  This strategy effectively splits the content into "overview" and "details" between the manual and website, respectively.
>
> * Website - With this strategy, everything is pretty much in the YP website.  This area would exist in a stand-alone fashion.  However, you could link to the website from within the existing YP documentation set from existing areas that deal with the build process.  Several exit points from within the manual set already exist.  We would obviously add a primary one as well that would likely originate from the YP Reference Manual's "Technical Details" Chapter.
>
> SOLICITATION FOR INFORMATION
>
> We can get started now on this by starting to define details for some obvious points or large areas of the flow.  What would be nice to get would be some graphical breakdowns of these areas of concern:
>
> * User-defined layers
> * YP provided layers
> * User configuration
> * Machine Configuration
> * Policy Configuration
> * Patches
> * YP provided recipes
> * User provided source code
> * YP provided source code
> * SCMs
> * Generated images
> * Generated SDKs/toolchains
> * Package Output
> * Source fetching process
> * Patch application process
> * Configuration process (fragments, etc.)

Hi Scott,

Just so I'm clear .. are you looking for graphical breakdowns to be 
created and sent,
or information offered so graphical breakdowns can be created ?

The reason I ask, is that if there's any interest in the linux-yocto
(for lack of a better term) flow being described graphically, I'm
happy to offer up information, but my graphical skills are limited
to what you find in a typical hackers bag of ticks :)

Cheers,

Bruce

> * Key variable use and effects
> * User-initiated commands along the way
>
> Much of this list is directly from our existing "The Yocto Project Development Environment" illustration.
>
> Thanks,
> Scott R.
>
>
>
> Scott Rifenbark
> Intel Corporation
> Yocto Project Documentation
> 503.712.2702
> 503.341.0418 (cell)
>
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

Re: Documenting YP Development Environment in more detail

[Richard Purdie]

On Thu, 2013-06-20 at 09:49 -0400, Bruce Ashfield wrote:
> On 13-06-20 04:40 AM, Rifenbark, Scott M wrote:
> > Hi,
> >
> > Recently, Paul, Ross, Richard and I had a video conference meeting where we had some initial discussion on how to satisfy YOCTO #2808 (https://bugzilla.yoctoproject.org/show_bug.cgi?id=2808), which calls for an illustration showing source and destination directories during a build using YP.  This bug originated from a discussion Dave Stewart and I had a while back around an idea of a more detailed "flow diagram" that would go into greater detail than the now famous and ubiquitous "The Yocto Project Development Environment" illustration, which appears in the YP Quick Start (http://www.yoctoproject.org/docs/1.4/yocto-project-qs/yocto-project-qs.html) and has been used in many other places and in many other forms (some quite elaborate).
> >
> > We can't really create a completely accurate, all-encompassing illustration that breaks apart the entire build process.  It is not a static thing and it is quite complicated inside the BitBake process.  We can, however, show where metadata comes from, how it is provided, what it defines, where and how source files are found, what processes occur, what output is produced, and where it ends up.  We can also provide some sort of idea of how key BitBake and environment variables affect things along the way.  The idea here is to dig deeper into this conceptual figure and root out the details to a level that would be useful to the user but not impossible to maintain or develop. This type of information can be communicated through a mix of several illustrations with supporting text.
> >
> > This first meeting started with some detailed discussion of the configuration inputs for a typical YP build but soon migrated to discussing the bigger picture and possible ways to provide more information.  It became obvious we were not going to dig the solution and all the needed information out in one short meeting. Consequently, I am sending out this email to help open up some discussion on the issue and to also solicit information for some basic blocks of information.
> >
> > Two things are needed at this point:  1) a presentation solution for this new and more detailed information, and 2) a starting point on some of the information itself.
> >
> > PRESENTATION SOLUTION
> >
> > Here are some thoughts on how to present this information.  There
> are disadvantages and advantages to each of these methods of which I
> will not list.  I would like to see what people think about them:
> >
> >   * Manual - Create a section in the "Technical Details" Chapter of
> the YP Reference Manual that holds this information.  The section
> would be pretty much self-contained and would consist of several
> illustrations that would stem from an overall figure that is similar
> to the figure we now have in the YP Quick Start.  However, we would
> use a drill-down strategy that would progressively reveal more detail
> through subsequent figures.  This method is similar to how hardware
> devices used to be documented where functional blocks would be
> connected and described in one area and then subsequent areas would
> elaborate more deeply on a particular block.  Linking within the
> manual could connect up the various functional blocks (inputs,
> outputs, and tasks) that comprise the overall flow.
> >
> > * Manual / Website Mix - Devise a mix between the YP Reference
> Manual and some pages in the YP Website that provide the information.
> Create a section in the "Technical details" Chapter of the YP
> Reference Manual that covers this information at a high level.  For
> example, the overall flow of the system with its "big-block" inputs
> and outputs and tasks could be discussed at some length.  Links in the
> text could go to the YP website where more detail would be revealed.
> This strategy effectively splits the content into "overview" and
> "details" between the manual and website, respectively.
> >
> > * Website - With this strategy, everything is pretty much in the YP
> website.  This area would exist in a stand-alone fashion.  However,
> you could link to the website from within the existing YP
> documentation set from existing areas that deal with the build
> process.  Several exit points from within the manual set already
> exist.  We would obviously add a primary one as well that would likely
> originate from the YP Reference Manual's "Technical Details" Chapter.

I'm wondering if a hybrid is possible here. Can we for example embed
image maps into the docbook so that if you hover over areas of the
image, you can then have links to the different sections?

> > SOLICITATION FOR INFORMATION
> >
> > We can get started now on this by starting to define details for
> some obvious points or large areas of the flow.  What would be nice to
> get would be some graphical breakdowns of these areas of concern:
> >
> > * User-defined layers
> > * YP provided layers
> > * User configuration
> > * Machine Configuration
> > * Policy Configuration
> > * Patches
> > * YP provided recipes
> > * User provided source code
> > * YP provided source code
> > * SCMs
> > * Generated images
> > * Generated SDKs/toolchains
> > * Package Output
> > * Source fetching process
> > * Patch application process
> > * Configuration process (fragments, etc.)
>
> Just so I'm clear .. are you looking for graphical breakdowns to be 
> created and sent,
> or information offered so graphical breakdowns can be created ?
> 
> The reason I ask, is that if there's any interest in the linux-yocto
> (for lack of a better term) flow being described graphically, I'm
> happy to offer up information, but my graphical skills are limited
> to what you find in a typical hackers bag of ticks :)

I suggested that Scott try and accumulate information for each topic
like the following:

Fetcher:

Code Areas: Bitbake Fetch Module (bitbake/lib/bb/fetch2, called from
classes/base.bbclass)
Tasks Covered: do_fetch/do_unpack
Key Variables: SRC_URI, checksums etc

Generated images:

Code Areas: classes/image*.bbclass classes/rootfs*.bbclass
Tasks Covered: do_rootfs
Key Variables: PACKAGE_INSTALL

along with a description about what the function
> Cheers,
> 
> Bruce
> 
> > * Key variable use and effects
> > * User-initiated commands along the way
> >
> > Much of this list is directly from our existing "The Yocto Project
Development Environment" illustration.
> >
> > Thanks,
> > Scott R.
> >
> >
> >
> > Scott Rifenbark
> > Intel Corporation
> > Yocto Project Documentation
> > 503.712.2702
> > 503.341.0418 (cell)
> >
> > _______________________________________________
> > yocto mailing list
> > yocto@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/yocto
> >
> 
> al block does and when its used. The above would then link into the
class reference, the variable glossary and so on. So its less about
graphics and more about giving Scott the information to create a kind of
details index of some of these parts of the system like an expanded
table of contents.

I've suggested a good start would be picking a few areas (like the
above) and trying to create the info and maybe see what kind of diagram
would present itself from that. If successful, it could then be expanded
to each area. I'd certainly hope that linux-yocto would be one area
covered.

Cheers,

Richard

Re: Documenting YP Development Environment in more detail

[Bruce Ashfield]

On 13-06-20 10:25 AM, Richard Purdie wrote:
> On Thu, 2013-06-20 at 09:49 -0400, Bruce Ashfield wrote:
>> On 13-06-20 04:40 AM, Rifenbark, Scott M wrote:
>>> Hi,
>>>
>>> Recently, Paul, Ross, Richard and I had a video conference meeting where we had some initial discussion on how to satisfy YOCTO #2808 (https://bugzilla.yoctoproject.org/show_bug.cgi?id=2808), which calls for an illustration showing source and destination directories during a build using YP.  This bug originated from a discussion Dave Stewart and I had a while back around an idea of a more detailed "flow diagram" that would go into greater detail than the now famous and ubiquitous "The Yocto Project Development Environment" illustration, which appears in the YP Quick Start (http://www.yoctoproject.org/docs/1.4/yocto-project-qs/yocto-project-qs.html) and has been used in many other places and in many other forms (some quite elaborate).
>>>
>>> We can't really create a completely accurate, all-encompassing illustration that breaks apart the entire build process.  It is not a static thing and it is quite complicated inside the BitBake process.  We can, however, show where metadata comes from, how it is provided, what it defines, where and how source files are found, what processes occur, what output is produced, and where it ends up.  We can also provide some sort of idea of how key BitBake and environment variables affect things along the way.  The idea here is to dig deeper into this conceptual figure and root out the details to a level that would be useful to the user but not impossible to maintain or develop. This type of information can be communicated through a mix of several illustrations with supporting text.
>>>
>>> This first meeting started with some detailed discussion of the configuration inputs for a typical YP build but soon migrated to discussing the bigger picture and possible ways to provide more information.  It became obvious we were not going to dig the solution and all the needed information out in one short meeting. Consequently, I am sending out this email to help open up some discussion on the issue and to also solicit information for some basic blocks of information.
>>>
>>> Two things are needed at this point:  1) a presentation solution for this new and more detailed information, and 2) a starting point on some of the information itself.
>>>
>>> PRESENTATION SOLUTION
>>>
>>> Here are some thoughts on how to present this information.  There
>> are disadvantages and advantages to each of these methods of which I
>> will not list.  I would like to see what people think about them:
>>>
>>>    * Manual - Create a section in the "Technical Details" Chapter of
>> the YP Reference Manual that holds this information.  The section
>> would be pretty much self-contained and would consist of several
>> illustrations that would stem from an overall figure that is similar
>> to the figure we now have in the YP Quick Start.  However, we would
>> use a drill-down strategy that would progressively reveal more detail
>> through subsequent figures.  This method is similar to how hardware
>> devices used to be documented where functional blocks would be
>> connected and described in one area and then subsequent areas would
>> elaborate more deeply on a particular block.  Linking within the
>> manual could connect up the various functional blocks (inputs,
>> outputs, and tasks) that comprise the overall flow.
>>>
>>> * Manual / Website Mix - Devise a mix between the YP Reference
>> Manual and some pages in the YP Website that provide the information.
>> Create a section in the "Technical details" Chapter of the YP
>> Reference Manual that covers this information at a high level.  For
>> example, the overall flow of the system with its "big-block" inputs
>> and outputs and tasks could be discussed at some length.  Links in the
>> text could go to the YP website where more detail would be revealed.
>> This strategy effectively splits the content into "overview" and
>> "details" between the manual and website, respectively.
>>>
>>> * Website - With this strategy, everything is pretty much in the YP
>> website.  This area would exist in a stand-alone fashion.  However,
>> you could link to the website from within the existing YP
>> documentation set from existing areas that deal with the build
>> process.  Several exit points from within the manual set already
>> exist.  We would obviously add a primary one as well that would likely
>> originate from the YP Reference Manual's "Technical Details" Chapter.
>
> I'm wondering if a hybrid is possible here. Can we for example embed
> image maps into the docbook so that if you hover over areas of the
> image, you can then have links to the different sections?
>
>>> SOLICITATION FOR INFORMATION
>>>
>>> We can get started now on this by starting to define details for
>> some obvious points or large areas of the flow.  What would be nice to
>> get would be some graphical breakdowns of these areas of concern:
>>>
>>> * User-defined layers
>>> * YP provided layers
>>> * User configuration
>>> * Machine Configuration
>>> * Policy Configuration
>>> * Patches
>>> * YP provided recipes
>>> * User provided source code
>>> * YP provided source code
>>> * SCMs
>>> * Generated images
>>> * Generated SDKs/toolchains
>>> * Package Output
>>> * Source fetching process
>>> * Patch application process
>>> * Configuration process (fragments, etc.)
>>
>> Just so I'm clear .. are you looking for graphical breakdowns to be
>> created and sent,
>> or information offered so graphical breakdowns can be created ?
>>
>> The reason I ask, is that if there's any interest in the linux-yocto
>> (for lack of a better term) flow being described graphically, I'm
>> happy to offer up information, but my graphical skills are limited
>> to what you find in a typical hackers bag of ticks :)
>
> I suggested that Scott try and accumulate information for each topic
> like the following:
>
> Fetcher:
>
> Code Areas: Bitbake Fetch Module (bitbake/lib/bb/fetch2, called from
> classes/base.bbclass)
> Tasks Covered: do_fetch/do_unpack
> Key Variables: SRC_URI, checksums etc
>
> Generated images:
>
> Code Areas: classes/image*.bbclass classes/rootfs*.bbclass
> Tasks Covered: do_rootfs
> Key Variables: PACKAGE_INSTALL
>
> along with a description about what the function

Aha. Great! Keeping things up to date will of course be the trick. I'll
be on standby if anything is needed from my front.

Cheers,

Bruce

>> Cheers,
>>
>> Bruce
>>
>>> * Key variable use and effects
>>> * User-initiated commands along the way
>>>
>>> Much of this list is directly from our existing "The Yocto Project
> Development Environment" illustration.
>>>
>>> Thanks,
>>> Scott R.
>>>
>>>
>>>
>>> Scott Rifenbark
>>> Intel Corporation
>>> Yocto Project Documentation
>>> 503.712.2702
>>> 503.341.0418 (cell)
>>>
>>> _______________________________________________
>>> yocto mailing list
>>> yocto@yoctoproject.org
>>> https://lists.yoctoproject.org/listinfo/yocto
>>>
>>
>> al block does and when its used. The above would then link into the
> class reference, the variable glossary and so on. So its less about
> graphics and more about giving Scott the information to create a kind of
> details index of some of these parts of the system like an expanded
> table of contents.
>
> I've suggested a good start would be picking a few areas (like the
> above) and trying to create the info and maybe see what kind of diagram
> would present itself from that. If successful, it could then be expanded
> to each area. I'd certainly hope that linux-yocto would be one area
> covered.
>
> Cheers,
>
> Richard
>
>
>
>
>
>
>
>

Re: Documenting YP Development Environment in more detail

[Bill Traynor]

On Thu, Jun 20, 2013 at 10:25 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Thu, 2013-06-20 at 09:49 -0400, Bruce Ashfield wrote:
>> On 13-06-20 04:40 AM, Rifenbark, Scott M wrote:
>> > Hi,
>> >
>> > Recently, Paul, Ross, Richard and I had a video conference meeting where we had some initial discussion on how to satisfy YOCTO #2808 (https://bugzilla.yoctoproject.org/show_bug.cgi?id=2808), which calls for an illustration showing source and destination directories during a build using YP.  This bug originated from a discussion Dave Stewart and I had a while back around an idea of a more detailed "flow diagram" that would go into greater detail than the now famous and ubiquitous "The Yocto Project Development Environment" illustration, which appears in the YP Quick Start (http://www.yoctoproject.org/docs/1.4/yocto-project-qs/yocto-project-qs.html) and has been used in many other places and in many other forms (some quite elaborate).
>> >
>> > We can't really create a completely accurate, all-encompassing illustration that breaks apart the entire build process.  It is not a static thing and it is quite complicated inside the BitBake process.  We can, however, show where metadata comes from, how it is provided, what it defines, where and how source files are found, what processes occur, what output is produced, and where it ends up.  We can also provide some sort of idea of how key BitBake and environment variables affect things along the way.  The idea here is to dig deeper into this conceptual figure and root out the details to a level that would be useful to the user but not impossible to maintain or develop. This type of information can be communicated through a mix of several illustrations with supporting text.
>> >
>> > This first meeting started with some detailed discussion of the configuration inputs for a typical YP build but soon migrated to discussing the bigger picture and possible ways to provide more information.  It became obvious we were not going to dig the solution and all the needed information out in one short meeting. Consequently, I am sending out this email to help open up some discussion on the issue and to also solicit information for some basic blocks of information.
>> >
>> > Two things are needed at this point:  1) a presentation solution for this new and more detailed information, and 2) a starting point on some of the information itself.
>> >
>> > PRESENTATION SOLUTION
>> >
>> > Here are some thoughts on how to present this information.  There
>> are disadvantages and advantages to each of these methods of which I
>> will not list.  I would like to see what people think about them:
>> >
>> >   * Manual - Create a section in the "Technical Details" Chapter of
>> the YP Reference Manual that holds this information.  The section
>> would be pretty much self-contained and would consist of several
>> illustrations that would stem from an overall figure that is similar
>> to the figure we now have in the YP Quick Start.  However, we would
>> use a drill-down strategy that would progressively reveal more detail
>> through subsequent figures.  This method is similar to how hardware
>> devices used to be documented where functional blocks would be
>> connected and described in one area and then subsequent areas would
>> elaborate more deeply on a particular block.  Linking within the
>> manual could connect up the various functional blocks (inputs,
>> outputs, and tasks) that comprise the overall flow.
>> >
>> > * Manual / Website Mix - Devise a mix between the YP Reference
>> Manual and some pages in the YP Website that provide the information.
>> Create a section in the "Technical details" Chapter of the YP
>> Reference Manual that covers this information at a high level.  For
>> example, the overall flow of the system with its "big-block" inputs
>> and outputs and tasks could be discussed at some length.  Links in the
>> text could go to the YP website where more detail would be revealed.
>> This strategy effectively splits the content into "overview" and
>> "details" between the manual and website, respectively.
>> >
>> > * Website - With this strategy, everything is pretty much in the YP
>> website.  This area would exist in a stand-alone fashion.  However,
>> you could link to the website from within the existing YP
>> documentation set from existing areas that deal with the build
>> process.  Several exit points from within the manual set already
>> exist.  We would obviously add a primary one as well that would likely
>> originate from the YP Reference Manual's "Technical Details" Chapter.
>
> I'm wondering if a hybrid is possible here. Can we for example embed
> image maps into the docbook so that if you hover over areas of the
> image, you can then have links to the different sections?

FWIW, docbook supports image maps via the mediaobject tag but they are
somewhat limited in their usefulness.

http://docbook.org/tdg5/en/html/mediaobject.html
http://www.sagehill.net/docbookxsl/Imagemaps.html

Scott, you may run into trouble when converting the docs between doc
types, in particular with scaling of images.

>
>> > SOLICITATION FOR INFORMATION
>> >
>> > We can get started now on this by starting to define details for
>> some obvious points or large areas of the flow.  What would be nice to
>> get would be some graphical breakdowns of these areas of concern:
>> >
>> > * User-defined layers
>> > * YP provided layers
>> > * User configuration
>> > * Machine Configuration
>> > * Policy Configuration
>> > * Patches
>> > * YP provided recipes
>> > * User provided source code
>> > * YP provided source code
>> > * SCMs
>> > * Generated images
>> > * Generated SDKs/toolchains
>> > * Package Output
>> > * Source fetching process
>> > * Patch application process
>> > * Configuration process (fragments, etc.)
>>
>> Just so I'm clear .. are you looking for graphical breakdowns to be
>> created and sent,
>> or information offered so graphical breakdowns can be created ?
>>
>> The reason I ask, is that if there's any interest in the linux-yocto
>> (for lack of a better term) flow being described graphically, I'm
>> happy to offer up information, but my graphical skills are limited
>> to what you find in a typical hackers bag of ticks :)
>
> I suggested that Scott try and accumulate information for each topic
> like the following:
>
> Fetcher:
>
> Code Areas: Bitbake Fetch Module (bitbake/lib/bb/fetch2, called from
> classes/base.bbclass)
> Tasks Covered: do_fetch/do_unpack
> Key Variables: SRC_URI, checksums etc
>
> Generated images:
>
> Code Areas: classes/image*.bbclass classes/rootfs*.bbclass
> Tasks Covered: do_rootfs
> Key Variables: PACKAGE_INSTALL
>
> along with a description about what the function
>> Cheers,
>>
>> Bruce
>>
>> > * Key variable use and effects
>> > * User-initiated commands along the way
>> >
>> > Much of this list is directly from our existing "The Yocto Project
> Development Environment" illustration.
>> >
>> > Thanks,
>> > Scott R.
>> >
>> >
>> >
>> > Scott Rifenbark
>> > Intel Corporation
>> > Yocto Project Documentation
>> > 503.712.2702
>> > 503.341.0418 (cell)
>> >
>> > _______________________________________________
>> > yocto mailing list
>> > yocto@yoctoproject.org
>> > https://lists.yoctoproject.org/listinfo/yocto
>> >
>>
>> al block does and when its used. The above would then link into the
> class reference, the variable glossary and so on. So its less about
> graphics and more about giving Scott the information to create a kind of
> details index of some of these parts of the system like an expanded
> table of contents.
>
> I've suggested a good start would be picking a few areas (like the
> above) and trying to create the info and maybe see what kind of diagram
> would present itself from that. If successful, it could then be expanded
> to each area. I'd certainly hope that linux-yocto would be one area
> covered.
>
> Cheers,
>
> Richard
>
>
>
>
>
>
>
>
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Re: Documenting YP Development Environment in more detail

[Rifenbark, Scott M]

>-----Original Message-----
>From: yocto-bounces@yoctoproject.org [mailto:yocto-
>bounces@yoctoproject.org] On Behalf Of Bill Traynor
>Sent: Thursday, June 20, 2013 8:12 AM
>To: Richard Purdie
>Cc: yocto@yoctoproject.org; Paul Eggleton; Osier-mixon, Jeffrey
>Subject: Re: [yocto] Documenting YP Development Environment in more
>detail
>
>On Thu, Jun 20, 2013 at 10:25 AM, Richard Purdie
><richard.purdie@linuxfoundation.org> wrote:
>> On Thu, 2013-06-20 at 09:49 -0400, Bruce Ashfield wrote:
>>> On 13-06-20 04:40 AM, Rifenbark, Scott M wrote:
>>> > Hi,
>>> >
>>> > Recently, Paul, Ross, Richard and I had a video conference meeting
>where we had some initial discussion on how to satisfy YOCTO #2808
>(https://bugzilla.yoctoproject.org/show_bug.cgi?id=2808), which calls
>for an illustration showing source and destination directories during a
>build using YP.  This bug originated from a discussion Dave Stewart and
>I had a while back around an idea of a more detailed "flow diagram" that
>would go into greater detail than the now famous and ubiquitous "The
>Yocto Project Development Environment" illustration, which appears in
>the YP Quick Start (http://www.yoctoproject.org/docs/1.4/yocto-project-
>qs/yocto-project-qs.html) and has been used in many other places and in
>many other forms (some quite elaborate).
>>> >
>>> > We can't really create a completely accurate, all-encompassing
>illustration that breaks apart the entire build process.  It is not a
>static thing and it is quite complicated inside the BitBake process.  We
>can, however, show where metadata comes from, how it is provided, what
>it defines, where and how source files are found, what processes occur,
>what output is produced, and where it ends up.  We can also provide some
>sort of idea of how key BitBake and environment variables affect things
>along the way.  The idea here is to dig deeper into this conceptual
>figure and root out the details to a level that would be useful to the
>user but not impossible to maintain or develop. This type of information
>can be communicated through a mix of several illustrations with
>supporting text.
>>> >
>>> > This first meeting started with some detailed discussion of the
>configuration inputs for a typical YP build but soon migrated to
>discussing the bigger picture and possible ways to provide more
>information.  It became obvious we were not going to dig the solution
>and all the needed information out in one short meeting. Consequently, I
>am sending out this email to help open up some discussion on the issue
>and to also solicit information for some basic blocks of information.
>>> >
>>> > Two things are needed at this point:  1) a presentation solution
>for this new and more detailed information, and 2) a starting point on
>some of the information itself.
>>> >
>>> > PRESENTATION SOLUTION
>>> >
>>> > Here are some thoughts on how to present this information.  There
>>> are disadvantages and advantages to each of these methods of which I
>>> will not list.  I would like to see what people think about them:
>>> >
>>> >   * Manual - Create a section in the "Technical Details" Chapter of
>>> the YP Reference Manual that holds this information.  The section
>>> would be pretty much self-contained and would consist of several
>>> illustrations that would stem from an overall figure that is similar
>>> to the figure we now have in the YP Quick Start.  However, we would
>>> use a drill-down strategy that would progressively reveal more detail
>>> through subsequent figures.  This method is similar to how hardware
>>> devices used to be documented where functional blocks would be
>>> connected and described in one area and then subsequent areas would
>>> elaborate more deeply on a particular block.  Linking within the
>>> manual could connect up the various functional blocks (inputs,
>>> outputs, and tasks) that comprise the overall flow.
>>> >
>>> > * Manual / Website Mix - Devise a mix between the YP Reference
>>> Manual and some pages in the YP Website that provide the information.
>>> Create a section in the "Technical details" Chapter of the YP
>>> Reference Manual that covers this information at a high level.  For
>>> example, the overall flow of the system with its "big-block" inputs
>>> and outputs and tasks could be discussed at some length.  Links in
>the
>>> text could go to the YP website where more detail would be revealed.
>>> This strategy effectively splits the content into "overview" and
>>> "details" between the manual and website, respectively.
>>> >
>>> > * Website - With this strategy, everything is pretty much in the YP
>>> website.  This area would exist in a stand-alone fashion.  However,
>>> you could link to the website from within the existing YP
>>> documentation set from existing areas that deal with the build
>>> process.  Several exit points from within the manual set already
>>> exist.  We would obviously add a primary one as well that would
>likely
>>> originate from the YP Reference Manual's "Technical Details" Chapter.
>>
>> I'm wondering if a hybrid is possible here. Can we for example embed
>> image maps into the docbook so that if you hover over areas of the
>> image, you can then have links to the different sections?
>
>FWIW, docbook supports image maps via the mediaobject tag but they are
>somewhat limited in their usefulness.
>
>http://docbook.org/tdg5/en/html/mediaobject.html
>http://www.sagehill.net/docbookxsl/Imagemaps.html
>
>Scott, you may run into trouble when converting the docs between doc
>types, in particular with scaling of images.

I will do some playing around with image maps and see what comes up. 
It would be nice to see if we could make some "hot-spots" within 
Illustrations.


>
>>
>>> > SOLICITATION FOR INFORMATION
>>> >
>>> > We can get started now on this by starting to define details for
>>> some obvious points or large areas of the flow.  What would be nice
>to
>>> get would be some graphical breakdowns of these areas of concern:
>>> >
>>> > * User-defined layers
>>> > * YP provided layers
>>> > * User configuration
>>> > * Machine Configuration
>>> > * Policy Configuration
>>> > * Patches
>>> > * YP provided recipes
>>> > * User provided source code
>>> > * YP provided source code
>>> > * SCMs
>>> > * Generated images
>>> > * Generated SDKs/toolchains
>>> > * Package Output
>>> > * Source fetching process
>>> > * Patch application process
>>> > * Configuration process (fragments, etc.)
>>>
>>> Just so I'm clear .. are you looking for graphical breakdowns to be
>>> created and sent,
>>> or information offered so graphical breakdowns can be created ?

Bruce - any kind of information in any form is useful.  If you need to hack up a drawing to get a point across... do so.  Or, if you can send textual information that is good too.

>>>
>>> The reason I ask, is that if there's any interest in the linux-yocto
>>> (for lack of a better term) flow being described graphically, I'm
>>> happy to offer up information, but my graphical skills are limited
>>> to what you find in a typical hackers bag of ticks :)
>>
>> I suggested that Scott try and accumulate information for each topic
>> like the following:
>>
>> Fetcher:
>>
>> Code Areas: Bitbake Fetch Module (bitbake/lib/bb/fetch2, called from
>> classes/base.bbclass)
>> Tasks Covered: do_fetch/do_unpack
>> Key Variables: SRC_URI, checksums etc
>>
>> Generated images:
>>
>> Code Areas: classes/image*.bbclass classes/rootfs*.bbclass
>> Tasks Covered: do_rootfs
>> Key Variables: PACKAGE_INSTALL
>>
>> along with a description about what the function
>>> Cheers,
>>>
>>> Bruce
>>>
>>> > * Key variable use and effects
>>> > * User-initiated commands along the way
>>> >
>>> > Much of this list is directly from our existing "The Yocto Project
>> Development Environment" illustration.
>>> >
>>> > Thanks,
>>> > Scott R.
>>> >
>>> >
>>> >
>>> > Scott Rifenbark
>>> > Intel Corporation
>>> > Yocto Project Documentation
>>> > 503.712.2702
>>> > 503.341.0418 (cell)
>>> >
>>> > _______________________________________________
>>> > yocto mailing list
>>> > yocto@yoctoproject.org
>>> > https://lists.yoctoproject.org/listinfo/yocto
>>> >
>>>
>>> al block does and when its used. The above would then link into the
>> class reference, the variable glossary and so on. So its less about
>> graphics and more about giving Scott the information to create a kind
>of
>> details index of some of these parts of the system like an expanded
>> table of contents.
>>
>> I've suggested a good start would be picking a few areas (like the
>> above) and trying to create the info and maybe see what kind of
>diagram
>> would present itself from that. If successful, it could then be
>expanded
>> to each area. I'd certainly hope that linux-yocto would be one area
>> covered.
>>
>> Cheers,
>>
>> Richard
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/yocto
>_______________________________________________
>yocto mailing list
>yocto@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/yocto

Re: Documenting YP Development Environment in more detail

[Stewart, David C]

On 6/20/13 8:23 AM, "Rifenbark, Scott M" <scott.m.rifenbark@intel.com>
wrote:

>
>
>>-----Original Message-----
>>From: yocto-bounces@yoctoproject.org [mailto:yocto-
>>bounces@yoctoproject.org] On Behalf Of Bill Traynor
>>Sent: Thursday, June 20, 2013 8:12 AM
>>To: Richard Purdie
>>Cc: yocto@yoctoproject.org; Paul Eggleton; Osier-mixon, Jeffrey
>>Subject: Re: [yocto] Documenting YP Development Environment in more
>>detail
>>
>>On Thu, Jun 20, 2013 at 10:25 AM, Richard Purdie
>><richard.purdie@linuxfoundation.org> wrote:
>>> On Thu, 2013-06-20 at 09:49 -0400, Bruce Ashfield wrote:
>>>> On 13-06-20 04:40 AM, Rifenbark, Scott M wrote:
>>>> > Hi,
>>>> >
>>>> > Recently, Paul, Ross, Richard and I had a video conference meeting
>>where we had some initial discussion on how to satisfy YOCTO #2808
>>(https://bugzilla.yoctoproject.org/show_bug.cgi?id=2808), which calls
>>for an illustration showing source and destination directories during a
>>build using YP.  This bug originated from a discussion Dave Stewart and
>>I had a while back around an idea of a more detailed "flow diagram" that
>>would go into greater detail than the now famous and ubiquitous "The
>>Yocto Project Development Environment" illustration, which appears in
>>the YP Quick Start (http://www.yoctoproject.org/docs/1.4/yocto-project-
>>qs/yocto-project-qs.html) and has been used in many other places and in
>>many other forms (some quite elaborate).
>>>> >
>>>> > We can't really create a completely accurate, all-encompassing
>>illustration that breaks apart the entire build process.  It is not a
>>static thing and it is quite complicated inside the BitBake process.  We
>>can, however, show where metadata comes from, how it is provided, what
>>it defines, where and how source files are found, what processes occur,
>>what output is produced, and where it ends up.  We can also provide some
>>sort of idea of how key BitBake and environment variables affect things
>>along the way.  The idea here is to dig deeper into this conceptual
>>figure and root out the details to a level that would be useful to the
>>user but not impossible to maintain or develop. This type of information
>>can be communicated through a mix of several illustrations with
>>supporting text.
>>>> >
>>>> > This first meeting started with some detailed discussion of the
>>configuration inputs for a typical YP build but soon migrated to
>>discussing the bigger picture and possible ways to provide more
>>information.  It became obvious we were not going to dig the solution
>>and all the needed information out in one short meeting. Consequently, I
>>am sending out this email to help open up some discussion on the issue
>>and to also solicit information for some basic blocks of information.

I would recommend that we don't over-engineer this thing if we can avoid
it.
Remember the goal: People have told me that they are confused about which
directories
are "active" in the build process, i.e. Where does stuff come from and
where does it go?
My advice would be to think of bitbake as a black box with inputs and
outputs.
I know that any directory could be *potentially* an input or output (why
else would it be there?)
But I'm hoping we can up level a little.

>>>> >
>>>> > Two things are needed at this point:  1) a presentation solution
>>for this new and more detailed information, and 2) a starting point on
>>some of the information itself.
>>>> >
>>>> > PRESENTATION SOLUTION
>>>> >
>>>> > Here are some thoughts on how to present this information.  There
>>>> are disadvantages and advantages to each of these methods of which I
>>>> will not list.  I would like to see what people think about them:
>>>> >
>>>> >   * Manual - Create a section in the "Technical Details" Chapter of
>>>> the YP Reference Manual that holds this information.  The section
>>>> would be pretty much self-contained and would consist of several
>>>> illustrations that would stem from an overall figure that is similar
>>>> to the figure we now have in the YP Quick Start.  However, we would
>>>> use a drill-down strategy that would progressively reveal more detail
>>>> through subsequent figures.  This method is similar to how hardware
>>>> devices used to be documented where functional blocks would be
>>>> connected and described in one area and then subsequent areas would
>>>> elaborate more deeply on a particular block.  Linking within the
>>>> manual could connect up the various functional blocks (inputs,
>>>> outputs, and tasks) that comprise the overall flow.
>>>> >
>>>> > * Manual / Website Mix - Devise a mix between the YP Reference
>>>> Manual and some pages in the YP Website that provide the information.
>>>> Create a section in the "Technical details" Chapter of the YP
>>>> Reference Manual that covers this information at a high level.  For
>>>> example, the overall flow of the system with its "big-block" inputs
>>>> and outputs and tasks could be discussed at some length.  Links in
>>the
>>>> text could go to the YP website where more detail would be revealed.
>>>> This strategy effectively splits the content into "overview" and
>>>> "details" between the manual and website, respectively.
>>>> >
>>>> > * Website - With this strategy, everything is pretty much in the YP
>>>> website.  This area would exist in a stand-alone fashion.  However,
>>>> you could link to the website from within the existing YP
>>>> documentation set from existing areas that deal with the build
>>>> process.  Several exit points from within the manual set already
>>>> exist.  We would obviously add a primary one as well that would
>>likely
>>>> originate from the YP Reference Manual's "Technical Details" Chapter.
>>>
>>> I'm wondering if a hybrid is possible here. Can we for example embed
>>> image maps into the docbook so that if you hover over areas of the
>>> image, you can then have links to the different sections?
>>
>>FWIW, docbook supports image maps via the mediaobject tag but they are
>>somewhat limited in their usefulness.
>>
>>http://docbook.org/tdg5/en/html/mediaobject.html
>>http://www.sagehill.net/docbookxsl/Imagemaps.html
>>
>>Scott, you may run into trouble when converting the docs between doc
>>types, in particular with scaling of images.
>
>I will do some playing around with image maps and see what comes up.
>It would be nice to see if we could make some "hot-spots" within
>Illustrations.
>
>
>>
>>>
>>>> > SOLICITATION FOR INFORMATION
>>>> >
>>>> > We can get started now on this by starting to define details for
>>>> some obvious points or large areas of the flow.  What would be nice
>>to
>>>> get would be some graphical breakdowns of these areas of concern:
>>>> >
>>>> > * User-defined layers
>>>> > * YP provided layers
>>>> > * User configuration
>>>> > * Machine Configuration
>>>> > * Policy Configuration
>>>> > * Patches
>>>> > * YP provided recipes
>>>> > * User provided source code
>>>> > * YP provided source code
>>>> > * SCMs
>>>> > * Generated images
>>>> > * Generated SDKs/toolchains
>>>> > * Package Output
>>>> > * Source fetching process
>>>> > * Patch application process
>>>> > * Configuration process (fragments, etc.)
>>>>
>>>> Just so I'm clear .. are you looking for graphical breakdowns to be
>>>> created and sent,
>>>> or information offered so graphical breakdowns can be created ?
>
>Bruce - any kind of information in any form is useful.  If you need to
>hack up a drawing to get a point across... do so.  Or, if you can send
>textual information that is good too.
>
>>>>
>>>> The reason I ask, is that if there's any interest in the linux-yocto
>>>> (for lack of a better term) flow being described graphically, I'm
>>>> happy to offer up information, but my graphical skills are limited
>>>> to what you find in a typical hackers bag of ticks :)
>>>
>>> I suggested that Scott try and accumulate information for each topic
>>> like the following:
>>>
>>> Fetcher:
>>>
>>> Code Areas: Bitbake Fetch Module (bitbake/lib/bb/fetch2, called from
>>> classes/base.bbclass)
>>> Tasks Covered: do_fetch/do_unpack
>>> Key Variables: SRC_URI, checksums etc
>>>
>>> Generated images:
>>>
>>> Code Areas: classes/image*.bbclass classes/rootfs*.bbclass
>>> Tasks Covered: do_rootfs
>>> Key Variables: PACKAGE_INSTALL
>>>
>>> along with a description about what the function
>>>> Cheers,
>>>>
>>>> Bruce
>>>>
>>>> > * Key variable use and effects
>>>> > * User-initiated commands along the way
>>>> >
>>>> > Much of this list is directly from our existing "The Yocto Project
>>> Development Environment" illustration.
>>>> >
>>>> > Thanks,
>>>> > Scott R.
>>>> >
>>>> >
>>>> >
>>>> > Scott Rifenbark
>>>> > Intel Corporation
>>>> > Yocto Project Documentation
>>>> > 503.712.2702
>>>> > 503.341.0418 (cell)
>>>> >
>>>> > _______________________________________________
>>>> > yocto mailing list
>>>> > yocto@yoctoproject.org
>>>> > https://lists.yoctoproject.org/listinfo/yocto
>>>> >
>>>>
>>>> al block does and when its used. The above would then link into the
>>> class reference, the variable glossary and so on. So its less about
>>> graphics and more about giving Scott the information to create a kind
>>of
>>> details index of some of these parts of the system like an expanded
>>> table of contents.
>>>
>>> I've suggested a good start would be picking a few areas (like the
>>> above) and trying to create the info and maybe see what kind of
>>diagram
>>> would present itself from that. If successful, it could then be
>>expanded
>>> to each area. I'd certainly hope that linux-yocto would be one area
>>> covered.
>>>
>>> Cheers,
>>>
>>> Richard
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> yocto mailing list
>>> yocto@yoctoproject.org
>>> https://lists.yoctoproject.org/listinfo/yocto
>>_______________________________________________
>>yocto mailing list
>>yocto@yoctoproject.org
>>https://lists.yoctoproject.org/listinfo/yocto
>_______________________________________________
>yocto mailing list
>yocto@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/yocto

Re: Documenting YP Development Environment in more detail

[Sean Liming]

> -----Original Message-----
> From: yocto-bounces@yoctoproject.org [mailto:yocto-
> bounces@yoctoproject.org] On Behalf Of Stewart, David C
> Sent: Thursday, June 20, 2013 11:23 AM
> To: Rifenbark, Scott M; Bill Traynor; Richard Purdie
> Cc: yocto@yoctoproject.org; Paul Eggleton; Osier-mixon, Jeffrey
> Subject: Re: [yocto] Documenting YP Development Environment in more
> detail
> 
> 
> 
> On 6/20/13 8:23 AM, "Rifenbark, Scott M" <scott.m.rifenbark@intel.com>
> wrote:
> 
> >
> >
> >>-----Original Message-----
> >>From: yocto-bounces@yoctoproject.org [mailto:yocto-
> >>bounces@yoctoproject.org] On Behalf Of Bill Traynor
> >>Sent: Thursday, June 20, 2013 8:12 AM
> >>To: Richard Purdie
> >>Cc: yocto@yoctoproject.org; Paul Eggleton; Osier-mixon, Jeffrey
> >>Subject: Re: [yocto] Documenting YP Development Environment in more
> >>detail
> >>
> >>On Thu, Jun 20, 2013 at 10:25 AM, Richard Purdie
> >><richard.purdie@linuxfoundation.org> wrote:
> >>> On Thu, 2013-06-20 at 09:49 -0400, Bruce Ashfield wrote:
> >>>> On 13-06-20 04:40 AM, Rifenbark, Scott M wrote:
> >>>> > Hi,
> >>>> >
> >>>> > Recently, Paul, Ross, Richard and I had a video conference
> >>>> > meeting
> >>where we had some initial discussion on how to satisfy YOCTO #2808
> >>(https://bugzilla.yoctoproject.org/show_bug.cgi?id=2808), which calls
> >>for an illustration showing source and destination directories during
> >>a build using YP.  This bug originated from a discussion Dave Stewart
> >>and I had a while back around an idea of a more detailed "flow
> >>diagram" that would go into greater detail than the now famous and
> >>ubiquitous "The Yocto Project Development Environment" illustration,
> >>which appears in the YP Quick Start
> >>(http://www.yoctoproject.org/docs/1.4/yocto-project-
> >>qs/yocto-project-qs.html) and has been used in many other places and
> >>in many other forms (some quite elaborate).
> >>>> >
> >>>> > We can't really create a completely accurate, all-encompassing
> >>illustration that breaks apart the entire build process.  It is not a
> >>static thing and it is quite complicated inside the BitBake process.
> >>We can, however, show where metadata comes from, how it is provided,
> >>what it defines, where and how source files are found, what processes
> >>occur, what output is produced, and where it ends up.  We can also
> >>provide some sort of idea of how key BitBake and environment variables
> >>affect things along the way.  The idea here is to dig deeper into this
> >>conceptual figure and root out the details to a level that would be
> >>useful to the user but not impossible to maintain or develop. This
> >>type of information can be communicated through a mix of several
> >>illustrations with supporting text.
> >>>> >
> >>>> > This first meeting started with some detailed discussion of the
> >>configuration inputs for a typical YP build but soon migrated to
> >>discussing the bigger picture and possible ways to provide more
> >>information.  It became obvious we were not going to dig the solution
> >>and all the needed information out in one short meeting. Consequently,
> >>I am sending out this email to help open up some discussion on the
> >>issue and to also solicit information for some basic blocks of
information.
> 
> I would recommend that we don't over-engineer this thing if we can avoid
it.
> Remember the goal: People have told me that they are confused about
> which directories are "active" in the build process, i.e. Where does stuff
> come from and where does it go?
> My advice would be to think of bitbake as a black box with inputs and
> outputs.
> I know that any directory could be *potentially* an input or output (why
else
> would it be there?) But I'm hoping we can up level a little.
> 
> >>>> >
> >>>> > Two things are needed at this point:  1) a presentation solution
> >>for this new and more detailed information, and 2) a starting point on
> >>some of the information itself.
> >>>> >
> >>>> > PRESENTATION SOLUTION
> >>>> >
> >>>> > Here are some thoughts on how to present this information.  There
> >>>> are disadvantages and advantages to each of these methods of which
> >>>> I will not list.  I would like to see what people think about them:
> >>>> >
> >>>> >   * Manual - Create a section in the "Technical Details" Chapter
> >>>> > of
> >>>> the YP Reference Manual that holds this information.  The section
> >>>> would be pretty much self-contained and would consist of several
> >>>> illustrations that would stem from an overall figure that is
> >>>> similar to the figure we now have in the YP Quick Start.  However,
> >>>> we would use a drill-down strategy that would progressively reveal
> >>>> more detail through subsequent figures.  This method is similar to
> >>>> how hardware devices used to be documented where functional
> blocks
> >>>> would be connected and described in one area and then subsequent
> >>>> areas would elaborate more deeply on a particular block.  Linking
> >>>> within the manual could connect up the various functional blocks
> >>>> (inputs, outputs, and tasks) that comprise the overall flow.
> >>>> >
> >>>> > * Manual / Website Mix - Devise a mix between the YP Reference
> >>>> Manual and some pages in the YP Website that provide the
> information.
> >>>> Create a section in the "Technical details" Chapter of the YP
> >>>> Reference Manual that covers this information at a high level.  For
> >>>> example, the overall flow of the system with its "big-block" inputs
> >>>> and outputs and tasks could be discussed at some length.  Links in
> >>the
> >>>> text could go to the YP website where more detail would be revealed.
> >>>> This strategy effectively splits the content into "overview" and
> >>>> "details" between the manual and website, respectively.
> >>>> >
> >>>> > * Website - With this strategy, everything is pretty much in the
> >>>> > YP
> >>>> website.  This area would exist in a stand-alone fashion.  However,
> >>>> you could link to the website from within the existing YP
> >>>> documentation set from existing areas that deal with the build
> >>>> process.  Several exit points from within the manual set already
> >>>> exist.  We would obviously add a primary one as well that would
> >>likely
> >>>> originate from the YP Reference Manual's "Technical Details" Chapter.
> >>>
> >>> I'm wondering if a hybrid is possible here. Can we for example embed
> >>> image maps into the docbook so that if you hover over areas of the
> >>> image, you can then have links to the different sections?
> >>
> >>FWIW, docbook supports image maps via the mediaobject tag but they are
> >>somewhat limited in their usefulness.
> >>
> >>http://docbook.org/tdg5/en/html/mediaobject.html
> >>http://www.sagehill.net/docbookxsl/Imagemaps.html
> >>
> >>Scott, you may run into trouble when converting the docs between doc
> >>types, in particular with scaling of images.
> >
> >I will do some playing around with image maps and see what comes up.
> >It would be nice to see if we could make some "hot-spots" within
> >Illustrations.
> >
> >
> >>
> >>>
> >>>> > SOLICITATION FOR INFORMATION
> >>>> >
> >>>> > We can get started now on this by starting to define details for
> >>>> some obvious points or large areas of the flow.  What would be nice
> >>to
> >>>> get would be some graphical breakdowns of these areas of concern:
> >>>> >
> >>>> > * User-defined layers
> >>>> > * YP provided layers
> >>>> > * User configuration
> >>>> > * Machine Configuration
> >>>> > * Policy Configuration
> >>>> > * Patches
> >>>> > * YP provided recipes
> >>>> > * User provided source code
> >>>> > * YP provided source code
> >>>> > * SCMs
> >>>> > * Generated images
> >>>> > * Generated SDKs/toolchains
> >>>> > * Package Output
> >>>> > * Source fetching process
> >>>> > * Patch application process
> >>>> > * Configuration process (fragments, etc.)
> >>>>
> >>>> Just so I'm clear .. are you looking for graphical breakdowns to be
> >>>> created and sent, or information offered so graphical breakdowns
> >>>> can be created ?
> >
> >Bruce - any kind of information in any form is useful.  If you need to
> >hack up a drawing to get a point across... do so.  Or, if you can send
> >textual information that is good too.
> >
> >>>>
> >>>> The reason I ask, is that if there's any interest in the
> >>>> linux-yocto (for lack of a better term) flow being described
> >>>> graphically, I'm happy to offer up information, but my graphical
> >>>> skills are limited to what you find in a typical hackers bag of
> >>>> ticks :)
> >>>
> >>> I suggested that Scott try and accumulate information for each topic
> >>> like the following:
> >>>
> >>> Fetcher:
> >>>
> >>> Code Areas: Bitbake Fetch Module (bitbake/lib/bb/fetch2, called from
> >>> classes/base.bbclass)
> >>> Tasks Covered: do_fetch/do_unpack
> >>> Key Variables: SRC_URI, checksums etc
> >>>
> >>> Generated images:
> >>>
> >>> Code Areas: classes/image*.bbclass classes/rootfs*.bbclass Tasks
> >>> Covered: do_rootfs Key Variables: PACKAGE_INSTALL
> >>>
> >>> along with a description about what the function
> >>>> Cheers,
> >>>>
> >>>> Bruce
> >>>>
> >>>> > * Key variable use and effects
> >>>> > * User-initiated commands along the way
> >>>> >
> >>>> > Much of this list is directly from our existing "The Yocto
> >>>> > Project
> >>> Development Environment" illustration.
> >>>> >
> >>>> > Thanks,
> >>>> > Scott R.
> >>>> >
> >>>> >
> >>>> >
> >>>> > Scott Rifenbark
> >>>> > Intel Corporation
> >>>> > Yocto Project Documentation
> >>>> > 503.712.2702
> >>>> > 503.341.0418 (cell)
> >>>> >
> >>>> > _______________________________________________
> >>>> > yocto mailing list
> >>>> > yocto@yoctoproject.org
> >>>> > https://lists.yoctoproject.org/listinfo/yocto
> >>>> >
> >>>>
> >>>> al block does and when its used. The above would then link into the
> >>> class reference, the variable glossary and so on. So its less about
> >>> graphics and more about giving Scott the information to create a
> >>> kind
> >>of
> >>> details index of some of these parts of the system like an expanded
> >>> table of contents.
> >>>
> >>> I've suggested a good start would be picking a few areas (like the
> >>> above) and trying to create the info and maybe see what kind of
> >>diagram
> >>> would present itself from that. If successful, it could then be
> >>expanded
> >>> to each area. I'd certainly hope that linux-yocto would be one area
> >>> covered.
> >>>
> >>> Cheers,
> >>>
> >>> Richard
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> yocto mailing list
> >>> yocto@yoctoproject.org
> >>> https://lists.yoctoproject.org/listinfo/yocto
> >>_______________________________________________
> >>yocto mailing list
> >>yocto@yoctoproject.org
> >>https://lists.yoctoproject.org/listinfo/yocto
> >_______________________________________________
> >yocto mailing list
> >yocto@yoctoproject.org
> >https://lists.yoctoproject.org/listinfo/yocto
> 
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto


If it is okay to jump in here, the keep it simple approach might be better.
Maybe walk through how a single package (hello world or other) is fetched,
expanded, packaged, placed in the image, etc. and show where everything goes
through the bitbake process.


Regards,

Sean Liming
Annabooks

Re: Documenting YP Development Environment in more detail

[Trevor Woerner]

I think this is an excellent project and am excited to see it coming together.

Are the meetings open for participation? I can't guarantee that I'd be
available when you run them, but I wouldn't mind listening in if I am.

If I were writing a book about Yocto/OE, in the first chapter I would
have the readers build their own filesystem/kernel from scratch (I
can't decide if I would also have them build their own cross-compiler
from scratch or if I'd cheat and let them use crosstool-NG). I think a
lot of the variables and tweaks involved in using Yocto/OE, and
certainly the steps Yocto/OE follows while doing its work, would be
more obvious to someone who has done what Yocto/OE is doing at least
once by hand.

I'm not suggesting any documentation effort should go down to the "do
it yourself from first principles at least once" level; but on the
other hand I think many people would understand Yocto/OE better if
they had that knowledge.

Re: Documenting YP Development Environment in more detail

[Rudolf Streif]

[-- Attachment #1: Type: text/plain, Size: 1558 bytes --]

Hi Trever et al:



> If I were writing a book about Yocto/OE,


This is a project I am currently working on, a book about the Yocto
Project. The goal is to enable readers to do practical projects with YP. As
the subject matter for the project described in the book I have chosen a
home automation project. Reason being, it interests me personally and it
uses different devices such as a UI-less controller, remotes with touch
screens etc.

I have gotten a lot of feedback from the YP training class I have developed
and been teaching for the Linux Foundation. I am putting this out here to
solicit more feedback from the community on what you think a book on YP
should include. For instance as an advanced topic I included a chapter on
how to run YP on AWS EC2 and I will be adding Autobuilder to it too.


> in the first chapter I would
> have the readers build their own filesystem/kernel from scratch (I
> can't decide if I would also have them build their own cross-compiler
> from scratch or if I'd cheat and let them use crosstool-NG).


I thought about that too but I found it distracting. It's like "let me show
you the hard way with crosstool-ng and buildroot and then I show you a
better way with YP." What I am doing though is an intro into Bitbake: how
to use just Bitbake to build something. That proved valuable during the
training classes. It's like a HelloWorld (and I published that before on
this mailing list) introducing the concepts of Bitbake with it's layers,
recipes, syntax etc.


Cheers,
Rudi

[-- Attachment #2: Type: text/html, Size: 2016 bytes --]

Re: Documenting YP Development Environment in more detail

[Bill Traynor]

On Mon, Jun 24, 2013 at 1:41 AM, Rudolf Streif
<rstreif@linuxfoundation.org> wrote:
> Hi Trever et al:
>
>
>>
>> If I were writing a book about Yocto/OE,
>
>
> This is a project I am currently working on, a book about the Yocto Project.
> The goal is to enable readers to do practical projects with YP. As the
> subject matter for the project described in the book I have chosen a home
> automation project. Reason being, it interests me personally and it uses
> different devices such as a UI-less controller, remotes with touch screens
> etc.
>
> I have gotten a lot of feedback from the YP training class I have developed
> and been teaching for the Linux Foundation. I am putting this out here to
> solicit more feedback from the community on what you think a book on YP
> should include. For instance as an advanced topic I included a chapter on
> how to run YP on AWS EC2 and I will be adding Autobuilder to it too.
>
>>
>> in the first chapter I would
>> have the readers build their own filesystem/kernel from scratch (I
>> can't decide if I would also have them build their own cross-compiler
>> from scratch or if I'd cheat and let them use crosstool-NG).
>
>
> I thought about that too but I found it distracting. It's like "let me show
> you the hard way with crosstool-ng and buildroot and then I show you a
> better way with YP." What I am doing though is an intro into Bitbake: how to
> use just Bitbake to build something. That proved valuable during the
> training classes. It's like a HelloWorld (and I published that before on
> this mailing list) introducing the concepts of Bitbake with it's layers,
> recipes, syntax etc.
>

FWIW, I'm incorporating a HelloWorld chapter into the new BitBake
manual.  It builds on your post of the same that you made to the
mailing list.

>
> Cheers,
> Rudi
>
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

Re: Documenting YP Development Environment in more detail

[Trevor Woerner]

Hi Rudi,

On 24 June 2013 01:41, Rudolf Streif <rstreif@linuxfoundation.org> wrote:
>> If I were writing a book about Yocto/OE,
> This is a project I am currently working on, a book about the Yocto Project.

Awesome! That's great news! I was hoping someone would "beat me to it"
:-) I look forward to reading it once it's done. Do you have a
publisher or an expected release date?

An appendix on Android's "repo" tool might make for an excellent
addition; it seems some Yocto projects are showing lots of interest in
using it.


>> in the first chapter I would
>> have the readers build their own filesystem/kernel from scratch
>
> I thought about that too but I found it distracting.

Fair enough. Maybe I'll write some blog-ish post about it for those
rare few who would like to see a bottom-up perspective :-)

Re: Documenting YP Development Environment in more detail

[Rudolf Streif]

[-- Attachment #1: Type: text/plain, Size: 860 bytes --]

Hi Trevor,


On Mon, Jun 24, 2013 at 5:56 AM, Trevor Woerner
<trevor.woerner@linaro.org>wrote:

> Awesome! That's great news! I was hoping someone would "beat me to it"
> :-) I look forward to reading it once it's done. Do you have a
> publisher or an expected release date?
>
> Yes, I do. Pearson and the release date is early next year (Embedded Linux
Conference is my target).


> An appendix on Android's "repo" tool might make for an excellent
> addition; it seems some Yocto projects are showing lots of interest in
> using it.
>
> The idea is to integrate/use it with YP? I know what repo does and that it
is built on top of git but I have not looked behind the scenes to figure
out how it does its job and how it can be used with something else like YP
instead of with the Android repositories and review system.

Cheers,
Rudi

[-- Attachment #2: Type: text/html, Size: 1398 bytes --]

Re: Documenting YP Development Environment in more detail

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 1062 bytes --]

On Tue, Jun 25, 2013 at 7:15 AM, Rudolf Streif
<rstreif@linuxfoundation.org>wrote:

>
>
>> An appendix on Android's "repo" tool might make for an excellent
>> addition; it seems some Yocto projects are showing lots of interest in
>> using it.
>>
>> The idea is to integrate/use it with YP? I know what repo does and that
> it is built on top of git but I have not looked behind the scenes to figure
> out how it does its job and how it can be used with something else like YP
> instead of with the Android repositories and review system.
>

not integrate with yocto directly. but use it to 'manage' the set of trees
that are required when using OE/Yocto. The gumstix guys are using it, see
[1], and we are moving toward it at linaro too, to manage our builds [2]

[1] https://github.com/gumstix/Gumstix-YoctoProject-Repo
[2] http://lists.linaro.org/pipermail/linaro-dev/2013-May/015998.html

then for organization that might need it, repo can be integrated with
gerrit too, but repo 'standalone' can still be used to manage set of trees.

[-- Attachment #2: Type: text/html, Size: 2049 bytes --]

Re: Documenting YP Development Environment in more detail

[Rudolf Streif]

[-- Attachment #1: Type: text/plain, Size: 579 bytes --]

> not integrate with yocto directly. but use it to 'manage' the set of trees
> that are required when using OE/Yocto. The gumstix guys are using it, see
> [1], and we are moving toward it at linaro too, to manage our builds [2]
>
> [1] https://github.com/gumstix/Gumstix-YoctoProject-Repo
> [2] http://lists.linaro.org/pipermail/linaro-dev/2013-May/015998.html
>
> then for organization that might need it, repo can be integrated with
> gerrit too, but repo 'standalone' can still be used to manage set of trees.
>
>
Thanks for the input. That's a good idea.

Rudi

[-- Attachment #2: Type: text/html, Size: 1279 bytes --]

Re: Documenting YP Development Environment in more detail

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 1369 bytes --]


On Jun 24, 2013, at 10:15 PM, Rudolf Streif <rstreif@linuxfoundation.org> wrote:

> Hi Trevor,
> 
> 
> On Mon, Jun 24, 2013 at 5:56 AM, Trevor Woerner <trevor.woerner@linaro.org> wrote:
> Awesome! That's great news! I was hoping someone would "beat me to it"
> :-) I look forward to reading it once it's done. Do you have a
> publisher or an expected release date?
> 
> Yes, I do. Pearson and the release date is early next year (Embedded Linux Conference is my target).
>  
> An appendix on Android's "repo" tool might make for an excellent
> addition; it seems some Yocto projects are showing lots of interest in
> using it.
> 
> The idea is to integrate/use it with YP? I know what repo does and that it is built on top of git but I have not looked behind the scenes to figure out how it does its job and how it can be used with something else like YP instead of with the Android repositories and review system. 
> 

We use android repo internally to manage the layers and I think there are projects out there like here

http://www.jann.cc/2012/08/08/building_freescale_s_community_yocto_bsp_for_the_olinuxino.html

which are also using android repo to manage layers.

> Cheers,
> Rudi 
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto


[-- Attachment #2: Type: text/html, Size: 2685 bytes --]

Re: Documenting YP Development Environment in more detail

[Nicolas Dechesne]

On Fri, Jun 28, 2013 at 10:02 PM, Khem Raj <raj.khem@gmail.com> wrote:
> We use android repo internally to manage the layers and I think there are
> projects out there like here
>
> http://www.jann.cc/2012/08/08/building_freescale_s_community_yocto_bsp_for_the_olinuxino.html
>
> which are also using android repo to manage layers.
>

indeed. gumstix too: https://github.com/gumstix/Gumstix-YoctoProject-Repo

and at Linaro we are moving all our OE jobs to use repo tool as well.

Re: Documenting YP Development Environment in more detail

[Trevor Woerner]

It looks like the community freescale work at github is using it too:
https://github.com/Freescale/fsl-community-bsp-platform

Re: Documenting YP Development Environment in more detail

[Trevor Woerner]

Coincidentally, Nicolas and I were recently involved in providing a
whirlwind tour of Yocto/OE to some developers last week.

On 20 June 2013 04:40, Rifenbark, Scott M <scott.m.rifenbark@intel.com> wrote:
> We can get started now on this by starting to define details for some obvious points or large areas of the flow.  What would be nice to get would be some graphical breakdowns of these areas of concern:
>
> * User-defined layers
> * YP provided layers
> * User configuration
> * Machine Configuration
> * Policy Configuration
> * Patches
> * YP provided recipes
> * User provided source code
> * YP provided source code
> * SCMs
> * Generated images
> * Generated SDKs/toolchains
> * Package Output
> * Source fetching process
> * Patch application process
> * Configuration process (fragments, etc.)
> * Key variable use and effects
> * User-initiated commands along the way


In addition to the above, excellent segments, I observed the following
based on my recent experience:

I think it would be fair to assume that most people reading this new
documentation didn't just install Linux for the first time a day or
two ago and are now wondering about Yocto. I think many people have
some sort of build experience. So I think it might be worth
considering having a "how do I ...?" section which attempts to cover
as many common questions/scenarios as possible. Perhaps it might be
interesting to directly compare Yocto/OE to other build systems (not,
of course, to try to get anyone's defenses up, but rather to be able
to answer questions such as "in build system X I do this to get that
result, how do I do the same thing in Yocto?"). Maybe these questions
and answers could be organized based on assuming a user is coming for
a specific other build system? Or perhaps have sections explaining why
certain procedures which are required in other build systems aren't
required in Yocto.

For example: some build systems require you to restart your entire
build process if there is a failure anywhere during this process. So
one of the questions we got was "is bitbake smart enough to restart
the build at the point of failure if there is an error, or does the
build have to be started from scratch every time?".

To some people it might seem bizarre that Yocto generates packages
(e.g. rpms or debs) when (by default) the core-image-minimal target
didn't have any package management tools. It took us a while to
clarify that Yocto is building these (e.g.) rpms by default as part of
its own internal process. The fact package management tools can be
installed with a very simple configuration file change and then these
rpms or debs can be used to update a running image is just an extra
bonus; but you're getting these (e.g.) rpms whether you want them or
not :-) Stranger still: choosing rpms instead of debs doesn't mean
your final system is going to look a lot more like Fedora than Debian.

Tied closely to the above, lots of clarity was required to explain
that Yocto builds each recipe individually, installs it (somewhere),
uses the files in "somewhere" to generate a set of packages, and then
uses the selected packages (i.e. the -doc, -dev, -dbg, ...) to create
the target's root filesystem image. I think the multiple "install"
stages was not intuitive. I think most people by default would simply
install the build output from each component directly into a sysroot
without the indirection of creating a set of packages which are then
selected from to make up the final image. Obviously what Yocto is
doing is brilliant and correct, it's just not obvious as to why it
would do things that way (I think).

As I mentioned in my other comments, sometimes the best way to explain
why Yocto is doing things the way it is can be best understood once a
person has actually tried to do these things by hand. Polluting the
sysroot during the build would be quite obviously wrong if/when a new
version of a component were required, or you wanted to remove a given
thing from the output image.

Re: Documenting YP Development Environment in more detail

[Trevor Woerner]

Has anyone else heard of vimcasts.org?

Short, concise, detailed screen-casts of specific Yocto tasks coupled
with "show notes", I think, would be an awesome other way of
delivering this information.

yoctocasts.org anyone?

Re: Documenting YP Development Environment in more detail

[Trevor Woerner]

On 23 June 2013 00:01, Trevor Woerner <trevor.woerner@linaro.org> wrote:
> Has anyone else heard of vimcasts.org?

    http://vimcasts.org/episodes/archive

is a good example of what I'm thinking about.