* RFC: Yocto BSP and kernel usability tools
@ 2011-12-19 16:55 Tom Zanussi
2011-12-22 20:16 ` Bruce Ashfield
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Tom Zanussi @ 2011-12-19 16:55 UTC (permalink / raw)
To: yocto@yoctoproject.org
Hi,
This is what I hope to implement in the M2 timeframe.
Any and all comments welcome...
Thanks,
Tom
---
Getting Yocto running on a new board is often a prerequisite task to
the real task of interest for new users, which is creating new
applications designed to run on the board, or creating new
board-specific embedded systems. In other words, most users just want
to get the hardware up and running and not have to think too much
about it before moving on to what they're really interested in, which
is the stuff running on top of it.
While there really can't be any such thing as a magic-bullet
application that would do that automatically i.e. generate and
maintain a guaranteed-working Yocto BSP for any given piece of
hardware, it should at least be possible to provide a tool that would
generate a base BSP usable as a starting point for that, and which
would additionally allow the user to tweak various BSP settings,
including kernel configuration options and simple kernel patches.
Such a tool should be usable with only a minimal understanding of the
Yocto build system and metadata, and with no knowledge at all of
kernel internals or the Yocto kernel recipe bindings, or of the
details of the internals of yocto kernel repositories, or in fact any
knowledge of git or any other specific build-related tool.
This document provides the design for such a tool. At a high level,
the goal of the tool is to provide a user the means of creating a new
Yocto BSP from which it should be possible (assuming a perfect
run-time outcome, admittedly unlikely on a first pass) to submit a
'pull request' for inclusion into any repo(s) that would accept an
'official' Yocto BSP. For example, the output of the tool should
produce metadata and kernel configuration that would be directly
mergeable into the meta-intel and the linux-yocto kernel repos. The
tool should also allow the BSP maintainer to afterwards at any time
make changes to the kernel configuration and/or submit kernel patches
for the BSP.
Note that although knowledge of the Yocto kernel or kernel internals
is not expected, the user is expected to know the basics of how to
deal with the linux kernel at the 'user' level, specifically
understanding and determining the specific kernel config options
needed for the user's BSP.
Specifically, the tool will allow for the following high-level tasks,
which are discussed in more detail below:
- create a new BSP from scratch from a set of user-settable parameters
- manage individual kernel config settings
- create and use groups of kernel config settings as KERNEL_FEATUREs
- apply patches to a BSP's machine branch
The following capabilities are specifically not provided by the tool:
- help in determining a 'correct' set of kernel config options to set.
The tool assumes the user already knows that, either through diffs
of .config files before and after using 'bitbake -c menuconfig', or
some other means.
- modification of the BSP metadata after BSP creation. The tool
provides an initial 'write-once' BSP-generation capability, but
doesn't allow it to be read in and modified after the initial pass.
The user will have to make further modifications manually. The one
exception to this is kernel features, which can be added to an
already existing kernel bbappend file.
- a guarantee or even expectation that the generated BSP will work on
the actual hardware it's targeted for - it's highly unlikely the BSP
will work 'out-of-the-box' and it's the developer's reponsibility to
do the often hard work of figuring out what settings and/or code are
actually needed to get the hardware to work as expected. The goal
of the tool is to make that job easier, not to actually do that job.
Also, though it's not explicitly a requirement of this tool, the
design should be sufficiently modular to allow for its participation
in generic logical pipelines. For instance, while the tool may
present to the user a text-based UI for gathering information, it
should also be able to operate without any kind of user interaction
and retrieve the data it needs from a file, for instance e.g.:
$ yocto-bsp create < x86-input.json
Another example that the design should not preclude would be its use
at the end of a bsp-import pipeline e.g.:
$ cat 3rdparty.bsp | 3rdparty2yocto | yocto-bsp create
The tool will initially be implemented as a set of command-line tools
which will essentially be thin layers on top of a modular Python API;
future tools may be GUI-based but would make use of the functionality
exposed through the same API. There might also be an opportunity for
future integration into existing tools such as hob; this isn't
explicitly catered for in the current design, but it should be
possible to do that based either on the functionality provided by the
command-line interface or equivalently via the Python module API.
A Quick Overview
----------------
The initial set of tools will comprise 'yocto-bsp', for creating Yocto
BSPs, and 'yocto-kernel', for BSP and BSP-related Yocto kernel
maintenance tasks on existing BSPs.
The interface and detailed description of each tool are described
below, and the underlying templating mechanism is described following
that, but it would be good for context to first try to give an overall
picture of how the tools were designed to work and be used together.
The envisioned usage model is that a user would first start out by
creating a new BSP for a given architecture by invoking the 'yocto-bsp
create' command. Choosing the architecture essentially chooses a set
of files that the BSP will be based on. This set of files is a
combination of a set of 'common' files that all BSPs inherit and a set
of architecture-specific files that match the specified architecture.
Neither set of files can be used unmodified on their own as simple
static copies of the originals in order to create a working BSP for a
real machine - many of those files need to be tailored for the
hardware using information supplied by the user. To accomplish this
in an automated way, both the specific portions of the BSP files that
need to be filled in with user input, and the means of (optionally)
getting that input from the user are included as in-line annotations
to the BSP files themselves using a simple templating scheme based on
conventions used by web2py. Essentially, in the default case, the
templating system is used to generate a series of text-based prompts
that ask the user questions about the BSP being created. The data
supplied by the user is then used to fill in the missing information
in the BSP files. The user can also choose to specify all the
information needed directly to the 'yocto-bsp create' command
invocation itself; the 'yocto-bsp list' command exists to provide the
user the list of 'properties' and enumerations of possible values for
those properties, which taken together represent the data items needed
to generate the BSP. The user can even more directly supply the
information as a json object representing a serialized python
dictionary, which is what the templating interface uses natively to
represent properties and values needed for BSP generation.
The end result is a Yocto-compliant BSP that should be directly
buildable by the Yocto build system. The generated BSP is completely
self-contained - it includes not only the BSP metadata, but also the
necessary kernel components, which are normally for BSPs contained in
the linux-yocto repo's meta and machine branches for the BSP. In
order to help make the tool more efficient for the user, the tool does
everything it needs to locally by initially using the linux-yocto
SRC_URI; once ready, the user can then migrate the kernel components
to an actual linux-yocto repo by using the 'yocto-kernel migrate'
command. Once that's done, the user is in a position to submit both
the BSP metadata and the linux-yocto commits upstream.
Once a BSP is created, the job of 'yocto-bsp' is done (in the initial
version of the tools - it may have sub-commands added later that could
possibly manage the non-kernel aspects of a BSP) and the
'yocto-kernel' command can be used going forward to maintain the
kernel portion of a BSP. 'yocto-kernel' allows both bare kernel
config items and higher-level 'kernel features' to be created, used
and removed, patches to be added to a BSP's machine branch, and the
kernel changes to a BSP migrated to a linux-yocto repo. These
have typically been difficult tasks for many users; the hope is that
'yocto-kernel' will make those tasks more manageable for everyone.
The sections below describe each tool in detail.
Following those sections is a detailed description of the templating
mechanism, useful for anyone who needs to create templates for
creating new BSPs using the 'yocto-bsp create' tool.
The top-level bsp-creation tool, 'yocto-bsp'
============================================
All of the functionality of the Yocto bsp-creation tool is available
via the single 'yocto-bsp' command. Access to the specific functions
provided by the tool is accomplished by specifying sub-commands to the
main 'yocto-bsp' command. The design of the command-line interface is
based on similar interfaces provided by git and perf, which is
familiar to many users and is easily extensible.
Usage help is always available for the command itself as well as for
all sub-commands. Executing the top-level command displays a list of
the available sub-commands, with descriptions:
$ yocto-bsp
usage: yocto-bsp [--version] [--help] COMMAND [ARGS]
The most commonly used 'yocto-bsp' commands are:
create Create a new Yocto BSP
list List available values for options and BSP properties
See 'yocto-bsp help COMMAND' for more information on a specific command.
'yocto-bsp create' - Create a new Yocto BSP
-------------------------------------------
NAME
yocto-bsp-create - Create a new Yocto BSP
SYNOPSIS
yocto-bsp create <bsp-name> <karch> <kmachine>
[--property <NAME:VAL> | -p <NAME:VAL>]
DESCRIPTION
This command creates a Yocto BSP based on the specified
parameters. The new BSP will be a new Yocto BSP layer contained
within the top-level directory specified as bsp-name i.e. not
'meta-bsp-name'. The value of the 'karch' parameter determines
the set of files that will be generated for the BSP, along with
the specific set of 'properties' that will be used to fill out the
BSP-specific portions of the BSP.
The BSP-specific properties that define the values that will be
used to generate a particular BSP can be specified on the
comman-line using the --property option for each property, which
is a colon-separated name:value pair.
The set of properties and enumerations of their possible values
can be displayed using the 'yocto-bsp list' command.
It isn't necessary however for the user to explicitly specify the
property values using --property options - if none are specified
in the 'yocto-bsp create' invocation, the user will be
interactively prompted for each of the required property values,
which will then be used as values for BSP generation.
The 'kmachine' parameter names the 'machine branch' that the BSP's
machine branch will be based on; the list of branches meaningful
for that purpose can also be listed using the 'yocto-bsp list'
command.
The 'yocto-bsp list' command can also be used to list the possible
values for the 'karch' parameter.
An typical example that would query the user for property values
would be:
$ yocto-bsp create meta-foo i386 yocto/standard/common-pc/foo
Because the user hasn't specified any values for the BSP
properties, yocto-bsp will query the user for each of the
unspecified property values required to create the BSP i.e. the
properties listed by 'yocto-bsp list properties'.
Here's an example using explicitly-specified properties:
$ yocto-bsp create meta-foo i386 yocto/standard/common-pc/foo
--property tunefile:tune-atom --property touchscreen:true
--property kfeature:cfg/smp [...]
'yocto-bsp list' - List available values for options and BSP properties
-----------------------------------------------------------------------
NAME
yocto-bsp-list - List available values for options and BSP properties
SYNOPSIS
yocto bsp list <karch | kmachine>
yocto bsp list <karch> properties
yocto bsp list <karch> property <xxx>
DESCRIPTION
This command enumerates the complete set of possible values for a
specified option or property needed by the BSP creation process.
The first form enumerates all the possible values that exist and
can be specified for the 'karch' or 'kmachine' parameters to the
'yocto bsp create' command. Example output for those options:
$ yocto-bsp list karch
i386
x86_64
arm
powerpc
mips
$ yocto-bsp list kmachine
yocto
yocto/standard
yocto/standard/common-pc
yocto/standard/common-pc-64
yocto/standard/preempt-rt
The second form enumerates all the possible properties that exist
and must have values specified for them in the 'yocto bsp create'
command for the given 'karch'. Example output for that command:
$ yocto-bsp list i386 properties
"machine": boolean,
"kmachine": choice,
"tunefile": choicelist,
"smp": boolean,
"xserver": choicelist
["xserver-choice": choice]
"kfeatures": choicelist
["kfeature": choice]
"touchscreen": boolean,
"keyboard": boolean
The third form enumerates all the possible values that exist and
can be specified for any of the enumerable properties of the given
'karch' in the 'yocto bsp create' command. Example output for
those properties:
$ yocto-bsp list i386 property xserver
xserver-vesa
xserver-i915
xserver-emgd
$ yocto-bsp list i386 property kfeatures
features/taskstats
cfg/sound
$ yocto-bsp list i386 property tunefile
i386
tune-atom
tune-core2
$ yocto-bsp list x86_64 property tunefile
x86_64
tune-x86_64
tune-corei7
tune-ivb
$ yocto-bsp list powerpc property tunefile
powerpc
tune-ppc603e
tune-ppce300c2
tune-ppce500
tune-ppce500mc
tune-ppce500v2
tune-ppce5500-32b
tune-ppce5500-64b
$ yocto-bsp list arm property tunefile
arm
tune-arm1136jf-s
tune-arm920t
tune-arm926ejs
tune-arm9tdmi
tune-armv7
$ yocto-bsp list mips property tunefile
mips
tune-mips32
The top-level kernel usability tool, 'yocto-kernel'
===================================================
All of the functionality of the Yocto kernel usability tool is
available via the single 'yocto-kernel' command. Access to the
specific functions provided by the tool is accomplished by specifying
sub-commands to the main 'yocto-kernel' command. The design of the
command-line interface is based on similar interfaces provided by git
and perf, which is familiar to many users and is easily extensible.
Usage help is always available for the command itself as well as for
all sub-commands. Executing the top-level command displays a list of
the available sub-commands, with descriptions:
$ yocto-kernel
usage: yocto-kernel [--version] [--help] COMMAND [ARGS]
The most commonly used 'yocto-kernel' commands are:
feature-list List available Yocto KERNEL_FEATUREs
feature-define Define a new Yocto KERNEL_FEATURE
feature-add Add Yocto KERNEL_FEATUREs to a BSP
feature-rm Remove Yocto KERNEL_FEATUREs from a BSP
config-list List the modifiable set of bare kernel config options for a BSP
config-add Add or modify bare kernel config options for a BSP
config-rm Remove bare kernel config options from a BSP
patch-add Patch the Yocto kernel for a specific BSP
migrate Move local kernel config/patches to a git repo
See 'yocto-kernel help COMMAND' for more information on a specific command.
'yocto-kernel feature list' - List available Yocto KERNEL_FEATUREs
------------------------------------------------------------------
NAME
yocto-kernel-feature-list - List available Yocto KERNEL_FEATUREs
SYNOPSIS
yocto-kernel feature list <bsp-name> --used
DESCRIPTION
This command lists all the kernel features available to a BSP.
This includes any features temporarily attached to the named BSP
via the 'feature define' command and thus the reason this command
takes bsp-name as a parameter.
If the --used param is specified, only the KERNEL_FEATURES used by
the BSP are listed.
'yocto-kernel feature define' - Define a new Yocto KERNEL_FEATURE
-----------------------------------------------------------------
NAME
yocto-kernel-feature-define - Define a new Yocto KERNEL_FEATURE
SYNOPSIS
yocto-kernel feature define <bsp-name> <feature-name> <feature-desc>
[<CONFIG_XXX=x> ...] [--dirname <dirname>] [--policy <hw|non-hw>]
DESCRIPTION
This command defines a new kernel feature, resulting in the
addition of two new files for feature xxx to the named BSP's
SRC_URI. If dirname is specified, the feature files will be named
features_dirname_xxx.cfg and features_dirname_xxx.scc. Otherwise,
the features will be named cfg_xxx.cfg and cfg_xxx.scc. The idea
behind the naming is that the names reflect the user's intent to
have the feature migrated to either the kernel-cache/cfg or
kernel-cache/features directory if/when the kernel portion of the
BSP is migrated into the linux-yocto kernel repo.
The policy stating whether the new feature is considered to be a
hardware or a non-hardware feature also needs to be specified in
order to create a new feature, and a description of the new
feature is also required; this will be added as a comment to the
.scc file.
NOTE: Although features are not BSP-specific, until the user has
gone through the steps of migrating a feature to the linux-yocto
repo, it needs to live somewhere accessible to the BSP metadata,
which logically would be the kernel SRC_URI for the BSP using it.
NOTE: It's up to the user to determine whether or not the config
options for a feature being added make sense or not - this command
does no sanity checking or verification of any kind to ensure that
the config options contained in a kernel feature really make sense
and will actually be set in in the final config. For example, if
a config option depends on other config options, it will be turned
off by kconfig if the other options aren't set correctly.
'yocto-kernel feature add' - Add Yocto KERNEL_FEATUREs to a BSP
---------------------------------------------------------------
NAME
yocto-kernel-feature-add - Add Yocto KERNEL_FEATUREs to a BSP
SYNOPSIS
yocto-kernel feature add <bsp-name> [<FEATURE> ...]
DESCRIPTION
This command adds one or more kernel features to a BSP. The set
of features available to be added by this command for a BSP can be
found via the 'yocto-kernel feature list' command. This command
essentially adds a new machine-specific KERNEL_FEATURE_append to
the linux-yocto .bbappend for the BSP.
'yocto-kernel feature rm' - Remove Yocto KERNEL_FEATUREs from a BSP
-------------------------------------------------------------------
NAME
yocto-kernel-feature-rm - Remove Yocto KERNEL_FEATUREs from a BSP
SYNOPSIS
yocto-kernel feature rm <bsp-name> [<FEATURE> ...]
DESCRIPTION
This command removes one or more kernel features from a BSP. The
set of features available to be removed by this command for a BSP
can be found via the 'yocto-kernel feature list' command, if the
--used option to that command is specified. This command
essentially removes a machine-specific KERNEL_FEATURE from the
linux-yocto .bbappend for the BSP.
'yocto-kernel config list' - List a BSP's modifiable config options
-------------------------------------------------------------------
NAME
yocto-kernel-config-list - List the modifiable set of bare kernel
config options for a BSP
SYNOPSIS
yocto-kernel config list <bsp-name>
DESCRIPTION
This command lists the 'modifiable' config items for a BSP
i.e. the items which are eligible for modification or removal by
other yocto-kernel commands.
'modifiable' config items are the config items contained a BSP's
foo.cfg base config.
'yocto-kernel config add' - Add or modify a BSP's kernel config options
-----------------------------------------------------------------------
NAME
yocto-kernel-config-add - Add or modify bare kernel config options
for a BSP
SYNOPSIS
yocto-kernel config add <bsp-name> [<CONFIG_XXX=x> ...]
DESCRIPTION
This command adds one or more CONFIG_XXX=x items to a BSP's
foo.cfg base config. If a config item is already present, the new
value will simply overwrite the old value.
NOTE: It's up to the user to determine whether or not the config
options being added make sense or not - this command does no
sanity checking or verification of any kind to ensure that a
config option really makes sense and will actually be set in in
the final config. For example, if a config option depends on
other config options, it will be turned off by kconfig if the
other options aren't set correctly.
'yocto-kernel config rm' - Remove BSP kernel config options
-----------------------------------------------------------
NAME
yocto-kernel-config-rm - Remove bare kernel config options from a
BSP
SYNOPSIS
yocto-kernel config rm <bsp-name> [<CONFIG_XXX> ...]
DESCRIPTION
This command removes (turns off) one or more CONFIG_XXX items to a
BSP's foo.cfg base config.
The set of config items available to be removed by this command
for a BSP can be found via the 'yocto-kernel config list'
command.
'yocto-kernel patch add' - Patch the Yocto kernel for a specific BSP
--------------------------------------------------------------------
NAME
yocto-kernel-patch-add - Patch the Yocto kernel for a specific BSP
SYNOPSIS
yocto-kernel patch add <bsp-name> [<PATCH> ...]
DESCRIPTION
This command adds one or more patches to a BSP's machine branch.
The patch will be added to the BSP's linux-yocto kernel SRC_URI
and will be guaranteed to be applied in the order specified.
NOTE: It's up to the user to determine whether or not the patches
being added makes sense or not - this command does no sanity
checking or verification of any kind to ensure that a patch can
actually be applied to the BSP's kernel branch; it's assumed that
the user has already done that.
'yocto-kernel migrate' - Move local kernel config/patches to a git repo
-----------------------------------------------------------------------
NAME
yocto-kernel-migrate - Move local kernel config/patches to a git repo
SYNOPSIS
yocto-kernel migrate <bsp-name> <local linux-yocto repo>
DESCRIPTION
This command converts the kernel configuration changes, patches,
and features collected in a BSP's linux-yocto SRC_URI into an
equivalent set of commits in a local linux-yocto repo. On
success, the local files will named in the SRC_URI will be
removed, and the linux-yocto .bbappend will be fixed up to point
to the new repo.
NOTE: The user is responsible for submitting the generated commits
upstream and subsequently modifying the kernel .bbappend to
use the upsteam kernel repo.
The design of the 'yocto-bsp create' templating mechanism
==========================================================
This section describes the details of the templating mechanism used by
'yocto-bsp' to generate BSPs.
Generating a new BSP is basically a templating exercise. A given BSP
is comprised of a combination of a set of common files that all BSPs
share along with a set of files specific to a given BSP type. Many of
the files in both sets need to be configured according to the needs of
the BSP; this also extends to the naming and existence of the files
themselves.
The templating scheme used in the Yocto tool is partly inspired by
ideas from web2py, but adapted to a more general input mechanism for
the specific problems posed by Yocto.
There are no special files, just marked-up BSP metadata and kernel
configuration, .scc, and patch files.
The Yocto BSP markup is meant to be extremely simple and limited to
pretty much this initial set of items; it is however designed to be
extensible enough to cover any future need.
Anything within double braces is considered to be a Yocto BSP markup
tag:
{{ foo }}
Anything within double braces and starting with {{= is considered to
implement simple variable replacement, where variables are by default
simply keys in a dictionary representing the BSP and containing values
for the whole BSP:
{{=machine}}
The above means that the whole expression will be replaced with the
value of the 'machine' entry in the BSP dictionary. The syntax is
actually shorthand for the equivalent Python code for accessing the
value of a given key, but isn't restricted to that e.g. it could
reference a local variable defined in a previous tag, for instance.
In every other case, the 'yocto-bsp' tool will pre-process the
contents looking for certain keywords at the beginning. The rest it
will treat as Python code. As of now, there's only one keyword that's
meaningful:
{{ input }}
which signals that anything contained is input-related. The
'yocto-bsp' implementation will use this information to prompt the
user for input. Under certain conditions e.g. when the user has
already supplied this information via a file for instance, anything
contained in this is considered a no-op and ignored.
Anything else is considered to be Python code and will be directly
incorporated into the BSP generation code.
{{ if keyboard == False: }}
Note also that any of the above can also be used in file and directory
names, which will also be processed in the same way; this allows us to
avoid extraneous files is consistent with the rest of the scheme.
Internally, the way things are envisioned to work is that the tool
will first read in the set of files that makes up the BSP for the
chosen architecture. It will then pre-process all the 'input' tags it
finds and use that to prepare a series of text-based prompts to
retrieve the needed information from the user. The end result of
presenting all that to the user will be a completd BSP dictionary
containing all the key/value pairs needed to fill in the BSP. Once
that information is available, it will make a second pass over the
remaining tags and fill them in with the Python code needed to
retrieve and replace the remaining tags with their corresponding
values from the BSP dictionary (or any other source possibly
introduced by other tags). The end result of this second pass will be
executed as Python code to produce the final BSP metadata.
For the first version, the 'input' tags will be used to construct a
series of text-based questions for the user. The design, however,
should make it relatively trivial to create an application that could
create real GUI that would do the same thing at some later point. For
example, such an application could use the input tags to generate a
web-based form, which would return the results gathered from the user
to a web service in the form of a json blob that could directly be
converted into the BSP Python dictionary needed. e.g. the browser
would return something like this to the web service, which would in
turn send it to the 'bsp-create' library function to create the final
BSP.
{
"machine": "foo",
"kmachine": "/yocto/standard/base",
"tunefile": "tune-atom",
"smp": true,
"xserver": {
"xserver-choice": "xserver-i915",
},
"kfeatures": [
"kfeature": "features/taskstats",
"kfeature": "cfg/sound",
]
},
touchscreen: false,
keyboard: true
}
Below are some concrete examples of specific tags and how they're used.
Here's a simple example of markup that describes information needed
from the user about touchscreen and keyboard capabilities in the
formfactor 'machconfig' file:
# Assume a USB mouse and keyboard are connected
{{ input type:"boolean" name:"touchscreen" msg:"Does your BSP have a touchscreen?" }}
HAVE_TOUCHSCREEN={{=touchscreen}}
{{ input type:"boolean" name:"keyboard" msg:"Does your BSP have a keyboard?" }}
HAVE_KEYBOARD={{=keyboard}}
For instance, the touchscreen 'input' tag above describes everything a
UI would need in order to retrieve an answer from the user as to
whether the hardware in question has a touchscreen or not.
The 'type' key comprises a small set of possible input types that
include things like 'boolean' (yes/no) choices and lists of choices
(type:"choicelist").
The 'name' key provides a means for later markup to access the value
of the variable containing the user's answer for that input. For
example, the name:"touchscreen" simply references the value in the BSP
dictionary that was filled in with the user's response to the question
which was formed and presented to the user using the "msg" tag in the
input tag.
Below the input tag is the actual content that will be rendered into
the 'machconfig' file, but with the {{=}} portion replaced with the
'touchscreen' value supplied by the user i.e.:
HAVE_TOUCHSCREEN={{=touchscreen}}
will be rendered as:
HAVE_TOUCHSCREEN=0
Assuming the user answered 'no'.
Another interesting thing to note about this example is that in Yocto
BSPs, the machconfig files are contained within a filesystem location
that includes the machine name in the path e.g.
recipes-bsp/formfactor/formfactor/foo/machconfig
To accomplish this, the directory names themselves contain markup that
gets processed and replaced by the tool:
recipes-bsp/formfactor/formfactor/{{=machine}}/machconfig
Here, the {{=machine} directory is renamed to /foo to match the
machine name of the BSP.
Here's a more involved example, markup included in the
i386/conf/{{=machine}}.conf file:
{{ input type:"boolean" name:"xserver" msg:"Do you need support for X?" }}
{{ if xserver: }}
XSERVER ?= "${XSERVER_IA32_BASE} \
${XSERVER_IA32_EXT} \
{{ input type:"choicelist" name:"xserver_choice" msg:"Please select an xserver for this machine" }}
{{ input type:"choice" val:"xserver-vesa" msg:"VESA xserver support" }}
${XSERVER_IA32_VESA} \
{{ input type:"choice" val:"xserver-emgd" msg:"EMGD xserver support (proprietary)" }}
${XSERVER_IA32_EMGD} \
{{ input type:"choice" val:"xserver-i915" desc:"i915 xserver support" }}
${XSERVER_IA32_I915} \
"
In this case, we start off with a boolean 'input' question asking if
the BSP needs to support X. The following line is straight Python
code which essentially skips the following sections (until a blank
line) if the user's answer was 'no', or if the answer was 'yes',
creates a further query to the user asking which xserver type to use.
In the latter case, the input type is a list of choices resulting in
the values specified by the 'val' tag appearing as the value for the
'xserver_choice' dictionary key.
Code further down in the .conf file can also make use of this
information to include or exclude content based on the value of the
xserver_choice variable:
{{ if xserver_choice == "xserver-emgd": }}
PREFERRED_VERSION_xserver-xorg ?= "1.9.3"
PREFERRED_VERSION_mesa-dri ?= "7.11"
PREFERRED_VERSION_emgd-driver-bin ?= "1.8"
{{ if xserver_choice == "xserver-vesa" or xserver_choice == "xserver-emgd": }}
APPEND += "video=vesafb vga=0x318"
Note again that the tags used to determine whether or not to include
content simply contain Python code used to directly implement the
needed logic. Not only is there no need for the user to learn a
special-purpose language, but gives the tool the flexibility to add
essentially any kind of processing logic needed in the future.
In the case of xserver choices, we also need to use the value of
xserver_choice to determine which xorg.conf file to add to the BSP.
Again, consistent with the rest of the markup processing, this is
seamlessly accomplished by adding markup to the filenames themselves:
recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver_choice == xserver_emgd: }} xorg.conf
recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver_choice == xserver_vesa: }} xorg.conf
recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver_choice == xserver_i915: }} xorg.conf
In this case, the filenames actually contain Python code that
describes the conditions needed to include those particular files.
Here's another example, again making use of general-purpose Python in
the markup to generate an open-ended list of choices for the user, in
this case in linux-yocto_3.0.bbappend, to generate a list of available
KERNEL_FEATURES for the user to choose from:
{{ input type:"checklist" name:"kfeatures" gen:"fetch-kernel-features()" msg:"Please choose the kernel features to use in this BSP on" }}
{{ for kfeature in kfeatures: }}
KERNEL_FEATURES_append_{{=machine}} += {{=kfeature}}
In this case an 'input' tag calls an internally-defined yocto-bsp
function named fetch_kernel_feature() that generates the list of
available KERNEL_FEATURES. The output of that call is a Python list
containing the features, which is assigned to the 'kfeatures'
variable. The next tag contains Python code that simply iterates over
that list, and uses the value of the current feature, 'kfeature', to
append the chosen kernel features to the list of features used by the
BSP.
Finally, because the kernel components are actually part of the BSP
metadata, the user-specified values are also available in the
processing of those files. For instance, in the below case, we can
use the xserver_choice to include or exclude specific kernel
configuration settings in the
recipes-kernel/linux/linux-yocto/{{=machine}}.scc file associated with
the BSP:
{{ if xserver_choice == "xserver-emgd": }}
include features/drm-emgd/drm-emgd.scc
---
For reference, here's an example of how the tools are expected to be
laid out. Everything is contained within a 'bsp-tools' directory
under '/scripts'. The 'common' portion, containing the files common
to every BSP are:
bsp-tools/
bsp-tools/yocto-bsp.py
bsp-tools/yocto-kernel.py
bsp-tools/bsp-cache
bsp-tools/bsp-cache/target
bsp-tools/bsp-cache/target/arch
bsp-tools/bsp-cache/target/arch/common
bsp-tools/bsp-cache/target/arch/common/README.sources
bsp-tools/bsp-cache/target/arch/common/binary
bsp-tools/bsp-cache/target/arch/common/binary/.gitignore
bsp-tools/bsp-cache/target/arch/common/recipes-kernel
bsp-tools/bsp-cache/target/arch/common/recipes-kernel/linux
bsp-tools/bsp-cache/target/arch/common/COPYING.MIT
bsp-tools/bsp-cache/target/arch/common/README
bsp-tools/bsp-cache/target/arch/common/recipes-graphics
bsp-tools/bsp-cache/target/arch/common/recipes-graphics/xorg-xserver
bsp-tools/bsp-cache/target/arch/common/recipes-graphics/xorg-xserver/{{ if xserver: }} xserver-xf86-config_0.1.bbappend
bsp-tools/bsp-cache/target/arch/common/conf
bsp-tools/bsp-cache/target/arch/common/conf/layer.conf
bsp-tools/bsp-cache/target/arch/common/recipes-core
bsp-tools/bsp-cache/target/arch/common/recipes-core/tasks
bsp-tools/bsp-cache/target/arch/common/recipes-core/tasks/task-core-tools.bbappend
bsp-tools/bsp-cache/target/arch/common/recipes-bsp
bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor
bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor_0.0.bbappend
bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor
bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor/{{=machine}}
bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor/{{=machine}}/machconfig
The BSP-specific files are contained in directories named for the
architecture. In this case, only the i386 files are expanded as an
example.
bsp-tools/bsp-cache/target/arch/arm
bsp-tools/bsp-cache/target/arch/powerpc
bsp-tools/bsp-cache/target/arch/i386
bsp-tools/bsp-cache/target/arch/i386/recipes-kernel
bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux
bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt_3.0.bbappend
bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto_3.0.bbappend
bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto
bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}.cfg
bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}.scc
bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}-preempt-rt.scc
bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}-standard.scc
bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt
bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}.cfg
bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}.scc
bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}-preempt-rt.scc
bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}-standard.scc
bsp-tools/bsp-cache/target/arch/i386/recipes-graphics
bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config
bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}
bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver-choice == xserver-emgd: }} xorg.conf
bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver-choice == xserver-vesa: }} xorg.conf
bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver-choice == xserver-i915: }} xorg.conf
bsp-tools/bsp-cache/target/arch/i386/conf
bsp-tools/bsp-cache/target/arch/i386/conf/machine
bsp-tools/bsp-cache/target/arch/i386/conf/machine/{{=machine}}.conf
bsp-tools/bsp-cache/target/arch/mips
bsp-tools/bsp-cache/target/arch/x86_64
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: RFC: Yocto BSP and kernel usability tools
2011-12-19 16:55 RFC: Yocto BSP and kernel usability tools Tom Zanussi
@ 2011-12-22 20:16 ` Bruce Ashfield
2011-12-22 21:31 ` Tom Zanussi
2011-12-28 0:36 ` McClintock Matthew-B29882
2012-01-31 22:52 ` Darren Hart
2 siblings, 1 reply; 7+ messages in thread
From: Bruce Ashfield @ 2011-12-22 20:16 UTC (permalink / raw)
To: Tom Zanussi; +Cc: yocto@yoctoproject.org
On 11-12-19 11:55 AM, Tom Zanussi wrote:
> Hi,
>
> This is what I hope to implement in the M2 timeframe.
>
> Any and all comments welcome...
>
> Thanks,
>
> Tom
>
> ---
>
> Getting Yocto running on a new board is often a prerequisite task to
> the real task of interest for new users, which is creating new
> applications designed to run on the board, or creating new
> board-specific embedded systems. In other words, most users just want
> to get the hardware up and running and not have to think too much
> about it before moving on to what they're really interested in, which
> is the stuff running on top of it.
>
> While there really can't be any such thing as a magic-bullet
> application that would do that automatically i.e. generate and
> maintain a guaranteed-working Yocto BSP for any given piece of
> hardware, it should at least be possible to provide a tool that would
> generate a base BSP usable as a starting point for that, and which
> would additionally allow the user to tweak various BSP settings,
> including kernel configuration options and simple kernel patches.
> Such a tool should be usable with only a minimal understanding of the
> Yocto build system and metadata, and with no knowledge at all of
> kernel internals or the Yocto kernel recipe bindings, or of the
> details of the internals of yocto kernel repositories, or in fact any
> knowledge of git or any other specific build-related tool.
>
> This document provides the design for such a tool. At a high level,
> the goal of the tool is to provide a user the means of creating a new
> Yocto BSP from which it should be possible (assuming a perfect
> run-time outcome, admittedly unlikely on a first pass) to submit a
> 'pull request' for inclusion into any repo(s) that would accept an
> 'official' Yocto BSP. For example, the output of the tool should
> produce metadata and kernel configuration that would be directly
> mergeable into the meta-intel and the linux-yocto kernel repos. The
> tool should also allow the BSP maintainer to afterwards at any time
> make changes to the kernel configuration and/or submit kernel patches
> for the BSP.
>
> Note that although knowledge of the Yocto kernel or kernel internals
> is not expected, the user is expected to know the basics of how to
> deal with the linux kernel at the 'user' level, specifically
> understanding and determining the specific kernel config options
> needed for the user's BSP.
>
> Specifically, the tool will allow for the following high-level tasks,
> which are discussed in more detail below:
>
> - create a new BSP from scratch from a set of user-settable parameters
> - manage individual kernel config settings
> - create and use groups of kernel config settings as KERNEL_FEATUREs
> - apply patches to a BSP's machine branch
>
> The following capabilities are specifically not provided by the tool:
>
> - help in determining a 'correct' set of kernel config options to set.
> The tool assumes the user already knows that, either through diffs
> of .config files before and after using 'bitbake -c menuconfig', or
> some other means.
Other means could also include the output of kconf_check. I could break
it out into something that could be iteratively run, if we think that
would be useful.
>
> - modification of the BSP metadata after BSP creation. The tool
> provides an initial 'write-once' BSP-generation capability, but
> doesn't allow it to be read in and modified after the initial pass.
> The user will have to make further modifications manually. The one
> exception to this is kernel features, which can be added to an
> already existing kernel bbappend file.
>
> - a guarantee or even expectation that the generated BSP will work on
> the actual hardware it's targeted for - it's highly unlikely the BSP
> will work 'out-of-the-box' and it's the developer's reponsibility to
> do the often hard work of figuring out what settings and/or code are
> actually needed to get the hardware to work as expected. The goal
> of the tool is to make that job easier, not to actually do that job.
>
> Also, though it's not explicitly a requirement of this tool, the
> design should be sufficiently modular to allow for its participation
> in generic logical pipelines. For instance, while the tool may
> present to the user a text-based UI for gathering information, it
> should also be able to operate without any kind of user interaction
> and retrieve the data it needs from a file, for instance e.g.:
>
> $ yocto-bsp create< x86-input.json
>
> Another example that the design should not preclude would be its use
> at the end of a bsp-import pipeline e.g.:
>
> $ cat 3rdparty.bsp | 3rdparty2yocto | yocto-bsp create
Setting expectations! ;) That would be nice though.
>
> The tool will initially be implemented as a set of command-line tools
> which will essentially be thin layers on top of a modular Python API;
> future tools may be GUI-based but would make use of the functionality
> exposed through the same API. There might also be an opportunity for
> future integration into existing tools such as hob; this isn't
> explicitly catered for in the current design, but it should be
> possible to do that based either on the functionality provided by the
> command-line interface or equivalently via the Python module API.
>
>
> A Quick Overview
> ----------------
>
> The initial set of tools will comprise 'yocto-bsp', for creating Yocto
> BSPs, and 'yocto-kernel', for BSP and BSP-related Yocto kernel
> maintenance tasks on existing BSPs.
>
> The interface and detailed description of each tool are described
> below, and the underlying templating mechanism is described following
> that, but it would be good for context to first try to give an overall
> picture of how the tools were designed to work and be used together.
>
> The envisioned usage model is that a user would first start out by
> creating a new BSP for a given architecture by invoking the 'yocto-bsp
> create' command. Choosing the architecture essentially chooses a set
> of files that the BSP will be based on. This set of files is a
> combination of a set of 'common' files that all BSPs inherit and a set
> of architecture-specific files that match the specified architecture.
I haven't read below yet, but is the use case covered for extending/cloning
an exiting BSP ? I find this to be a common case as well.
>
> Neither set of files can be used unmodified on their own as simple
> static copies of the originals in order to create a working BSP for a
> real machine - many of those files need to be tailored for the
> hardware using information supplied by the user. To accomplish this
> in an automated way, both the specific portions of the BSP files that
> need to be filled in with user input, and the means of (optionally)
> getting that input from the user are included as in-line annotations
> to the BSP files themselves using a simple templating scheme based on
> conventions used by web2py. Essentially, in the default case, the
> templating system is used to generate a series of text-based prompts
> that ask the user questions about the BSP being created. The data
> supplied by the user is then used to fill in the missing information
> in the BSP files. The user can also choose to specify all the
> information needed directly to the 'yocto-bsp create' command
> invocation itself; the 'yocto-bsp list' command exists to provide the
> user the list of 'properties' and enumerations of possible values for
> those properties, which taken together represent the data items needed
> to generate the BSP. The user can even more directly supply the
> information as a json object representing a serialized python
> dictionary, which is what the templating interface uses natively to
> represent properties and values needed for BSP generation.
>
> The end result is a Yocto-compliant BSP that should be directly
> buildable by the Yocto build system. The generated BSP is completely
> self-contained - it includes not only the BSP metadata, but also the
> necessary kernel components, which are normally for BSPs contained in
> the linux-yocto repo's meta and machine branches for the BSP. In
> order to help make the tool more efficient for the user, the tool does
> everything it needs to locally by initially using the linux-yocto
> SRC_URI; once ready, the user can then migrate the kernel components
> to an actual linux-yocto repo by using the 'yocto-kernel migrate'
> command. Once that's done, the user is in a position to submit both
> the BSP metadata and the linux-yocto commits upstream.
Nice. This is the best case scenario.
>
> Once a BSP is created, the job of 'yocto-bsp' is done (in the initial
> version of the tools - it may have sub-commands added later that could
> possibly manage the non-kernel aspects of a BSP) and the
> 'yocto-kernel' command can be used going forward to maintain the
> kernel portion of a BSP. 'yocto-kernel' allows both bare kernel
> config items and higher-level 'kernel features' to be created, used
> and removed, patches to be added to a BSP's machine branch, and the
> kernel changes to a BSP migrated to a linux-yocto repo. These
> have typically been difficult tasks for many users; the hope is that
> 'yocto-kernel' will make those tasks more manageable for everyone.
>
> The sections below describe each tool in detail.
>
> Following those sections is a detailed description of the templating
> mechanism, useful for anyone who needs to create templates for
> creating new BSPs using the 'yocto-bsp create' tool.
>
>
> The top-level bsp-creation tool, 'yocto-bsp'
> ============================================
>
> All of the functionality of the Yocto bsp-creation tool is available
> via the single 'yocto-bsp' command. Access to the specific functions
> provided by the tool is accomplished by specifying sub-commands to the
> main 'yocto-bsp' command. The design of the command-line interface is
> based on similar interfaces provided by git and perf, which is
> familiar to many users and is easily extensible.
>
> Usage help is always available for the command itself as well as for
> all sub-commands. Executing the top-level command displays a list of
> the available sub-commands, with descriptions:
>
> $ yocto-bsp
>
> usage: yocto-bsp [--version] [--help] COMMAND [ARGS]
>
> The most commonly used 'yocto-bsp' commands are:
> create Create a new Yocto BSP
> list List available values for options and BSP properties
>
> See 'yocto-bsp help COMMAND' for more information on a specific command.
>
>
> 'yocto-bsp create' - Create a new Yocto BSP
> -------------------------------------------
>
> NAME
> yocto-bsp-create - Create a new Yocto BSP
>
> SYNOPSIS
> yocto-bsp create<bsp-name> <karch> <kmachine>
> [--property<NAME:VAL> | -p<NAME:VAL>]
>
> DESCRIPTION
> This command creates a Yocto BSP based on the specified
> parameters. The new BSP will be a new Yocto BSP layer contained
> within the top-level directory specified as bsp-name i.e. not
> 'meta-bsp-name'. The value of the 'karch' parameter determines
> the set of files that will be generated for the BSP, along with
> the specific set of 'properties' that will be used to fill out the
> BSP-specific portions of the BSP.
>
> The BSP-specific properties that define the values that will be
> used to generate a particular BSP can be specified on the
> comman-line using the --property option for each property, which
> is a colon-separated name:value pair.
>
> The set of properties and enumerations of their possible values
> can be displayed using the 'yocto-bsp list' command.
>
> It isn't necessary however for the user to explicitly specify the
> property values using --property options - if none are specified
> in the 'yocto-bsp create' invocation, the user will be
> interactively prompted for each of the required property values,
> which will then be used as values for BSP generation.
>
> The 'kmachine' parameter names the 'machine branch' that the BSP's
> machine branch will be based on; the list of branches meaningful
> for that purpose can also be listed using the 'yocto-bsp list'
> command.
This may essentially be the BSP clone/extend I was asking about. In
fact, I'm almost sure of it!
>
> The 'yocto-bsp list' command can also be used to list the possible
> values for the 'karch' parameter.
>
> An typical example that would query the user for property values
> would be:
>
> $ yocto-bsp create meta-foo i386 yocto/standard/common-pc/foo
>
> Because the user hasn't specified any values for the BSP
> properties, yocto-bsp will query the user for each of the
> unspecified property values required to create the BSP i.e. the
> properties listed by 'yocto-bsp list properties'.
>
> Here's an example using explicitly-specified properties:
>
> $ yocto-bsp create meta-foo i386 yocto/standard/common-pc/foo
> --property tunefile:tune-atom --property touchscreen:true
> --property kfeature:cfg/smp [...]
>
>
> 'yocto-bsp list' - List available values for options and BSP properties
> -----------------------------------------------------------------------
>
> NAME
> yocto-bsp-list - List available values for options and BSP properties
>
> SYNOPSIS
> yocto bsp list<karch | kmachine>
> yocto bsp list<karch> properties
> yocto bsp list<karch> property<xxx>
>
> DESCRIPTION
> This command enumerates the complete set of possible values for a
> specified option or property needed by the BSP creation process.
>
> The first form enumerates all the possible values that exist and
> can be specified for the 'karch' or 'kmachine' parameters to the
> 'yocto bsp create' command. Example output for those options:
>
> $ yocto-bsp list karch
> i386
> x86_64
> arm
> powerpc
> mips
>
> $ yocto-bsp list kmachine
> yocto
> yocto/standard
> yocto/standard/common-pc
> yocto/standard/common-pc-64
> yocto/standard/preempt-rt
>
> The second form enumerates all the possible properties that exist
> and must have values specified for them in the 'yocto bsp create'
> command for the given 'karch'. Example output for that command:
>
> $ yocto-bsp list i386 properties
> "machine": boolean,
> "kmachine": choice,
> "tunefile": choicelist,
> "smp": boolean,
> "xserver": choicelist
> ["xserver-choice": choice]
> "kfeatures": choicelist
> ["kfeature": choice]
> "touchscreen": boolean,
> "keyboard": boolean
>
> The third form enumerates all the possible values that exist and
> can be specified for any of the enumerable properties of the given
> 'karch' in the 'yocto bsp create' command. Example output for
> those properties:
>
> $ yocto-bsp list i386 property xserver
> xserver-vesa
> xserver-i915
> xserver-emgd
>
> $ yocto-bsp list i386 property kfeatures
> features/taskstats
> cfg/sound
>
> $ yocto-bsp list i386 property tunefile
> i386
> tune-atom
> tune-core2
>
> $ yocto-bsp list x86_64 property tunefile
> x86_64
> tune-x86_64
> tune-corei7
> tune-ivb
>
> $ yocto-bsp list powerpc property tunefile
> powerpc
> tune-ppc603e
> tune-ppce300c2
> tune-ppce500
> tune-ppce500mc
> tune-ppce500v2
> tune-ppce5500-32b
> tune-ppce5500-64b
>
> $ yocto-bsp list arm property tunefile
> arm
> tune-arm1136jf-s
> tune-arm920t
> tune-arm926ejs
> tune-arm9tdmi
> tune-armv7
>
> $ yocto-bsp list mips property tunefile
> mips
> tune-mips32
>
>
> The top-level kernel usability tool, 'yocto-kernel'
> ===================================================
>
> All of the functionality of the Yocto kernel usability tool is
> available via the single 'yocto-kernel' command. Access to the
> specific functions provided by the tool is accomplished by specifying
> sub-commands to the main 'yocto-kernel' command. The design of the
> command-line interface is based on similar interfaces provided by git
> and perf, which is familiar to many users and is easily extensible.
>
> Usage help is always available for the command itself as well as for
> all sub-commands. Executing the top-level command displays a list of
> the available sub-commands, with descriptions:
>
> $ yocto-kernel
>
> usage: yocto-kernel [--version] [--help] COMMAND [ARGS]
>
> The most commonly used 'yocto-kernel' commands are:
> feature-list List available Yocto KERNEL_FEATUREs
> feature-define Define a new Yocto KERNEL_FEATURE
> feature-add Add Yocto KERNEL_FEATUREs to a BSP
> feature-rm Remove Yocto KERNEL_FEATUREs from a BSP
> config-list List the modifiable set of bare kernel config options for a BSP
> config-add Add or modify bare kernel config options for a BSP
> config-rm Remove bare kernel config options from a BSP
> patch-add Patch the Yocto kernel for a specific BSP
> migrate Move local kernel config/patches to a git repo
Looks like a good list to me. I'm sure there will be more in the
future!
>
> See 'yocto-kernel help COMMAND' for more information on a specific command.
>
>
> 'yocto-kernel feature list' - List available Yocto KERNEL_FEATUREs
> ------------------------------------------------------------------
>
> NAME
> yocto-kernel-feature-list - List available Yocto KERNEL_FEATUREs
>
> SYNOPSIS
> yocto-kernel feature list<bsp-name> --used
>
> DESCRIPTION
> This command lists all the kernel features available to a BSP.
> This includes any features temporarily attached to the named BSP
> via the 'feature define' command and thus the reason this command
> takes bsp-name as a parameter.
.. and in yocto 1.2, we'll make sure there's either an embedded .gitconfig
style file this can use, or some sort of standardized naming system
that can be scraped. I'm actually in favour of both.
>
> If the --used param is specified, only the KERNEL_FEATURES used by
> the BSP are listed.
>
>
> 'yocto-kernel feature define' - Define a new Yocto KERNEL_FEATURE
> -----------------------------------------------------------------
>
> NAME
> yocto-kernel-feature-define - Define a new Yocto KERNEL_FEATURE
>
> SYNOPSIS
> yocto-kernel feature define<bsp-name> <feature-name> <feature-desc>
> [<CONFIG_XXX=x> ...] [--dirname<dirname>] [--policy<hw|non-hw>]
>
> DESCRIPTION
> This command defines a new kernel feature, resulting in the
> addition of two new files for feature xxx to the named BSP's
> SRC_URI. If dirname is specified, the feature files will be named
> features_dirname_xxx.cfg and features_dirname_xxx.scc. Otherwise,
> the features will be named cfg_xxx.cfg and cfg_xxx.scc. The idea
> behind the naming is that the names reflect the user's intent to
> have the feature migrated to either the kernel-cache/cfg or
> kernel-cache/features directory if/when the kernel portion of the
> BSP is migrated into the linux-yocto kernel repo.
>
> The policy stating whether the new feature is considered to be a
> hardware or a non-hardware feature also needs to be specified in
> order to create a new feature, and a description of the new
> feature is also required; this will be added as a comment to the
> .scc file.
>
> NOTE: Although features are not BSP-specific, until the user has
> gone through the steps of migrating a feature to the linux-yocto
> repo, it needs to live somewhere accessible to the BSP metadata,
> which logically would be the kernel SRC_URI for the BSP using it.
>
> NOTE: It's up to the user to determine whether or not the config
> options for a feature being added make sense or not - this command
> does no sanity checking or verification of any kind to ensure that
> the config options contained in a kernel feature really make sense
> and will actually be set in in the final config. For example, if
> a config option depends on other config options, it will be turned
> off by kconfig if the other options aren't set correctly.
FYI: some changes that I'm making in 1.2, will allow just the .scc file
to be listed on the SRC_URI and everything else left off, and the tools
will pickup what they need from the source directory.
>
>
> 'yocto-kernel feature add' - Add Yocto KERNEL_FEATUREs to a BSP
> ---------------------------------------------------------------
>
> NAME
> yocto-kernel-feature-add - Add Yocto KERNEL_FEATUREs to a BSP
>
> SYNOPSIS
> yocto-kernel feature add<bsp-name> [<FEATURE> ...]
>
> DESCRIPTION
>
> This command adds one or more kernel features to a BSP. The set
> of features available to be added by this command for a BSP can be
> found via the 'yocto-kernel feature list' command. This command
> essentially adds a new machine-specific KERNEL_FEATURE_append to
> the linux-yocto .bbappend for the BSP.
>
>
> 'yocto-kernel feature rm' - Remove Yocto KERNEL_FEATUREs from a BSP
> -------------------------------------------------------------------
>
> NAME
> yocto-kernel-feature-rm - Remove Yocto KERNEL_FEATUREs from a BSP
>
> SYNOPSIS
> yocto-kernel feature rm<bsp-name> [<FEATURE> ...]
>
> DESCRIPTION
>
> This command removes one or more kernel features from a BSP. The
> set of features available to be removed by this command for a BSP
> can be found via the 'yocto-kernel feature list' command, if the
> --used option to that command is specified. This command
> essentially removes a machine-specific KERNEL_FEATURE from the
> linux-yocto .bbappend for the BSP.
>
>
> 'yocto-kernel config list' - List a BSP's modifiable config options
> -------------------------------------------------------------------
>
> NAME
> yocto-kernel-config-list - List the modifiable set of bare kernel
> config options for a BSP
>
> SYNOPSIS
> yocto-kernel config list<bsp-name>
>
> DESCRIPTION
> This command lists the 'modifiable' config items for a BSP
> i.e. the items which are eligible for modification or removal by
> other yocto-kernel commands.
>
> 'modifiable' config items are the config items contained a BSP's
> foo.cfg base config.
>
>
> 'yocto-kernel config add' - Add or modify a BSP's kernel config options
> -----------------------------------------------------------------------
>
> NAME
> yocto-kernel-config-add - Add or modify bare kernel config options
> for a BSP
>
> SYNOPSIS
> yocto-kernel config add<bsp-name> [<CONFIG_XXX=x> ...]
>
> DESCRIPTION
> This command adds one or more CONFIG_XXX=x items to a BSP's
> foo.cfg base config. If a config item is already present, the new
> value will simply overwrite the old value.
>
> NOTE: It's up to the user to determine whether or not the config
> options being added make sense or not - this command does no
> sanity checking or verification of any kind to ensure that a
> config option really makes sense and will actually be set in in
> the final config. For example, if a config option depends on
> other config options, it will be turned off by kconfig if the
> other options aren't set correctly.
>
>
> 'yocto-kernel config rm' - Remove BSP kernel config options
> -----------------------------------------------------------
>
> NAME
> yocto-kernel-config-rm - Remove bare kernel config options from a
> BSP
>
> SYNOPSIS
> yocto-kernel config rm<bsp-name> [<CONFIG_XXX> ...]
>
> DESCRIPTION
> This command removes (turns off) one or more CONFIG_XXX items to a
> BSP's foo.cfg base config.
>
> The set of config items available to be removed by this command
> for a BSP can be found via the 'yocto-kernel config list'
> command.
I can see how this could be tui based commands in the future.
>
>
> 'yocto-kernel patch add' - Patch the Yocto kernel for a specific BSP
> --------------------------------------------------------------------
>
> NAME
> yocto-kernel-patch-add - Patch the Yocto kernel for a specific BSP
>
> SYNOPSIS
> yocto-kernel patch add<bsp-name> [<PATCH> ...]
>
> DESCRIPTION
> This command adds one or more patches to a BSP's machine branch.
> The patch will be added to the BSP's linux-yocto kernel SRC_URI
> and will be guaranteed to be applied in the order specified.
>
> NOTE: It's up to the user to determine whether or not the patches
> being added makes sense or not - this command does no sanity
> checking or verification of any kind to ensure that a patch can
> actually be applied to the BSP's kernel branch; it's assumed that
> the user has already done that.
>
>
> 'yocto-kernel migrate' - Move local kernel config/patches to a git repo
> -----------------------------------------------------------------------
>
> NAME
> yocto-kernel-migrate - Move local kernel config/patches to a git repo
>
> SYNOPSIS
> yocto-kernel migrate<bsp-name> <local linux-yocto repo>
>
> DESCRIPTION
> This command converts the kernel configuration changes, patches,
> and features collected in a BSP's linux-yocto SRC_URI into an
> equivalent set of commits in a local linux-yocto repo. On
> success, the local files will named in the SRC_URI will be
> removed, and the linux-yocto .bbappend will be fixed up to point
> to the new repo.
>
> NOTE: The user is responsible for submitting the generated commits
> upstream and subsequently modifying the kernel .bbappend to
> use the upsteam kernel repo.
I like this one :)
>
>
> The design of the 'yocto-bsp create' templating mechanism
> ==========================================================
>
> This section describes the details of the templating mechanism used by
> 'yocto-bsp' to generate BSPs.
>
> Generating a new BSP is basically a templating exercise. A given BSP
> is comprised of a combination of a set of common files that all BSPs
> share along with a set of files specific to a given BSP type. Many of
> the files in both sets need to be configured according to the needs of
> the BSP; this also extends to the naming and existence of the files
> themselves.
>
> The templating scheme used in the Yocto tool is partly inspired by
> ideas from web2py, but adapted to a more general input mechanism for
> the specific problems posed by Yocto.
>
> There are no special files, just marked-up BSP metadata and kernel
> configuration, .scc, and patch files.
>
> The Yocto BSP markup is meant to be extremely simple and limited to
> pretty much this initial set of items; it is however designed to be
> extensible enough to cover any future need.
>
> Anything within double braces is considered to be a Yocto BSP markup
> tag:
>
> {{ foo }}
>
> Anything within double braces and starting with {{= is considered to
> implement simple variable replacement, where variables are by default
> simply keys in a dictionary representing the BSP and containing values
> for the whole BSP:
>
> {{=machine}}
>
> The above means that the whole expression will be replaced with the
> value of the 'machine' entry in the BSP dictionary. The syntax is
> actually shorthand for the equivalent Python code for accessing the
> value of a given key, but isn't restricted to that e.g. it could
> reference a local variable defined in a previous tag, for instance.
>
> In every other case, the 'yocto-bsp' tool will pre-process the
> contents looking for certain keywords at the beginning. The rest it
> will treat as Python code. As of now, there's only one keyword that's
> meaningful:
>
> {{ input }}
>
> which signals that anything contained is input-related. The
> 'yocto-bsp' implementation will use this information to prompt the
> user for input. Under certain conditions e.g. when the user has
> already supplied this information via a file for instance, anything
> contained in this is considered a no-op and ignored.
>
> Anything else is considered to be Python code and will be directly
> incorporated into the BSP generation code.
>
> {{ if keyboard == False: }}
>
> Note also that any of the above can also be used in file and directory
> names, which will also be processed in the same way; this allows us to
> avoid extraneous files is consistent with the rest of the scheme.
>
> Internally, the way things are envisioned to work is that the tool
> will first read in the set of files that makes up the BSP for the
> chosen architecture. It will then pre-process all the 'input' tags it
> finds and use that to prepare a series of text-based prompts to
> retrieve the needed information from the user. The end result of
> presenting all that to the user will be a completd BSP dictionary
> containing all the key/value pairs needed to fill in the BSP. Once
> that information is available, it will make a second pass over the
> remaining tags and fill them in with the Python code needed to
> retrieve and replace the remaining tags with their corresponding
> values from the BSP dictionary (or any other source possibly
> introduced by other tags). The end result of this second pass will be
> executed as Python code to produce the final BSP metadata.
>
> For the first version, the 'input' tags will be used to construct a
> series of text-based questions for the user. The design, however,
> should make it relatively trivial to create an application that could
> create real GUI that would do the same thing at some later point. For
> example, such an application could use the input tags to generate a
> web-based form, which would return the results gathered from the user
> to a web service in the form of a json blob that could directly be
> converted into the BSP Python dictionary needed. e.g. the browser
> would return something like this to the web service, which would in
> turn send it to the 'bsp-create' library function to create the final
> BSP.
>
> {
> "machine": "foo",
> "kmachine": "/yocto/standard/base",
> "tunefile": "tune-atom",
> "smp": true,
> "xserver": {
> "xserver-choice": "xserver-i915",
> },
> "kfeatures": [
> "kfeature": "features/taskstats",
> "kfeature": "cfg/sound",
> ]
> },
> touchscreen: false,
> keyboard: true
> }
>
> Below are some concrete examples of specific tags and how they're used.
>
> Here's a simple example of markup that describes information needed
> from the user about touchscreen and keyboard capabilities in the
> formfactor 'machconfig' file:
>
> # Assume a USB mouse and keyboard are connected
> {{ input type:"boolean" name:"touchscreen" msg:"Does your BSP have a touchscreen?" }}
> HAVE_TOUCHSCREEN={{=touchscreen}}
> {{ input type:"boolean" name:"keyboard" msg:"Does your BSP have a keyboard?" }}
> HAVE_KEYBOARD={{=keyboard}}
>
> For instance, the touchscreen 'input' tag above describes everything a
> UI would need in order to retrieve an answer from the user as to
> whether the hardware in question has a touchscreen or not.
>
> The 'type' key comprises a small set of possible input types that
> include things like 'boolean' (yes/no) choices and lists of choices
> (type:"choicelist").
>
> The 'name' key provides a means for later markup to access the value
> of the variable containing the user's answer for that input. For
> example, the name:"touchscreen" simply references the value in the BSP
> dictionary that was filled in with the user's response to the question
> which was formed and presented to the user using the "msg" tag in the
> input tag.
>
> Below the input tag is the actual content that will be rendered into
> the 'machconfig' file, but with the {{=}} portion replaced with the
> 'touchscreen' value supplied by the user i.e.:
>
> HAVE_TOUCHSCREEN={{=touchscreen}}
>
> will be rendered as:
>
> HAVE_TOUCHSCREEN=0
>
> Assuming the user answered 'no'.
>
> Another interesting thing to note about this example is that in Yocto
> BSPs, the machconfig files are contained within a filesystem location
> that includes the machine name in the path e.g.
>
> recipes-bsp/formfactor/formfactor/foo/machconfig
>
> To accomplish this, the directory names themselves contain markup that
> gets processed and replaced by the tool:
>
> recipes-bsp/formfactor/formfactor/{{=machine}}/machconfig
>
> Here, the {{=machine} directory is renamed to /foo to match the
> machine name of the BSP.
>
> Here's a more involved example, markup included in the
> i386/conf/{{=machine}}.conf file:
>
> {{ input type:"boolean" name:"xserver" msg:"Do you need support for X?" }}
> {{ if xserver: }}
> XSERVER ?= "${XSERVER_IA32_BASE} \
> ${XSERVER_IA32_EXT} \
> {{ input type:"choicelist" name:"xserver_choice" msg:"Please select an xserver for this machine" }}
> {{ input type:"choice" val:"xserver-vesa" msg:"VESA xserver support" }}
> ${XSERVER_IA32_VESA} \
> {{ input type:"choice" val:"xserver-emgd" msg:"EMGD xserver support (proprietary)" }}
> ${XSERVER_IA32_EMGD} \
> {{ input type:"choice" val:"xserver-i915" desc:"i915 xserver support" }}
> ${XSERVER_IA32_I915} \
> "
>
> In this case, we start off with a boolean 'input' question asking if
> the BSP needs to support X. The following line is straight Python
> code which essentially skips the following sections (until a blank
> line) if the user's answer was 'no', or if the answer was 'yes',
> creates a further query to the user asking which xserver type to use.
> In the latter case, the input type is a list of choices resulting in
> the values specified by the 'val' tag appearing as the value for the
> 'xserver_choice' dictionary key.
>
> Code further down in the .conf file can also make use of this
> information to include or exclude content based on the value of the
> xserver_choice variable:
>
> {{ if xserver_choice == "xserver-emgd": }}
> PREFERRED_VERSION_xserver-xorg ?= "1.9.3"
> PREFERRED_VERSION_mesa-dri ?= "7.11"
> PREFERRED_VERSION_emgd-driver-bin ?= "1.8"
>
> {{ if xserver_choice == "xserver-vesa" or xserver_choice == "xserver-emgd": }}
> APPEND += "video=vesafb vga=0x318"
>
> Note again that the tags used to determine whether or not to include
> content simply contain Python code used to directly implement the
> needed logic. Not only is there no need for the user to learn a
> special-purpose language, but gives the tool the flexibility to add
> essentially any kind of processing logic needed in the future.
>
> In the case of xserver choices, we also need to use the value of
> xserver_choice to determine which xorg.conf file to add to the BSP.
> Again, consistent with the rest of the markup processing, this is
> seamlessly accomplished by adding markup to the filenames themselves:
>
> recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver_choice == xserver_emgd: }} xorg.conf
> recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver_choice == xserver_vesa: }} xorg.conf
> recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver_choice == xserver_i915: }} xorg.conf
>
> In this case, the filenames actually contain Python code that
> describes the conditions needed to include those particular files.
>
> Here's another example, again making use of general-purpose Python in
> the markup to generate an open-ended list of choices for the user, in
> this case in linux-yocto_3.0.bbappend, to generate a list of available
> KERNEL_FEATURES for the user to choose from:
>
> {{ input type:"checklist" name:"kfeatures" gen:"fetch-kernel-features()" msg:"Please choose the kernel features to use in this BSP on" }}
> {{ for kfeature in kfeatures: }}
> KERNEL_FEATURES_append_{{=machine}} += {{=kfeature}}
>
> In this case an 'input' tag calls an internally-defined yocto-bsp
> function named fetch_kernel_feature() that generates the list of
> available KERNEL_FEATURES. The output of that call is a Python list
> containing the features, which is assigned to the 'kfeatures'
> variable. The next tag contains Python code that simply iterates over
> that list, and uses the value of the current feature, 'kfeature', to
> append the chosen kernel features to the list of features used by the
> BSP.
>
> Finally, because the kernel components are actually part of the BSP
> metadata, the user-specified values are also available in the
> processing of those files. For instance, in the below case, we can
> use the xserver_choice to include or exclude specific kernel
> configuration settings in the
> recipes-kernel/linux/linux-yocto/{{=machine}}.scc file associated with
> the BSP:
>
> {{ if xserver_choice == "xserver-emgd": }}
> include features/drm-emgd/drm-emgd.scc
Seems reasonable to me.
>
> ---
>
> For reference, here's an example of how the tools are expected to be
> laid out. Everything is contained within a 'bsp-tools' directory
> under '/scripts'. The 'common' portion, containing the files common
> to every BSP are:
>
> bsp-tools/
> bsp-tools/yocto-bsp.py
> bsp-tools/yocto-kernel.py
> bsp-tools/bsp-cache
> bsp-tools/bsp-cache/target
> bsp-tools/bsp-cache/target/arch
> bsp-tools/bsp-cache/target/arch/common
> bsp-tools/bsp-cache/target/arch/common/README.sources
> bsp-tools/bsp-cache/target/arch/common/binary
> bsp-tools/bsp-cache/target/arch/common/binary/.gitignore
> bsp-tools/bsp-cache/target/arch/common/recipes-kernel
> bsp-tools/bsp-cache/target/arch/common/recipes-kernel/linux
> bsp-tools/bsp-cache/target/arch/common/COPYING.MIT
> bsp-tools/bsp-cache/target/arch/common/README
> bsp-tools/bsp-cache/target/arch/common/recipes-graphics
> bsp-tools/bsp-cache/target/arch/common/recipes-graphics/xorg-xserver
> bsp-tools/bsp-cache/target/arch/common/recipes-graphics/xorg-xserver/{{ if xserver: }} xserver-xf86-config_0.1.bbappend
> bsp-tools/bsp-cache/target/arch/common/conf
> bsp-tools/bsp-cache/target/arch/common/conf/layer.conf
> bsp-tools/bsp-cache/target/arch/common/recipes-core
> bsp-tools/bsp-cache/target/arch/common/recipes-core/tasks
> bsp-tools/bsp-cache/target/arch/common/recipes-core/tasks/task-core-tools.bbappend
> bsp-tools/bsp-cache/target/arch/common/recipes-bsp
> bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor
> bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor_0.0.bbappend
> bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor
> bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor/{{=machine}}
> bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor/{{=machine}}/machconfig
>
> The BSP-specific files are contained in directories named for the
> architecture. In this case, only the i386 files are expanded as an
> example.
>
> bsp-tools/bsp-cache/target/arch/arm
> bsp-tools/bsp-cache/target/arch/powerpc
> bsp-tools/bsp-cache/target/arch/i386
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt_3.0.bbappend
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto_3.0.bbappend
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}.cfg
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}.scc
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}-preempt-rt.scc
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}-standard.scc
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}.cfg
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}.scc
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}-preempt-rt.scc
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}-standard.scc
Once my 1.2 updates are in (They are waiting for the patch list
abstraction to merge), subdirectories for features will work fine
as well.
> bsp-tools/bsp-cache/target/arch/i386/recipes-graphics
> bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config
> bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}
> bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver-choice == xserver-emgd: }} xorg.conf
> bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver-choice == xserver-vesa: }} xorg.conf
> bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver-choice == xserver-i915: }} xorg.conf
> bsp-tools/bsp-cache/target/arch/i386/conf
> bsp-tools/bsp-cache/target/arch/i386/conf/machine
> bsp-tools/bsp-cache/target/arch/i386/conf/machine/{{=machine}}.conf
> bsp-tools/bsp-cache/target/arch/mips
> bsp-tools/bsp-cache/target/arch/x86_64
I made it to the end. Is there a prize ? This looks really good/interesting.
Cheers,
Bruce
>
>
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: RFC: Yocto BSP and kernel usability tools
2011-12-22 20:16 ` Bruce Ashfield
@ 2011-12-22 21:31 ` Tom Zanussi
0 siblings, 0 replies; 7+ messages in thread
From: Tom Zanussi @ 2011-12-22 21:31 UTC (permalink / raw)
To: Bruce Ashfield; +Cc: yocto@yoctoproject.org
On Thu, 2011-12-22 at 15:16 -0500, Bruce Ashfield wrote:
> On 11-12-19 11:55 AM, Tom Zanussi wrote:
> > Hi,
> >
> > This is what I hope to implement in the M2 timeframe.
> >
> > Any and all comments welcome...
> >
> > Thanks,
> >
> > Tom
> >
> > ---
> >
> > Getting Yocto running on a new board is often a prerequisite task to
> > the real task of interest for new users, which is creating new
> > applications designed to run on the board, or creating new
> > board-specific embedded systems. In other words, most users just want
> > to get the hardware up and running and not have to think too much
> > about it before moving on to what they're really interested in, which
> > is the stuff running on top of it.
> >
> > While there really can't be any such thing as a magic-bullet
> > application that would do that automatically i.e. generate and
> > maintain a guaranteed-working Yocto BSP for any given piece of
> > hardware, it should at least be possible to provide a tool that would
> > generate a base BSP usable as a starting point for that, and which
> > would additionally allow the user to tweak various BSP settings,
> > including kernel configuration options and simple kernel patches.
> > Such a tool should be usable with only a minimal understanding of the
> > Yocto build system and metadata, and with no knowledge at all of
> > kernel internals or the Yocto kernel recipe bindings, or of the
> > details of the internals of yocto kernel repositories, or in fact any
> > knowledge of git or any other specific build-related tool.
> >
> > This document provides the design for such a tool. At a high level,
> > the goal of the tool is to provide a user the means of creating a new
> > Yocto BSP from which it should be possible (assuming a perfect
> > run-time outcome, admittedly unlikely on a first pass) to submit a
> > 'pull request' for inclusion into any repo(s) that would accept an
> > 'official' Yocto BSP. For example, the output of the tool should
> > produce metadata and kernel configuration that would be directly
> > mergeable into the meta-intel and the linux-yocto kernel repos. The
> > tool should also allow the BSP maintainer to afterwards at any time
> > make changes to the kernel configuration and/or submit kernel patches
> > for the BSP.
> >
> > Note that although knowledge of the Yocto kernel or kernel internals
> > is not expected, the user is expected to know the basics of how to
> > deal with the linux kernel at the 'user' level, specifically
> > understanding and determining the specific kernel config options
> > needed for the user's BSP.
> >
> > Specifically, the tool will allow for the following high-level tasks,
> > which are discussed in more detail below:
> >
> > - create a new BSP from scratch from a set of user-settable parameters
> > - manage individual kernel config settings
> > - create and use groups of kernel config settings as KERNEL_FEATUREs
> > - apply patches to a BSP's machine branch
> >
> > The following capabilities are specifically not provided by the tool:
> >
> > - help in determining a 'correct' set of kernel config options to set.
> > The tool assumes the user already knows that, either through diffs
> > of .config files before and after using 'bitbake -c menuconfig', or
> > some other means.
>
> Other means could also include the output of kconf_check. I could break
> it out into something that could be iteratively run, if we think that
> would be useful.
>
Yeah, definitely, anything to remove the guesswork as to whether a set
of options will actually make it to the final .config. For the first
iteration I kind of wanted to avoid any UI issues like invoking
menuconfig and also wanted to avoid the extra complexity of taking
config diffs etc. but I think you're right - at minimum we should be
able to tell the user immediately if the options they're trying to use
don't make sense, and I'll make use of any easily invokable kconf check
functions that might exist for that purpose. ;-)
> >
> > - modification of the BSP metadata after BSP creation. The tool
> > provides an initial 'write-once' BSP-generation capability, but
> > doesn't allow it to be read in and modified after the initial pass.
> > The user will have to make further modifications manually. The one
> > exception to this is kernel features, which can be added to an
> > already existing kernel bbappend file.
> >
> > - a guarantee or even expectation that the generated BSP will work on
> > the actual hardware it's targeted for - it's highly unlikely the BSP
> > will work 'out-of-the-box' and it's the developer's reponsibility to
> > do the often hard work of figuring out what settings and/or code are
> > actually needed to get the hardware to work as expected. The goal
> > of the tool is to make that job easier, not to actually do that job.
> >
> > Also, though it's not explicitly a requirement of this tool, the
> > design should be sufficiently modular to allow for its participation
> > in generic logical pipelines. For instance, while the tool may
> > present to the user a text-based UI for gathering information, it
> > should also be able to operate without any kind of user interaction
> > and retrieve the data it needs from a file, for instance e.g.:
> >
> > $ yocto-bsp create< x86-input.json
> >
> > Another example that the design should not preclude would be its use
> > at the end of a bsp-import pipeline e.g.:
> >
> > $ cat 3rdparty.bsp | 3rdparty2yocto | yocto-bsp create
>
> Setting expectations! ;) That would be nice though.
>
Yeah, this has actually been asked for already, so I need to make sure
it's possible, which I think it will be - in fact once the tools are in
a working state, I'll probably try something like that pretty quickly.
> >
> > The tool will initially be implemented as a set of command-line tools
> > which will essentially be thin layers on top of a modular Python API;
> > future tools may be GUI-based but would make use of the functionality
> > exposed through the same API. There might also be an opportunity for
> > future integration into existing tools such as hob; this isn't
> > explicitly catered for in the current design, but it should be
> > possible to do that based either on the functionality provided by the
> > command-line interface or equivalently via the Python module API.
> >
> >
> > A Quick Overview
> > ----------------
> >
> > The initial set of tools will comprise 'yocto-bsp', for creating Yocto
> > BSPs, and 'yocto-kernel', for BSP and BSP-related Yocto kernel
> > maintenance tasks on existing BSPs.
> >
> > The interface and detailed description of each tool are described
> > below, and the underlying templating mechanism is described following
> > that, but it would be good for context to first try to give an overall
> > picture of how the tools were designed to work and be used together.
> >
> > The envisioned usage model is that a user would first start out by
> > creating a new BSP for a given architecture by invoking the 'yocto-bsp
> > create' command. Choosing the architecture essentially chooses a set
> > of files that the BSP will be based on. This set of files is a
> > combination of a set of 'common' files that all BSPs inherit and a set
> > of architecture-specific files that match the specified architecture.
>
> I haven't read below yet, but is the use case covered for extending/cloning
> an exiting BSP ? I find this to be a common case as well.
>
Not really. I mean, I don't see a big problem with making a copy of an
existing BSP, doing a search and replace of the obvious things that need
fixing up for a new BSP, having it either re-use or create a new kernel
branch and fixing up the new BSP to point to either the new or existing
branch, but it wasn't the plan for this tool - doing much more than
those simple things for any given BSP out in the wild seems too
ambitious, but I may be misunderstanding the use case or obvious ways of
accommodating it...
> >
> > Neither set of files can be used unmodified on their own as simple
> > static copies of the originals in order to create a working BSP for a
> > real machine - many of those files need to be tailored for the
> > hardware using information supplied by the user. To accomplish this
> > in an automated way, both the specific portions of the BSP files that
> > need to be filled in with user input, and the means of (optionally)
> > getting that input from the user are included as in-line annotations
> > to the BSP files themselves using a simple templating scheme based on
> > conventions used by web2py. Essentially, in the default case, the
> > templating system is used to generate a series of text-based prompts
> > that ask the user questions about the BSP being created. The data
> > supplied by the user is then used to fill in the missing information
> > in the BSP files. The user can also choose to specify all the
> > information needed directly to the 'yocto-bsp create' command
> > invocation itself; the 'yocto-bsp list' command exists to provide the
> > user the list of 'properties' and enumerations of possible values for
> > those properties, which taken together represent the data items needed
> > to generate the BSP. The user can even more directly supply the
> > information as a json object representing a serialized python
> > dictionary, which is what the templating interface uses natively to
> > represent properties and values needed for BSP generation.
> >
> > The end result is a Yocto-compliant BSP that should be directly
> > buildable by the Yocto build system. The generated BSP is completely
> > self-contained - it includes not only the BSP metadata, but also the
> > necessary kernel components, which are normally for BSPs contained in
> > the linux-yocto repo's meta and machine branches for the BSP. In
> > order to help make the tool more efficient for the user, the tool does
> > everything it needs to locally by initially using the linux-yocto
> > SRC_URI; once ready, the user can then migrate the kernel components
> > to an actual linux-yocto repo by using the 'yocto-kernel migrate'
> > command. Once that's done, the user is in a position to submit both
> > the BSP metadata and the linux-yocto commits upstream.
>
> Nice. This is the best case scenario.
>
> >
> > Once a BSP is created, the job of 'yocto-bsp' is done (in the initial
> > version of the tools - it may have sub-commands added later that could
> > possibly manage the non-kernel aspects of a BSP) and the
> > 'yocto-kernel' command can be used going forward to maintain the
> > kernel portion of a BSP. 'yocto-kernel' allows both bare kernel
> > config items and higher-level 'kernel features' to be created, used
> > and removed, patches to be added to a BSP's machine branch, and the
> > kernel changes to a BSP migrated to a linux-yocto repo. These
> > have typically been difficult tasks for many users; the hope is that
> > 'yocto-kernel' will make those tasks more manageable for everyone.
> >
> > The sections below describe each tool in detail.
> >
> > Following those sections is a detailed description of the templating
> > mechanism, useful for anyone who needs to create templates for
> > creating new BSPs using the 'yocto-bsp create' tool.
> >
> >
> > The top-level bsp-creation tool, 'yocto-bsp'
> > ============================================
> >
> > All of the functionality of the Yocto bsp-creation tool is available
> > via the single 'yocto-bsp' command. Access to the specific functions
> > provided by the tool is accomplished by specifying sub-commands to the
> > main 'yocto-bsp' command. The design of the command-line interface is
> > based on similar interfaces provided by git and perf, which is
> > familiar to many users and is easily extensible.
> >
> > Usage help is always available for the command itself as well as for
> > all sub-commands. Executing the top-level command displays a list of
> > the available sub-commands, with descriptions:
> >
> > $ yocto-bsp
> >
> > usage: yocto-bsp [--version] [--help] COMMAND [ARGS]
> >
> > The most commonly used 'yocto-bsp' commands are:
> > create Create a new Yocto BSP
> > list List available values for options and BSP properties
> >
> > See 'yocto-bsp help COMMAND' for more information on a specific command.
> >
> >
> > 'yocto-bsp create' - Create a new Yocto BSP
> > -------------------------------------------
> >
> > NAME
> > yocto-bsp-create - Create a new Yocto BSP
> >
> > SYNOPSIS
> > yocto-bsp create<bsp-name> <karch> <kmachine>
> > [--property<NAME:VAL> | -p<NAME:VAL>]
> >
> > DESCRIPTION
> > This command creates a Yocto BSP based on the specified
> > parameters. The new BSP will be a new Yocto BSP layer contained
> > within the top-level directory specified as bsp-name i.e. not
> > 'meta-bsp-name'. The value of the 'karch' parameter determines
> > the set of files that will be generated for the BSP, along with
> > the specific set of 'properties' that will be used to fill out the
> > BSP-specific portions of the BSP.
> >
> > The BSP-specific properties that define the values that will be
> > used to generate a particular BSP can be specified on the
> > comman-line using the --property option for each property, which
> > is a colon-separated name:value pair.
> >
> > The set of properties and enumerations of their possible values
> > can be displayed using the 'yocto-bsp list' command.
> >
> > It isn't necessary however for the user to explicitly specify the
> > property values using --property options - if none are specified
> > in the 'yocto-bsp create' invocation, the user will be
> > interactively prompted for each of the required property values,
> > which will then be used as values for BSP generation.
> >
> > The 'kmachine' parameter names the 'machine branch' that the BSP's
> > machine branch will be based on; the list of branches meaningful
> > for that purpose can also be listed using the 'yocto-bsp list'
> > command.
>
> This may essentially be the BSP clone/extend I was asking about. In
> fact, I'm almost sure of it!
>
Not sure - if the extent of it is basically to create a new branch based
on an existing one, then yeah.
[ ... ]
> >
> > 'yocto-kernel feature list' - List available Yocto KERNEL_FEATUREs
> > ------------------------------------------------------------------
> >
> > NAME
> > yocto-kernel-feature-list - List available Yocto KERNEL_FEATUREs
> >
> > SYNOPSIS
> > yocto-kernel feature list<bsp-name> --used
> >
> > DESCRIPTION
> > This command lists all the kernel features available to a BSP.
> > This includes any features temporarily attached to the named BSP
> > via the 'feature define' command and thus the reason this command
> > takes bsp-name as a parameter.
>
> .. and in yocto 1.2, we'll make sure there's either an embedded .gitconfig
> style file this can use, or some sort of standardized naming system
> that can be scraped. I'm actually in favour of both.
>
Yeah, whatever makes sense here. In my template examples, I assumed the
existence of a fetch-kernel-features() function that would do that,
hadn't considered the implementation yet, though.
> >
> > If the --used param is specified, only the KERNEL_FEATURES used by
> > the BSP are listed.
> >
> >
> > 'yocto-kernel feature define' - Define a new Yocto KERNEL_FEATURE
> > -----------------------------------------------------------------
> >
> > NAME
> > yocto-kernel-feature-define - Define a new Yocto KERNEL_FEATURE
> >
> > SYNOPSIS
> > yocto-kernel feature define<bsp-name> <feature-name> <feature-desc>
> > [<CONFIG_XXX=x> ...] [--dirname<dirname>] [--policy<hw|non-hw>]
> >
> > DESCRIPTION
> > This command defines a new kernel feature, resulting in the
> > addition of two new files for feature xxx to the named BSP's
> > SRC_URI. If dirname is specified, the feature files will be named
> > features_dirname_xxx.cfg and features_dirname_xxx.scc. Otherwise,
> > the features will be named cfg_xxx.cfg and cfg_xxx.scc. The idea
> > behind the naming is that the names reflect the user's intent to
> > have the feature migrated to either the kernel-cache/cfg or
> > kernel-cache/features directory if/when the kernel portion of the
> > BSP is migrated into the linux-yocto kernel repo.
> >
> > The policy stating whether the new feature is considered to be a
> > hardware or a non-hardware feature also needs to be specified in
> > order to create a new feature, and a description of the new
> > feature is also required; this will be added as a comment to the
> > .scc file.
> >
> > NOTE: Although features are not BSP-specific, until the user has
> > gone through the steps of migrating a feature to the linux-yocto
> > repo, it needs to live somewhere accessible to the BSP metadata,
> > which logically would be the kernel SRC_URI for the BSP using it.
> >
> > NOTE: It's up to the user to determine whether or not the config
> > options for a feature being added make sense or not - this command
> > does no sanity checking or verification of any kind to ensure that
> > the config options contained in a kernel feature really make sense
> > and will actually be set in in the final config. For example, if
> > a config option depends on other config options, it will be turned
> > off by kconfig if the other options aren't set correctly.
>
> FYI: some changes that I'm making in 1.2, will allow just the .scc file
> to be listed on the SRC_URI and everything else left off, and the tools
> will pickup what they need from the source directory.
>
Excellent, that'll make things much cleaner, thanks.
[ ... ]
> >
> >
> > 'yocto-kernel migrate' - Move local kernel config/patches to a git repo
> > -----------------------------------------------------------------------
> >
> > NAME
> > yocto-kernel-migrate - Move local kernel config/patches to a git repo
> >
> > SYNOPSIS
> > yocto-kernel migrate<bsp-name> <local linux-yocto repo>
> >
> > DESCRIPTION
> > This command converts the kernel configuration changes, patches,
> > and features collected in a BSP's linux-yocto SRC_URI into an
> > equivalent set of commits in a local linux-yocto repo. On
> > success, the local files will named in the SRC_URI will be
> > removed, and the linux-yocto .bbappend will be fixed up to point
> > to the new repo.
> >
> > NOTE: The user is responsible for submitting the generated commits
> > upstream and subsequently modifying the kernel .bbappend to
> > use the upsteam kernel repo.
>
> I like this one :)
>
I thought you might be happy that I didn't go so far as to actually have
the tool automatically submitting the results. ;-)
[ ...]
> >
> > The BSP-specific files are contained in directories named for the
> > architecture. In this case, only the i386 files are expanded as an
> > example.
> >
> > bsp-tools/bsp-cache/target/arch/arm
> > bsp-tools/bsp-cache/target/arch/powerpc
> > bsp-tools/bsp-cache/target/arch/i386
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt_3.0.bbappend
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto_3.0.bbappend
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}.cfg
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}.scc
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}-preempt-rt.scc
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}-standard.scc
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}.cfg
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}.scc
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}-preempt-rt.scc
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}-standard.scc
>
> Once my 1.2 updates are in (They are waiting for the patch list
> abstraction to merge), subdirectories for features will work fine
> as well.
>
Also great news that will really simplify things, thanks.
> > bsp-tools/bsp-cache/target/arch/i386/recipes-graphics
> > bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config
> > bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}
> > bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver-choice == xserver-emgd: }} xorg.conf
> > bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver-choice == xserver-vesa: }} xorg.conf
> > bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver-choice == xserver-i915: }} xorg.conf
> > bsp-tools/bsp-cache/target/arch/i386/conf
> > bsp-tools/bsp-cache/target/arch/i386/conf/machine
> > bsp-tools/bsp-cache/target/arch/i386/conf/machine/{{=machine}}.conf
> > bsp-tools/bsp-cache/target/arch/mips
> > bsp-tools/bsp-cache/target/arch/x86_64
>
> I made it to the end. Is there a prize ? This looks really good/interesting.
>
Thanks for persevering, I really appreciate the input and comments!
Tom
> Cheers,
>
> Bruce
>
> >
> >
> > _______________________________________________
> > yocto mailing list
> > yocto@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/yocto
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: RFC: Yocto BSP and kernel usability tools
2011-12-19 16:55 RFC: Yocto BSP and kernel usability tools Tom Zanussi
2011-12-22 20:16 ` Bruce Ashfield
@ 2011-12-28 0:36 ` McClintock Matthew-B29882
2011-12-29 17:32 ` Tom Zanussi
2012-01-31 22:52 ` Darren Hart
2 siblings, 1 reply; 7+ messages in thread
From: McClintock Matthew-B29882 @ 2011-12-28 0:36 UTC (permalink / raw)
To: Tom Zanussi; +Cc: yocto@yoctoproject.org
(This probably belongs somewhere inline but I got lost on the long email)
A lot of times users would already have a git tree containing linux
and/or u-boot already working for their boards. Perhaps you could let
the user point at external linux and/or u-boot trees with defconfigs,
u-boot configs, device trees, etc for their target BSP?
-M
On Mon, Dec 19, 2011 at 10:55 AM, Tom Zanussi <tom.zanussi@intel.com> wrote:
> Hi,
>
> This is what I hope to implement in the M2 timeframe.
>
> Any and all comments welcome...
>
> Thanks,
>
> Tom
>
> ---
>
> Getting Yocto running on a new board is often a prerequisite task to
> the real task of interest for new users, which is creating new
> applications designed to run on the board, or creating new
> board-specific embedded systems. In other words, most users just want
> to get the hardware up and running and not have to think too much
> about it before moving on to what they're really interested in, which
> is the stuff running on top of it.
>
> While there really can't be any such thing as a magic-bullet
> application that would do that automatically i.e. generate and
> maintain a guaranteed-working Yocto BSP for any given piece of
> hardware, it should at least be possible to provide a tool that would
> generate a base BSP usable as a starting point for that, and which
> would additionally allow the user to tweak various BSP settings,
> including kernel configuration options and simple kernel patches.
> Such a tool should be usable with only a minimal understanding of the
> Yocto build system and metadata, and with no knowledge at all of
> kernel internals or the Yocto kernel recipe bindings, or of the
> details of the internals of yocto kernel repositories, or in fact any
> knowledge of git or any other specific build-related tool.
>
> This document provides the design for such a tool. At a high level,
> the goal of the tool is to provide a user the means of creating a new
> Yocto BSP from which it should be possible (assuming a perfect
> run-time outcome, admittedly unlikely on a first pass) to submit a
> 'pull request' for inclusion into any repo(s) that would accept an
> 'official' Yocto BSP. For example, the output of the tool should
> produce metadata and kernel configuration that would be directly
> mergeable into the meta-intel and the linux-yocto kernel repos. The
> tool should also allow the BSP maintainer to afterwards at any time
> make changes to the kernel configuration and/or submit kernel patches
> for the BSP.
>
> Note that although knowledge of the Yocto kernel or kernel internals
> is not expected, the user is expected to know the basics of how to
> deal with the linux kernel at the 'user' level, specifically
> understanding and determining the specific kernel config options
> needed for the user's BSP.
>
> Specifically, the tool will allow for the following high-level tasks,
> which are discussed in more detail below:
>
> - create a new BSP from scratch from a set of user-settable parameters
> - manage individual kernel config settings
> - create and use groups of kernel config settings as KERNEL_FEATUREs
> - apply patches to a BSP's machine branch
>
> The following capabilities are specifically not provided by the tool:
>
> - help in determining a 'correct' set of kernel config options to set.
> The tool assumes the user already knows that, either through diffs
> of .config files before and after using 'bitbake -c menuconfig', or
> some other means.
>
> - modification of the BSP metadata after BSP creation. The tool
> provides an initial 'write-once' BSP-generation capability, but
> doesn't allow it to be read in and modified after the initial pass.
> The user will have to make further modifications manually. The one
> exception to this is kernel features, which can be added to an
> already existing kernel bbappend file.
>
> - a guarantee or even expectation that the generated BSP will work on
> the actual hardware it's targeted for - it's highly unlikely the BSP
> will work 'out-of-the-box' and it's the developer's reponsibility to
> do the often hard work of figuring out what settings and/or code are
> actually needed to get the hardware to work as expected. The goal
> of the tool is to make that job easier, not to actually do that job.
>
> Also, though it's not explicitly a requirement of this tool, the
> design should be sufficiently modular to allow for its participation
> in generic logical pipelines. For instance, while the tool may
> present to the user a text-based UI for gathering information, it
> should also be able to operate without any kind of user interaction
> and retrieve the data it needs from a file, for instance e.g.:
>
> $ yocto-bsp create < x86-input.json
>
> Another example that the design should not preclude would be its use
> at the end of a bsp-import pipeline e.g.:
>
> $ cat 3rdparty.bsp | 3rdparty2yocto | yocto-bsp create
>
> The tool will initially be implemented as a set of command-line tools
> which will essentially be thin layers on top of a modular Python API;
> future tools may be GUI-based but would make use of the functionality
> exposed through the same API. There might also be an opportunity for
> future integration into existing tools such as hob; this isn't
> explicitly catered for in the current design, but it should be
> possible to do that based either on the functionality provided by the
> command-line interface or equivalently via the Python module API.
>
>
> A Quick Overview
> ----------------
>
> The initial set of tools will comprise 'yocto-bsp', for creating Yocto
> BSPs, and 'yocto-kernel', for BSP and BSP-related Yocto kernel
> maintenance tasks on existing BSPs.
>
> The interface and detailed description of each tool are described
> below, and the underlying templating mechanism is described following
> that, but it would be good for context to first try to give an overall
> picture of how the tools were designed to work and be used together.
>
> The envisioned usage model is that a user would first start out by
> creating a new BSP for a given architecture by invoking the 'yocto-bsp
> create' command. Choosing the architecture essentially chooses a set
> of files that the BSP will be based on. This set of files is a
> combination of a set of 'common' files that all BSPs inherit and a set
> of architecture-specific files that match the specified architecture.
>
> Neither set of files can be used unmodified on their own as simple
> static copies of the originals in order to create a working BSP for a
> real machine - many of those files need to be tailored for the
> hardware using information supplied by the user. To accomplish this
> in an automated way, both the specific portions of the BSP files that
> need to be filled in with user input, and the means of (optionally)
> getting that input from the user are included as in-line annotations
> to the BSP files themselves using a simple templating scheme based on
> conventions used by web2py. Essentially, in the default case, the
> templating system is used to generate a series of text-based prompts
> that ask the user questions about the BSP being created. The data
> supplied by the user is then used to fill in the missing information
> in the BSP files. The user can also choose to specify all the
> information needed directly to the 'yocto-bsp create' command
> invocation itself; the 'yocto-bsp list' command exists to provide the
> user the list of 'properties' and enumerations of possible values for
> those properties, which taken together represent the data items needed
> to generate the BSP. The user can even more directly supply the
> information as a json object representing a serialized python
> dictionary, which is what the templating interface uses natively to
> represent properties and values needed for BSP generation.
>
> The end result is a Yocto-compliant BSP that should be directly
> buildable by the Yocto build system. The generated BSP is completely
> self-contained - it includes not only the BSP metadata, but also the
> necessary kernel components, which are normally for BSPs contained in
> the linux-yocto repo's meta and machine branches for the BSP. In
> order to help make the tool more efficient for the user, the tool does
> everything it needs to locally by initially using the linux-yocto
> SRC_URI; once ready, the user can then migrate the kernel components
> to an actual linux-yocto repo by using the 'yocto-kernel migrate'
> command. Once that's done, the user is in a position to submit both
> the BSP metadata and the linux-yocto commits upstream.
>
> Once a BSP is created, the job of 'yocto-bsp' is done (in the initial
> version of the tools - it may have sub-commands added later that could
> possibly manage the non-kernel aspects of a BSP) and the
> 'yocto-kernel' command can be used going forward to maintain the
> kernel portion of a BSP. 'yocto-kernel' allows both bare kernel
> config items and higher-level 'kernel features' to be created, used
> and removed, patches to be added to a BSP's machine branch, and the
> kernel changes to a BSP migrated to a linux-yocto repo. These
> have typically been difficult tasks for many users; the hope is that
> 'yocto-kernel' will make those tasks more manageable for everyone.
>
> The sections below describe each tool in detail.
>
> Following those sections is a detailed description of the templating
> mechanism, useful for anyone who needs to create templates for
> creating new BSPs using the 'yocto-bsp create' tool.
>
>
> The top-level bsp-creation tool, 'yocto-bsp'
> ============================================
>
> All of the functionality of the Yocto bsp-creation tool is available
> via the single 'yocto-bsp' command. Access to the specific functions
> provided by the tool is accomplished by specifying sub-commands to the
> main 'yocto-bsp' command. The design of the command-line interface is
> based on similar interfaces provided by git and perf, which is
> familiar to many users and is easily extensible.
>
> Usage help is always available for the command itself as well as for
> all sub-commands. Executing the top-level command displays a list of
> the available sub-commands, with descriptions:
>
> $ yocto-bsp
>
> usage: yocto-bsp [--version] [--help] COMMAND [ARGS]
>
> The most commonly used 'yocto-bsp' commands are:
> create Create a new Yocto BSP
> list List available values for options and BSP properties
>
> See 'yocto-bsp help COMMAND' for more information on a specific command.
>
>
> 'yocto-bsp create' - Create a new Yocto BSP
> -------------------------------------------
>
> NAME
> yocto-bsp-create - Create a new Yocto BSP
>
> SYNOPSIS
> yocto-bsp create <bsp-name> <karch> <kmachine>
> [--property <NAME:VAL> | -p <NAME:VAL>]
>
> DESCRIPTION
> This command creates a Yocto BSP based on the specified
> parameters. The new BSP will be a new Yocto BSP layer contained
> within the top-level directory specified as bsp-name i.e. not
> 'meta-bsp-name'. The value of the 'karch' parameter determines
> the set of files that will be generated for the BSP, along with
> the specific set of 'properties' that will be used to fill out the
> BSP-specific portions of the BSP.
>
> The BSP-specific properties that define the values that will be
> used to generate a particular BSP can be specified on the
> comman-line using the --property option for each property, which
> is a colon-separated name:value pair.
>
> The set of properties and enumerations of their possible values
> can be displayed using the 'yocto-bsp list' command.
>
> It isn't necessary however for the user to explicitly specify the
> property values using --property options - if none are specified
> in the 'yocto-bsp create' invocation, the user will be
> interactively prompted for each of the required property values,
> which will then be used as values for BSP generation.
>
> The 'kmachine' parameter names the 'machine branch' that the BSP's
> machine branch will be based on; the list of branches meaningful
> for that purpose can also be listed using the 'yocto-bsp list'
> command.
>
> The 'yocto-bsp list' command can also be used to list the possible
> values for the 'karch' parameter.
>
> An typical example that would query the user for property values
> would be:
>
> $ yocto-bsp create meta-foo i386 yocto/standard/common-pc/foo
>
> Because the user hasn't specified any values for the BSP
> properties, yocto-bsp will query the user for each of the
> unspecified property values required to create the BSP i.e. the
> properties listed by 'yocto-bsp list properties'.
>
> Here's an example using explicitly-specified properties:
>
> $ yocto-bsp create meta-foo i386 yocto/standard/common-pc/foo
> --property tunefile:tune-atom --property touchscreen:true
> --property kfeature:cfg/smp [...]
>
>
> 'yocto-bsp list' - List available values for options and BSP properties
> -----------------------------------------------------------------------
>
> NAME
> yocto-bsp-list - List available values for options and BSP properties
>
> SYNOPSIS
> yocto bsp list <karch | kmachine>
> yocto bsp list <karch> properties
> yocto bsp list <karch> property <xxx>
>
> DESCRIPTION
> This command enumerates the complete set of possible values for a
> specified option or property needed by the BSP creation process.
>
> The first form enumerates all the possible values that exist and
> can be specified for the 'karch' or 'kmachine' parameters to the
> 'yocto bsp create' command. Example output for those options:
>
> $ yocto-bsp list karch
> i386
> x86_64
> arm
> powerpc
> mips
>
> $ yocto-bsp list kmachine
> yocto
> yocto/standard
> yocto/standard/common-pc
> yocto/standard/common-pc-64
> yocto/standard/preempt-rt
>
> The second form enumerates all the possible properties that exist
> and must have values specified for them in the 'yocto bsp create'
> command for the given 'karch'. Example output for that command:
>
> $ yocto-bsp list i386 properties
> "machine": boolean,
> "kmachine": choice,
> "tunefile": choicelist,
> "smp": boolean,
> "xserver": choicelist
> ["xserver-choice": choice]
> "kfeatures": choicelist
> ["kfeature": choice]
> "touchscreen": boolean,
> "keyboard": boolean
>
> The third form enumerates all the possible values that exist and
> can be specified for any of the enumerable properties of the given
> 'karch' in the 'yocto bsp create' command. Example output for
> those properties:
>
> $ yocto-bsp list i386 property xserver
> xserver-vesa
> xserver-i915
> xserver-emgd
>
> $ yocto-bsp list i386 property kfeatures
> features/taskstats
> cfg/sound
>
> $ yocto-bsp list i386 property tunefile
> i386
> tune-atom
> tune-core2
>
> $ yocto-bsp list x86_64 property tunefile
> x86_64
> tune-x86_64
> tune-corei7
> tune-ivb
>
> $ yocto-bsp list powerpc property tunefile
> powerpc
> tune-ppc603e
> tune-ppce300c2
> tune-ppce500
> tune-ppce500mc
> tune-ppce500v2
> tune-ppce5500-32b
> tune-ppce5500-64b
>
> $ yocto-bsp list arm property tunefile
> arm
> tune-arm1136jf-s
> tune-arm920t
> tune-arm926ejs
> tune-arm9tdmi
> tune-armv7
>
> $ yocto-bsp list mips property tunefile
> mips
> tune-mips32
>
>
> The top-level kernel usability tool, 'yocto-kernel'
> ===================================================
>
> All of the functionality of the Yocto kernel usability tool is
> available via the single 'yocto-kernel' command. Access to the
> specific functions provided by the tool is accomplished by specifying
> sub-commands to the main 'yocto-kernel' command. The design of the
> command-line interface is based on similar interfaces provided by git
> and perf, which is familiar to many users and is easily extensible.
>
> Usage help is always available for the command itself as well as for
> all sub-commands. Executing the top-level command displays a list of
> the available sub-commands, with descriptions:
>
> $ yocto-kernel
>
> usage: yocto-kernel [--version] [--help] COMMAND [ARGS]
>
> The most commonly used 'yocto-kernel' commands are:
> feature-list List available Yocto KERNEL_FEATUREs
> feature-define Define a new Yocto KERNEL_FEATURE
> feature-add Add Yocto KERNEL_FEATUREs to a BSP
> feature-rm Remove Yocto KERNEL_FEATUREs from a BSP
> config-list List the modifiable set of bare kernel config options for a BSP
> config-add Add or modify bare kernel config options for a BSP
> config-rm Remove bare kernel config options from a BSP
> patch-add Patch the Yocto kernel for a specific BSP
> migrate Move local kernel config/patches to a git repo
>
> See 'yocto-kernel help COMMAND' for more information on a specific command.
>
>
> 'yocto-kernel feature list' - List available Yocto KERNEL_FEATUREs
> ------------------------------------------------------------------
>
> NAME
> yocto-kernel-feature-list - List available Yocto KERNEL_FEATUREs
>
> SYNOPSIS
> yocto-kernel feature list <bsp-name> --used
>
> DESCRIPTION
> This command lists all the kernel features available to a BSP.
> This includes any features temporarily attached to the named BSP
> via the 'feature define' command and thus the reason this command
> takes bsp-name as a parameter.
>
> If the --used param is specified, only the KERNEL_FEATURES used by
> the BSP are listed.
>
>
> 'yocto-kernel feature define' - Define a new Yocto KERNEL_FEATURE
> -----------------------------------------------------------------
>
> NAME
> yocto-kernel-feature-define - Define a new Yocto KERNEL_FEATURE
>
> SYNOPSIS
> yocto-kernel feature define <bsp-name> <feature-name> <feature-desc>
> [<CONFIG_XXX=x> ...] [--dirname <dirname>] [--policy <hw|non-hw>]
>
> DESCRIPTION
> This command defines a new kernel feature, resulting in the
> addition of two new files for feature xxx to the named BSP's
> SRC_URI. If dirname is specified, the feature files will be named
> features_dirname_xxx.cfg and features_dirname_xxx.scc. Otherwise,
> the features will be named cfg_xxx.cfg and cfg_xxx.scc. The idea
> behind the naming is that the names reflect the user's intent to
> have the feature migrated to either the kernel-cache/cfg or
> kernel-cache/features directory if/when the kernel portion of the
> BSP is migrated into the linux-yocto kernel repo.
>
> The policy stating whether the new feature is considered to be a
> hardware or a non-hardware feature also needs to be specified in
> order to create a new feature, and a description of the new
> feature is also required; this will be added as a comment to the
> .scc file.
>
> NOTE: Although features are not BSP-specific, until the user has
> gone through the steps of migrating a feature to the linux-yocto
> repo, it needs to live somewhere accessible to the BSP metadata,
> which logically would be the kernel SRC_URI for the BSP using it.
>
> NOTE: It's up to the user to determine whether or not the config
> options for a feature being added make sense or not - this command
> does no sanity checking or verification of any kind to ensure that
> the config options contained in a kernel feature really make sense
> and will actually be set in in the final config. For example, if
> a config option depends on other config options, it will be turned
> off by kconfig if the other options aren't set correctly.
>
>
> 'yocto-kernel feature add' - Add Yocto KERNEL_FEATUREs to a BSP
> ---------------------------------------------------------------
>
> NAME
> yocto-kernel-feature-add - Add Yocto KERNEL_FEATUREs to a BSP
>
> SYNOPSIS
> yocto-kernel feature add <bsp-name> [<FEATURE> ...]
>
> DESCRIPTION
>
> This command adds one or more kernel features to a BSP. The set
> of features available to be added by this command for a BSP can be
> found via the 'yocto-kernel feature list' command. This command
> essentially adds a new machine-specific KERNEL_FEATURE_append to
> the linux-yocto .bbappend for the BSP.
>
>
> 'yocto-kernel feature rm' - Remove Yocto KERNEL_FEATUREs from a BSP
> -------------------------------------------------------------------
>
> NAME
> yocto-kernel-feature-rm - Remove Yocto KERNEL_FEATUREs from a BSP
>
> SYNOPSIS
> yocto-kernel feature rm <bsp-name> [<FEATURE> ...]
>
> DESCRIPTION
>
> This command removes one or more kernel features from a BSP. The
> set of features available to be removed by this command for a BSP
> can be found via the 'yocto-kernel feature list' command, if the
> --used option to that command is specified. This command
> essentially removes a machine-specific KERNEL_FEATURE from the
> linux-yocto .bbappend for the BSP.
>
>
> 'yocto-kernel config list' - List a BSP's modifiable config options
> -------------------------------------------------------------------
>
> NAME
> yocto-kernel-config-list - List the modifiable set of bare kernel
> config options for a BSP
>
> SYNOPSIS
> yocto-kernel config list <bsp-name>
>
> DESCRIPTION
> This command lists the 'modifiable' config items for a BSP
> i.e. the items which are eligible for modification or removal by
> other yocto-kernel commands.
>
> 'modifiable' config items are the config items contained a BSP's
> foo.cfg base config.
>
>
> 'yocto-kernel config add' - Add or modify a BSP's kernel config options
> -----------------------------------------------------------------------
>
> NAME
> yocto-kernel-config-add - Add or modify bare kernel config options
> for a BSP
>
> SYNOPSIS
> yocto-kernel config add <bsp-name> [<CONFIG_XXX=x> ...]
>
> DESCRIPTION
> This command adds one or more CONFIG_XXX=x items to a BSP's
> foo.cfg base config. If a config item is already present, the new
> value will simply overwrite the old value.
>
> NOTE: It's up to the user to determine whether or not the config
> options being added make sense or not - this command does no
> sanity checking or verification of any kind to ensure that a
> config option really makes sense and will actually be set in in
> the final config. For example, if a config option depends on
> other config options, it will be turned off by kconfig if the
> other options aren't set correctly.
>
>
> 'yocto-kernel config rm' - Remove BSP kernel config options
> -----------------------------------------------------------
>
> NAME
> yocto-kernel-config-rm - Remove bare kernel config options from a
> BSP
>
> SYNOPSIS
> yocto-kernel config rm <bsp-name> [<CONFIG_XXX> ...]
>
> DESCRIPTION
> This command removes (turns off) one or more CONFIG_XXX items to a
> BSP's foo.cfg base config.
>
> The set of config items available to be removed by this command
> for a BSP can be found via the 'yocto-kernel config list'
> command.
>
>
> 'yocto-kernel patch add' - Patch the Yocto kernel for a specific BSP
> --------------------------------------------------------------------
>
> NAME
> yocto-kernel-patch-add - Patch the Yocto kernel for a specific BSP
>
> SYNOPSIS
> yocto-kernel patch add <bsp-name> [<PATCH> ...]
>
> DESCRIPTION
> This command adds one or more patches to a BSP's machine branch.
> The patch will be added to the BSP's linux-yocto kernel SRC_URI
> and will be guaranteed to be applied in the order specified.
>
> NOTE: It's up to the user to determine whether or not the patches
> being added makes sense or not - this command does no sanity
> checking or verification of any kind to ensure that a patch can
> actually be applied to the BSP's kernel branch; it's assumed that
> the user has already done that.
>
>
> 'yocto-kernel migrate' - Move local kernel config/patches to a git repo
> -----------------------------------------------------------------------
>
> NAME
> yocto-kernel-migrate - Move local kernel config/patches to a git repo
>
> SYNOPSIS
> yocto-kernel migrate <bsp-name> <local linux-yocto repo>
>
> DESCRIPTION
> This command converts the kernel configuration changes, patches,
> and features collected in a BSP's linux-yocto SRC_URI into an
> equivalent set of commits in a local linux-yocto repo. On
> success, the local files will named in the SRC_URI will be
> removed, and the linux-yocto .bbappend will be fixed up to point
> to the new repo.
>
> NOTE: The user is responsible for submitting the generated commits
> upstream and subsequently modifying the kernel .bbappend to
> use the upsteam kernel repo.
>
>
> The design of the 'yocto-bsp create' templating mechanism
> ==========================================================
>
> This section describes the details of the templating mechanism used by
> 'yocto-bsp' to generate BSPs.
>
> Generating a new BSP is basically a templating exercise. A given BSP
> is comprised of a combination of a set of common files that all BSPs
> share along with a set of files specific to a given BSP type. Many of
> the files in both sets need to be configured according to the needs of
> the BSP; this also extends to the naming and existence of the files
> themselves.
>
> The templating scheme used in the Yocto tool is partly inspired by
> ideas from web2py, but adapted to a more general input mechanism for
> the specific problems posed by Yocto.
>
> There are no special files, just marked-up BSP metadata and kernel
> configuration, .scc, and patch files.
>
> The Yocto BSP markup is meant to be extremely simple and limited to
> pretty much this initial set of items; it is however designed to be
> extensible enough to cover any future need.
>
> Anything within double braces is considered to be a Yocto BSP markup
> tag:
>
> {{ foo }}
>
> Anything within double braces and starting with {{= is considered to
> implement simple variable replacement, where variables are by default
> simply keys in a dictionary representing the BSP and containing values
> for the whole BSP:
>
> {{=machine}}
>
> The above means that the whole expression will be replaced with the
> value of the 'machine' entry in the BSP dictionary. The syntax is
> actually shorthand for the equivalent Python code for accessing the
> value of a given key, but isn't restricted to that e.g. it could
> reference a local variable defined in a previous tag, for instance.
>
> In every other case, the 'yocto-bsp' tool will pre-process the
> contents looking for certain keywords at the beginning. The rest it
> will treat as Python code. As of now, there's only one keyword that's
> meaningful:
>
> {{ input }}
>
> which signals that anything contained is input-related. The
> 'yocto-bsp' implementation will use this information to prompt the
> user for input. Under certain conditions e.g. when the user has
> already supplied this information via a file for instance, anything
> contained in this is considered a no-op and ignored.
>
> Anything else is considered to be Python code and will be directly
> incorporated into the BSP generation code.
>
> {{ if keyboard == False: }}
>
> Note also that any of the above can also be used in file and directory
> names, which will also be processed in the same way; this allows us to
> avoid extraneous files is consistent with the rest of the scheme.
>
> Internally, the way things are envisioned to work is that the tool
> will first read in the set of files that makes up the BSP for the
> chosen architecture. It will then pre-process all the 'input' tags it
> finds and use that to prepare a series of text-based prompts to
> retrieve the needed information from the user. The end result of
> presenting all that to the user will be a completd BSP dictionary
> containing all the key/value pairs needed to fill in the BSP. Once
> that information is available, it will make a second pass over the
> remaining tags and fill them in with the Python code needed to
> retrieve and replace the remaining tags with their corresponding
> values from the BSP dictionary (or any other source possibly
> introduced by other tags). The end result of this second pass will be
> executed as Python code to produce the final BSP metadata.
>
> For the first version, the 'input' tags will be used to construct a
> series of text-based questions for the user. The design, however,
> should make it relatively trivial to create an application that could
> create real GUI that would do the same thing at some later point. For
> example, such an application could use the input tags to generate a
> web-based form, which would return the results gathered from the user
> to a web service in the form of a json blob that could directly be
> converted into the BSP Python dictionary needed. e.g. the browser
> would return something like this to the web service, which would in
> turn send it to the 'bsp-create' library function to create the final
> BSP.
>
> {
> "machine": "foo",
> "kmachine": "/yocto/standard/base",
> "tunefile": "tune-atom",
> "smp": true,
> "xserver": {
> "xserver-choice": "xserver-i915",
> },
> "kfeatures": [
> "kfeature": "features/taskstats",
> "kfeature": "cfg/sound",
> ]
> },
> touchscreen: false,
> keyboard: true
> }
>
> Below are some concrete examples of specific tags and how they're used.
>
> Here's a simple example of markup that describes information needed
> from the user about touchscreen and keyboard capabilities in the
> formfactor 'machconfig' file:
>
> # Assume a USB mouse and keyboard are connected
> {{ input type:"boolean" name:"touchscreen" msg:"Does your BSP have a touchscreen?" }}
> HAVE_TOUCHSCREEN={{=touchscreen}}
> {{ input type:"boolean" name:"keyboard" msg:"Does your BSP have a keyboard?" }}
> HAVE_KEYBOARD={{=keyboard}}
>
> For instance, the touchscreen 'input' tag above describes everything a
> UI would need in order to retrieve an answer from the user as to
> whether the hardware in question has a touchscreen or not.
>
> The 'type' key comprises a small set of possible input types that
> include things like 'boolean' (yes/no) choices and lists of choices
> (type:"choicelist").
>
> The 'name' key provides a means for later markup to access the value
> of the variable containing the user's answer for that input. For
> example, the name:"touchscreen" simply references the value in the BSP
> dictionary that was filled in with the user's response to the question
> which was formed and presented to the user using the "msg" tag in the
> input tag.
>
> Below the input tag is the actual content that will be rendered into
> the 'machconfig' file, but with the {{=}} portion replaced with the
> 'touchscreen' value supplied by the user i.e.:
>
> HAVE_TOUCHSCREEN={{=touchscreen}}
>
> will be rendered as:
>
> HAVE_TOUCHSCREEN=0
>
> Assuming the user answered 'no'.
>
> Another interesting thing to note about this example is that in Yocto
> BSPs, the machconfig files are contained within a filesystem location
> that includes the machine name in the path e.g.
>
> recipes-bsp/formfactor/formfactor/foo/machconfig
>
> To accomplish this, the directory names themselves contain markup that
> gets processed and replaced by the tool:
>
> recipes-bsp/formfactor/formfactor/{{=machine}}/machconfig
>
> Here, the {{=machine} directory is renamed to /foo to match the
> machine name of the BSP.
>
> Here's a more involved example, markup included in the
> i386/conf/{{=machine}}.conf file:
>
> {{ input type:"boolean" name:"xserver" msg:"Do you need support for X?" }}
> {{ if xserver: }}
> XSERVER ?= "${XSERVER_IA32_BASE} \
> ${XSERVER_IA32_EXT} \
> {{ input type:"choicelist" name:"xserver_choice" msg:"Please select an xserver for this machine" }}
> {{ input type:"choice" val:"xserver-vesa" msg:"VESA xserver support" }}
> ${XSERVER_IA32_VESA} \
> {{ input type:"choice" val:"xserver-emgd" msg:"EMGD xserver support (proprietary)" }}
> ${XSERVER_IA32_EMGD} \
> {{ input type:"choice" val:"xserver-i915" desc:"i915 xserver support" }}
> ${XSERVER_IA32_I915} \
> "
>
> In this case, we start off with a boolean 'input' question asking if
> the BSP needs to support X. The following line is straight Python
> code which essentially skips the following sections (until a blank
> line) if the user's answer was 'no', or if the answer was 'yes',
> creates a further query to the user asking which xserver type to use.
> In the latter case, the input type is a list of choices resulting in
> the values specified by the 'val' tag appearing as the value for the
> 'xserver_choice' dictionary key.
>
> Code further down in the .conf file can also make use of this
> information to include or exclude content based on the value of the
> xserver_choice variable:
>
> {{ if xserver_choice == "xserver-emgd": }}
> PREFERRED_VERSION_xserver-xorg ?= "1.9.3"
> PREFERRED_VERSION_mesa-dri ?= "7.11"
> PREFERRED_VERSION_emgd-driver-bin ?= "1.8"
>
> {{ if xserver_choice == "xserver-vesa" or xserver_choice == "xserver-emgd": }}
> APPEND += "video=vesafb vga=0x318"
>
> Note again that the tags used to determine whether or not to include
> content simply contain Python code used to directly implement the
> needed logic. Not only is there no need for the user to learn a
> special-purpose language, but gives the tool the flexibility to add
> essentially any kind of processing logic needed in the future.
>
> In the case of xserver choices, we also need to use the value of
> xserver_choice to determine which xorg.conf file to add to the BSP.
> Again, consistent with the rest of the markup processing, this is
> seamlessly accomplished by adding markup to the filenames themselves:
>
> recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver_choice == xserver_emgd: }} xorg.conf
> recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver_choice == xserver_vesa: }} xorg.conf
> recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver_choice == xserver_i915: }} xorg.conf
>
> In this case, the filenames actually contain Python code that
> describes the conditions needed to include those particular files.
>
> Here's another example, again making use of general-purpose Python in
> the markup to generate an open-ended list of choices for the user, in
> this case in linux-yocto_3.0.bbappend, to generate a list of available
> KERNEL_FEATURES for the user to choose from:
>
> {{ input type:"checklist" name:"kfeatures" gen:"fetch-kernel-features()" msg:"Please choose the kernel features to use in this BSP on" }}
> {{ for kfeature in kfeatures: }}
> KERNEL_FEATURES_append_{{=machine}} += {{=kfeature}}
>
> In this case an 'input' tag calls an internally-defined yocto-bsp
> function named fetch_kernel_feature() that generates the list of
> available KERNEL_FEATURES. The output of that call is a Python list
> containing the features, which is assigned to the 'kfeatures'
> variable. The next tag contains Python code that simply iterates over
> that list, and uses the value of the current feature, 'kfeature', to
> append the chosen kernel features to the list of features used by the
> BSP.
>
> Finally, because the kernel components are actually part of the BSP
> metadata, the user-specified values are also available in the
> processing of those files. For instance, in the below case, we can
> use the xserver_choice to include or exclude specific kernel
> configuration settings in the
> recipes-kernel/linux/linux-yocto/{{=machine}}.scc file associated with
> the BSP:
>
> {{ if xserver_choice == "xserver-emgd": }}
> include features/drm-emgd/drm-emgd.scc
>
> ---
>
> For reference, here's an example of how the tools are expected to be
> laid out. Everything is contained within a 'bsp-tools' directory
> under '/scripts'. The 'common' portion, containing the files common
> to every BSP are:
>
> bsp-tools/
> bsp-tools/yocto-bsp.py
> bsp-tools/yocto-kernel.py
> bsp-tools/bsp-cache
> bsp-tools/bsp-cache/target
> bsp-tools/bsp-cache/target/arch
> bsp-tools/bsp-cache/target/arch/common
> bsp-tools/bsp-cache/target/arch/common/README.sources
> bsp-tools/bsp-cache/target/arch/common/binary
> bsp-tools/bsp-cache/target/arch/common/binary/.gitignore
> bsp-tools/bsp-cache/target/arch/common/recipes-kernel
> bsp-tools/bsp-cache/target/arch/common/recipes-kernel/linux
> bsp-tools/bsp-cache/target/arch/common/COPYING.MIT
> bsp-tools/bsp-cache/target/arch/common/README
> bsp-tools/bsp-cache/target/arch/common/recipes-graphics
> bsp-tools/bsp-cache/target/arch/common/recipes-graphics/xorg-xserver
> bsp-tools/bsp-cache/target/arch/common/recipes-graphics/xorg-xserver/{{ if xserver: }} xserver-xf86-config_0.1.bbappend
> bsp-tools/bsp-cache/target/arch/common/conf
> bsp-tools/bsp-cache/target/arch/common/conf/layer.conf
> bsp-tools/bsp-cache/target/arch/common/recipes-core
> bsp-tools/bsp-cache/target/arch/common/recipes-core/tasks
> bsp-tools/bsp-cache/target/arch/common/recipes-core/tasks/task-core-tools.bbappend
> bsp-tools/bsp-cache/target/arch/common/recipes-bsp
> bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor
> bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor_0.0.bbappend
> bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor
> bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor/{{=machine}}
> bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor/{{=machine}}/machconfig
>
> The BSP-specific files are contained in directories named for the
> architecture. In this case, only the i386 files are expanded as an
> example.
>
> bsp-tools/bsp-cache/target/arch/arm
> bsp-tools/bsp-cache/target/arch/powerpc
> bsp-tools/bsp-cache/target/arch/i386
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt_3.0.bbappend
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto_3.0.bbappend
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}.cfg
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}.scc
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}-preempt-rt.scc
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}-standard.scc
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}.cfg
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}.scc
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}-preempt-rt.scc
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}-standard.scc
> bsp-tools/bsp-cache/target/arch/i386/recipes-graphics
> bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config
> bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}
> bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver-choice == xserver-emgd: }} xorg.conf
> bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver-choice == xserver-vesa: }} xorg.conf
> bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver-choice == xserver-i915: }} xorg.conf
> bsp-tools/bsp-cache/target/arch/i386/conf
> bsp-tools/bsp-cache/target/arch/i386/conf/machine
> bsp-tools/bsp-cache/target/arch/i386/conf/machine/{{=machine}}.conf
> bsp-tools/bsp-cache/target/arch/mips
> bsp-tools/bsp-cache/target/arch/x86_64
>
>
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: RFC: Yocto BSP and kernel usability tools
2011-12-28 0:36 ` McClintock Matthew-B29882
@ 2011-12-29 17:32 ` Tom Zanussi
0 siblings, 0 replies; 7+ messages in thread
From: Tom Zanussi @ 2011-12-29 17:32 UTC (permalink / raw)
To: McClintock Matthew-B29882; +Cc: yocto@yoctoproject.org
On Wed, 2011-12-28 at 00:36 +0000, McClintock Matthew-B29882 wrote:
> (This probably belongs somewhere inline but I got lost on the long email)
>
> A lot of times users would already have a git tree containing linux
> and/or u-boot already working for their boards. Perhaps you could let
> the user point at external linux and/or u-boot trees with defconfigs,
> u-boot configs, device trees, etc for their target BSP?
>
Yes, definitely, I think that makes sense and should be easily doable
using the templating mechanism e.g. using the 'choicelist' input method
the user could be presented with a choice of kernel recipes to use
similarly to how xserver-choice is defined in the examples.
Choosing for example the Yocto 3.0 kernel would select what's already
been described, but additional choices could select alternate kernels
and allow the user to point those recipes at specific git repos or
tarballs, attach defconfigs, .dts files or whatever to those kernels'
SRC_URIs. There would need to be a way for the user to specify the
files to include, etc, but that could also be easily done by adding some
python code for that purpose (or maybe that's already covered by the
code that adds patches to a SRC_URI).
In any case, I haven't thought about all the details yet but I think the
problem can be generalized even further and those details will come out
in the implementation. But as a general answer to your question, I
think that yes, it is possible and desirable, and I'll at least try to
have a simple example of it implemented in the first pass.
Tom
> -M
>
> On Mon, Dec 19, 2011 at 10:55 AM, Tom Zanussi <tom.zanussi@intel.com> wrote:
> > Hi,
> >
> > This is what I hope to implement in the M2 timeframe.
> >
> > Any and all comments welcome...
> >
> > Thanks,
> >
> > Tom
> >
> > ---
> >
[snip]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: RFC: Yocto BSP and kernel usability tools
2011-12-19 16:55 RFC: Yocto BSP and kernel usability tools Tom Zanussi
2011-12-22 20:16 ` Bruce Ashfield
2011-12-28 0:36 ` McClintock Matthew-B29882
@ 2012-01-31 22:52 ` Darren Hart
2012-02-01 0:07 ` Tom Zanussi
2 siblings, 1 reply; 7+ messages in thread
From: Darren Hart @ 2012-01-31 22:52 UTC (permalink / raw)
To: Tom Zanussi; +Cc: yocto@yoctoproject.org
Hi Tom,
This is really late in coming, my apologies. This looks to be the
beginnings of a very powerful set of tools, and the detail you started
with should help ensure a successful initial version. Nice work.
Some general questions that don't belong in any one particular place inline:
o You mention the use of json files. Is there a web UI planned for this?
Or did you just decide that the requirements for the UI were the same as
a web UI so you opted to use an established format rather than create a
new one?
o As a follow-on to the previous, it is always nice if the input files
to tools such as this can be easily human readable (and IMO this
excludes things like XML and serialized python objects). A simple INI
type config seems like it might be more accessible and python has a
module for this. I imagine you've considered this, I'm curious as to
your rationale for the method outlined below.
o I have some concern about yet another description format of BSPs and
essentially kernel configuration. We already ask users to think in terms
of SCC and CFG fragments, rather than defconfigs. This effectively asks
users to start thinking in terms of json name value pairs to map to
bitbake syntax to describe scc and cfg fragments so we can assemble a
.config. Having the tools and structure in place eliminates a lot of the
guesswork, so in the end I think it is probably worth it ... I just
worry about so many levels of abstraction - and I do not have a better
alternative in mind.
o I see the tools for KMACHINE but nothing for KBRANCH. The distinction
here is subtle and not described in any of our documentations (BSP
guide, Kernel Arch and Usage, or Yocto Developer guide). Ideally we use
standard/default/base for new BSPs and only add the meta-data to the
linux-yocto kernel. Is this workflow supported? I ask because I'm
failing to accomplish this with the sys940x and seem to have to add the
sys940x branch in order for kernel_configme to pick up the sys940x meta
data.
--
Darren
On 12/19/2011 08:55 AM, Tom Zanussi wrote:
> Hi,
>
> This is what I hope to implement in the M2 timeframe.
>
> Any and all comments welcome...
>
> Thanks,
>
> Tom
>
> ---
>
> Getting Yocto running on a new board is often a prerequisite task to
> the real task of interest for new users, which is creating new
> applications designed to run on the board, or creating new
> board-specific embedded systems. In other words, most users just want
> to get the hardware up and running and not have to think too much
> about it before moving on to what they're really interested in, which
> is the stuff running on top of it.
>
> While there really can't be any such thing as a magic-bullet
> application that would do that automatically i.e. generate and
> maintain a guaranteed-working Yocto BSP for any given piece of
> hardware, it should at least be possible to provide a tool that would
> generate a base BSP usable as a starting point for that, and which
> would additionally allow the user to tweak various BSP settings,
> including kernel configuration options and simple kernel patches.
> Such a tool should be usable with only a minimal understanding of the
> Yocto build system and metadata, and with no knowledge at all of
> kernel internals or the Yocto kernel recipe bindings, or of the
> details of the internals of yocto kernel repositories, or in fact any
> knowledge of git or any other specific build-related tool.
>
> This document provides the design for such a tool. At a high level,
> the goal of the tool is to provide a user the means of creating a new
> Yocto BSP from which it should be possible (assuming a perfect
> run-time outcome, admittedly unlikely on a first pass) to submit a
> 'pull request' for inclusion into any repo(s) that would accept an
> 'official' Yocto BSP. For example, the output of the tool should
> produce metadata and kernel configuration that would be directly
> mergeable into the meta-intel and the linux-yocto kernel repos. The
> tool should also allow the BSP maintainer to afterwards at any time
> make changes to the kernel configuration and/or submit kernel patches
> for the BSP.
>
> Note that although knowledge of the Yocto kernel or kernel internals
> is not expected, the user is expected to know the basics of how to
> deal with the linux kernel at the 'user' level, specifically
> understanding and determining the specific kernel config options
> needed for the user's BSP.
>
> Specifically, the tool will allow for the following high-level tasks,
> which are discussed in more detail below:
>
> - create a new BSP from scratch from a set of user-settable parameters
> - manage individual kernel config settings
> - create and use groups of kernel config settings as KERNEL_FEATUREs
> - apply patches to a BSP's machine branch
>
> The following capabilities are specifically not provided by the tool:
>
> - help in determining a 'correct' set of kernel config options to set.
> The tool assumes the user already knows that, either through diffs
> of .config files before and after using 'bitbake -c menuconfig', or
> some other means.
>
> - modification of the BSP metadata after BSP creation. The tool
> provides an initial 'write-once' BSP-generation capability, but
> doesn't allow it to be read in and modified after the initial pass.
> The user will have to make further modifications manually. The one
> exception to this is kernel features, which can be added to an
> already existing kernel bbappend file.
>
> - a guarantee or even expectation that the generated BSP will work on
> the actual hardware it's targeted for - it's highly unlikely the BSP
> will work 'out-of-the-box' and it's the developer's reponsibility to
> do the often hard work of figuring out what settings and/or code are
> actually needed to get the hardware to work as expected. The goal
> of the tool is to make that job easier, not to actually do that job.
>
> Also, though it's not explicitly a requirement of this tool, the
> design should be sufficiently modular to allow for its participation
> in generic logical pipelines. For instance, while the tool may
> present to the user a text-based UI for gathering information, it
> should also be able to operate without any kind of user interaction
> and retrieve the data it needs from a file, for instance e.g.:
>
> $ yocto-bsp create < x86-input.json
>
> Another example that the design should not preclude would be its use
> at the end of a bsp-import pipeline e.g.:
>
> $ cat 3rdparty.bsp | 3rdparty2yocto | yocto-bsp create
>
> The tool will initially be implemented as a set of command-line tools
> which will essentially be thin layers on top of a modular Python API;
> future tools may be GUI-based but would make use of the functionality
> exposed through the same API. There might also be an opportunity for
> future integration into existing tools such as hob; this isn't
> explicitly catered for in the current design, but it should be
> possible to do that based either on the functionality provided by the
> command-line interface or equivalently via the Python module API.
>
>
> A Quick Overview
> ----------------
>
> The initial set of tools will comprise 'yocto-bsp', for creating Yocto
> BSPs, and 'yocto-kernel', for BSP and BSP-related Yocto kernel
> maintenance tasks on existing BSPs.
>
> The interface and detailed description of each tool are described
> below, and the underlying templating mechanism is described following
> that, but it would be good for context to first try to give an overall
> picture of how the tools were designed to work and be used together.
>
> The envisioned usage model is that a user would first start out by
> creating a new BSP for a given architecture by invoking the 'yocto-bsp
> create' command. Choosing the architecture essentially chooses a set
> of files that the BSP will be based on. This set of files is a
> combination of a set of 'common' files that all BSPs inherit and a set
> of architecture-specific files that match the specified architecture.
>
> Neither set of files can be used unmodified on their own as simple
> static copies of the originals in order to create a working BSP for a
> real machine - many of those files need to be tailored for the
> hardware using information supplied by the user. To accomplish this
> in an automated way, both the specific portions of the BSP files that
> need to be filled in with user input, and the means of (optionally)
> getting that input from the user are included as in-line annotations
> to the BSP files themselves using a simple templating scheme based on
> conventions used by web2py. Essentially, in the default case, the
> templating system is used to generate a series of text-based prompts
> that ask the user questions about the BSP being created. The data
> supplied by the user is then used to fill in the missing information
> in the BSP files. The user can also choose to specify all the
> information needed directly to the 'yocto-bsp create' command
> invocation itself; the 'yocto-bsp list' command exists to provide the
> user the list of 'properties' and enumerations of possible values for
> those properties, which taken together represent the data items needed
> to generate the BSP. The user can even more directly supply the
> information as a json object representing a serialized python
> dictionary, which is what the templating interface uses natively to
> represent properties and values needed for BSP generation.
>
> The end result is a Yocto-compliant BSP that should be directly
> buildable by the Yocto build system. The generated BSP is completely
> self-contained - it includes not only the BSP metadata, but also the
> necessary kernel components, which are normally for BSPs contained in
> the linux-yocto repo's meta and machine branches for the BSP. In
> order to help make the tool more efficient for the user, the tool does
> everything it needs to locally by initially using the linux-yocto
> SRC_URI; once ready, the user can then migrate the kernel components
> to an actual linux-yocto repo by using the 'yocto-kernel migrate'
> command. Once that's done, the user is in a position to submit both
> the BSP metadata and the linux-yocto commits upstream.
>
> Once a BSP is created, the job of 'yocto-bsp' is done (in the initial
> version of the tools - it may have sub-commands added later that could
> possibly manage the non-kernel aspects of a BSP) and the
> 'yocto-kernel' command can be used going forward to maintain the
> kernel portion of a BSP. 'yocto-kernel' allows both bare kernel
> config items and higher-level 'kernel features' to be created, used
> and removed, patches to be added to a BSP's machine branch, and the
> kernel changes to a BSP migrated to a linux-yocto repo. These
> have typically been difficult tasks for many users; the hope is that
> 'yocto-kernel' will make those tasks more manageable for everyone.
>
> The sections below describe each tool in detail.
>
> Following those sections is a detailed description of the templating
> mechanism, useful for anyone who needs to create templates for
> creating new BSPs using the 'yocto-bsp create' tool.
>
>
> The top-level bsp-creation tool, 'yocto-bsp'
> ============================================
>
> All of the functionality of the Yocto bsp-creation tool is available
> via the single 'yocto-bsp' command. Access to the specific functions
> provided by the tool is accomplished by specifying sub-commands to the
> main 'yocto-bsp' command. The design of the command-line interface is
> based on similar interfaces provided by git and perf, which is
> familiar to many users and is easily extensible.
>
> Usage help is always available for the command itself as well as for
> all sub-commands. Executing the top-level command displays a list of
> the available sub-commands, with descriptions:
>
> $ yocto-bsp
>
> usage: yocto-bsp [--version] [--help] COMMAND [ARGS]
>
> The most commonly used 'yocto-bsp' commands are:
> create Create a new Yocto BSP
> list List available values for options and BSP properties
>
> See 'yocto-bsp help COMMAND' for more information on a specific command.
>
>
> 'yocto-bsp create' - Create a new Yocto BSP
> -------------------------------------------
>
> NAME
> yocto-bsp-create - Create a new Yocto BSP
>
> SYNOPSIS
> yocto-bsp create <bsp-name> <karch> <kmachine>
> [--property <NAME:VAL> | -p <NAME:VAL>]
>
> DESCRIPTION
> This command creates a Yocto BSP based on the specified
> parameters. The new BSP will be a new Yocto BSP layer contained
> within the top-level directory specified as bsp-name i.e. not
> 'meta-bsp-name'. The value of the 'karch' parameter determines
> the set of files that will be generated for the BSP, along with
> the specific set of 'properties' that will be used to fill out the
> BSP-specific portions of the BSP.
>
> The BSP-specific properties that define the values that will be
> used to generate a particular BSP can be specified on the
> comman-line using the --property option for each property, which
> is a colon-separated name:value pair.
>
> The set of properties and enumerations of their possible values
> can be displayed using the 'yocto-bsp list' command.
>
> It isn't necessary however for the user to explicitly specify the
> property values using --property options - if none are specified
> in the 'yocto-bsp create' invocation, the user will be
> interactively prompted for each of the required property values,
> which will then be used as values for BSP generation.
>
> The 'kmachine' parameter names the 'machine branch' that the BSP's
> machine branch will be based on; the list of branches meaningful
> for that purpose can also be listed using the 'yocto-bsp list'
> command.
>
> The 'yocto-bsp list' command can also be used to list the possible
> values for the 'karch' parameter.
>
> An typical example that would query the user for property values
> would be:
>
> $ yocto-bsp create meta-foo i386 yocto/standard/common-pc/foo
>
> Because the user hasn't specified any values for the BSP
> properties, yocto-bsp will query the user for each of the
> unspecified property values required to create the BSP i.e. the
> properties listed by 'yocto-bsp list properties'.
>
> Here's an example using explicitly-specified properties:
>
> $ yocto-bsp create meta-foo i386 yocto/standard/common-pc/foo
> --property tunefile:tune-atom --property touchscreen:true
> --property kfeature:cfg/smp [...]
>
>
> 'yocto-bsp list' - List available values for options and BSP properties
> -----------------------------------------------------------------------
>
> NAME
> yocto-bsp-list - List available values for options and BSP properties
>
> SYNOPSIS
> yocto bsp list <karch | kmachine>
> yocto bsp list <karch> properties
> yocto bsp list <karch> property <xxx>
>
> DESCRIPTION
> This command enumerates the complete set of possible values for a
> specified option or property needed by the BSP creation process.
>
> The first form enumerates all the possible values that exist and
> can be specified for the 'karch' or 'kmachine' parameters to the
> 'yocto bsp create' command. Example output for those options:
>
> $ yocto-bsp list karch
> i386
> x86_64
> arm
> powerpc
> mips
>
> $ yocto-bsp list kmachine
> yocto
> yocto/standard
> yocto/standard/common-pc
> yocto/standard/common-pc-64
> yocto/standard/preempt-rt
>
> The second form enumerates all the possible properties that exist
> and must have values specified for them in the 'yocto bsp create'
> command for the given 'karch'. Example output for that command:
>
> $ yocto-bsp list i386 properties
> "machine": boolean,
> "kmachine": choice,
> "tunefile": choicelist,
> "smp": boolean,
> "xserver": choicelist
> ["xserver-choice": choice]
> "kfeatures": choicelist
> ["kfeature": choice]
> "touchscreen": boolean,
> "keyboard": boolean
>
> The third form enumerates all the possible values that exist and
> can be specified for any of the enumerable properties of the given
> 'karch' in the 'yocto bsp create' command. Example output for
> those properties:
>
> $ yocto-bsp list i386 property xserver
> xserver-vesa
> xserver-i915
> xserver-emgd
>
> $ yocto-bsp list i386 property kfeatures
> features/taskstats
> cfg/sound
>
> $ yocto-bsp list i386 property tunefile
> i386
> tune-atom
> tune-core2
>
> $ yocto-bsp list x86_64 property tunefile
> x86_64
> tune-x86_64
> tune-corei7
> tune-ivb
>
> $ yocto-bsp list powerpc property tunefile
> powerpc
> tune-ppc603e
> tune-ppce300c2
> tune-ppce500
> tune-ppce500mc
> tune-ppce500v2
> tune-ppce5500-32b
> tune-ppce5500-64b
>
> $ yocto-bsp list arm property tunefile
> arm
> tune-arm1136jf-s
> tune-arm920t
> tune-arm926ejs
> tune-arm9tdmi
> tune-armv7
>
> $ yocto-bsp list mips property tunefile
> mips
> tune-mips32
>
>
> The top-level kernel usability tool, 'yocto-kernel'
> ===================================================
>
> All of the functionality of the Yocto kernel usability tool is
> available via the single 'yocto-kernel' command. Access to the
> specific functions provided by the tool is accomplished by specifying
> sub-commands to the main 'yocto-kernel' command. The design of the
> command-line interface is based on similar interfaces provided by git
> and perf, which is familiar to many users and is easily extensible.
>
> Usage help is always available for the command itself as well as for
> all sub-commands. Executing the top-level command displays a list of
> the available sub-commands, with descriptions:
>
> $ yocto-kernel
>
> usage: yocto-kernel [--version] [--help] COMMAND [ARGS]
>
> The most commonly used 'yocto-kernel' commands are:
> feature-list List available Yocto KERNEL_FEATUREs
> feature-define Define a new Yocto KERNEL_FEATURE
> feature-add Add Yocto KERNEL_FEATUREs to a BSP
> feature-rm Remove Yocto KERNEL_FEATUREs from a BSP
> config-list List the modifiable set of bare kernel config options for a BSP
> config-add Add or modify bare kernel config options for a BSP
> config-rm Remove bare kernel config options from a BSP
> patch-add Patch the Yocto kernel for a specific BSP
> migrate Move local kernel config/patches to a git repo
>
> See 'yocto-kernel help COMMAND' for more information on a specific command.
>
>
> 'yocto-kernel feature list' - List available Yocto KERNEL_FEATUREs
> ------------------------------------------------------------------
>
> NAME
> yocto-kernel-feature-list - List available Yocto KERNEL_FEATUREs
>
> SYNOPSIS
> yocto-kernel feature list <bsp-name> --used
>
> DESCRIPTION
> This command lists all the kernel features available to a BSP.
> This includes any features temporarily attached to the named BSP
> via the 'feature define' command and thus the reason this command
> takes bsp-name as a parameter.
>
> If the --used param is specified, only the KERNEL_FEATURES used by
> the BSP are listed.
>
>
> 'yocto-kernel feature define' - Define a new Yocto KERNEL_FEATURE
> -----------------------------------------------------------------
>
> NAME
> yocto-kernel-feature-define - Define a new Yocto KERNEL_FEATURE
>
> SYNOPSIS
> yocto-kernel feature define <bsp-name> <feature-name> <feature-desc>
> [<CONFIG_XXX=x> ...] [--dirname <dirname>] [--policy <hw|non-hw>]
>
> DESCRIPTION
> This command defines a new kernel feature, resulting in the
> addition of two new files for feature xxx to the named BSP's
> SRC_URI. If dirname is specified, the feature files will be named
> features_dirname_xxx.cfg and features_dirname_xxx.scc. Otherwise,
> the features will be named cfg_xxx.cfg and cfg_xxx.scc. The idea
> behind the naming is that the names reflect the user's intent to
> have the feature migrated to either the kernel-cache/cfg or
> kernel-cache/features directory if/when the kernel portion of the
> BSP is migrated into the linux-yocto kernel repo.
>
> The policy stating whether the new feature is considered to be a
> hardware or a non-hardware feature also needs to be specified in
> order to create a new feature, and a description of the new
> feature is also required; this will be added as a comment to the
> .scc file.
>
> NOTE: Although features are not BSP-specific, until the user has
> gone through the steps of migrating a feature to the linux-yocto
> repo, it needs to live somewhere accessible to the BSP metadata,
> which logically would be the kernel SRC_URI for the BSP using it.
>
> NOTE: It's up to the user to determine whether or not the config
> options for a feature being added make sense or not - this command
> does no sanity checking or verification of any kind to ensure that
> the config options contained in a kernel feature really make sense
> and will actually be set in in the final config. For example, if
> a config option depends on other config options, it will be turned
> off by kconfig if the other options aren't set correctly.
>
>
> 'yocto-kernel feature add' - Add Yocto KERNEL_FEATUREs to a BSP
> ---------------------------------------------------------------
>
> NAME
> yocto-kernel-feature-add - Add Yocto KERNEL_FEATUREs to a BSP
>
> SYNOPSIS
> yocto-kernel feature add <bsp-name> [<FEATURE> ...]
>
> DESCRIPTION
>
> This command adds one or more kernel features to a BSP. The set
> of features available to be added by this command for a BSP can be
> found via the 'yocto-kernel feature list' command. This command
> essentially adds a new machine-specific KERNEL_FEATURE_append to
> the linux-yocto .bbappend for the BSP.
>
>
> 'yocto-kernel feature rm' - Remove Yocto KERNEL_FEATUREs from a BSP
> -------------------------------------------------------------------
>
> NAME
> yocto-kernel-feature-rm - Remove Yocto KERNEL_FEATUREs from a BSP
>
> SYNOPSIS
> yocto-kernel feature rm <bsp-name> [<FEATURE> ...]
>
> DESCRIPTION
>
> This command removes one or more kernel features from a BSP. The
> set of features available to be removed by this command for a BSP
> can be found via the 'yocto-kernel feature list' command, if the
> --used option to that command is specified. This command
> essentially removes a machine-specific KERNEL_FEATURE from the
> linux-yocto .bbappend for the BSP.
>
>
> 'yocto-kernel config list' - List a BSP's modifiable config options
> -------------------------------------------------------------------
>
> NAME
> yocto-kernel-config-list - List the modifiable set of bare kernel
> config options for a BSP
>
> SYNOPSIS
> yocto-kernel config list <bsp-name>
>
> DESCRIPTION
> This command lists the 'modifiable' config items for a BSP
> i.e. the items which are eligible for modification or removal by
> other yocto-kernel commands.
>
> 'modifiable' config items are the config items contained a BSP's
> foo.cfg base config.
>
>
> 'yocto-kernel config add' - Add or modify a BSP's kernel config options
> -----------------------------------------------------------------------
>
> NAME
> yocto-kernel-config-add - Add or modify bare kernel config options
> for a BSP
>
> SYNOPSIS
> yocto-kernel config add <bsp-name> [<CONFIG_XXX=x> ...]
>
> DESCRIPTION
> This command adds one or more CONFIG_XXX=x items to a BSP's
> foo.cfg base config. If a config item is already present, the new
> value will simply overwrite the old value.
>
> NOTE: It's up to the user to determine whether or not the config
> options being added make sense or not - this command does no
> sanity checking or verification of any kind to ensure that a
> config option really makes sense and will actually be set in in
> the final config. For example, if a config option depends on
> other config options, it will be turned off by kconfig if the
> other options aren't set correctly.
>
>
> 'yocto-kernel config rm' - Remove BSP kernel config options
> -----------------------------------------------------------
>
> NAME
> yocto-kernel-config-rm - Remove bare kernel config options from a
> BSP
>
> SYNOPSIS
> yocto-kernel config rm <bsp-name> [<CONFIG_XXX> ...]
>
> DESCRIPTION
> This command removes (turns off) one or more CONFIG_XXX items to a
> BSP's foo.cfg base config.
>
> The set of config items available to be removed by this command
> for a BSP can be found via the 'yocto-kernel config list'
> command.
>
>
> 'yocto-kernel patch add' - Patch the Yocto kernel for a specific BSP
> --------------------------------------------------------------------
>
> NAME
> yocto-kernel-patch-add - Patch the Yocto kernel for a specific BSP
>
> SYNOPSIS
> yocto-kernel patch add <bsp-name> [<PATCH> ...]
>
> DESCRIPTION
> This command adds one or more patches to a BSP's machine branch.
> The patch will be added to the BSP's linux-yocto kernel SRC_URI
> and will be guaranteed to be applied in the order specified.
>
> NOTE: It's up to the user to determine whether or not the patches
> being added makes sense or not - this command does no sanity
> checking or verification of any kind to ensure that a patch can
> actually be applied to the BSP's kernel branch; it's assumed that
> the user has already done that.
>
>
> 'yocto-kernel migrate' - Move local kernel config/patches to a git repo
> -----------------------------------------------------------------------
>
> NAME
> yocto-kernel-migrate - Move local kernel config/patches to a git repo
>
> SYNOPSIS
> yocto-kernel migrate <bsp-name> <local linux-yocto repo>
>
> DESCRIPTION
> This command converts the kernel configuration changes, patches,
> and features collected in a BSP's linux-yocto SRC_URI into an
> equivalent set of commits in a local linux-yocto repo. On
> success, the local files will named in the SRC_URI will be
> removed, and the linux-yocto .bbappend will be fixed up to point
> to the new repo.
>
> NOTE: The user is responsible for submitting the generated commits
> upstream and subsequently modifying the kernel .bbappend to
> use the upsteam kernel repo.
>
>
> The design of the 'yocto-bsp create' templating mechanism
> ==========================================================
>
> This section describes the details of the templating mechanism used by
> 'yocto-bsp' to generate BSPs.
>
> Generating a new BSP is basically a templating exercise. A given BSP
> is comprised of a combination of a set of common files that all BSPs
> share along with a set of files specific to a given BSP type. Many of
> the files in both sets need to be configured according to the needs of
> the BSP; this also extends to the naming and existence of the files
> themselves.
>
> The templating scheme used in the Yocto tool is partly inspired by
> ideas from web2py, but adapted to a more general input mechanism for
> the specific problems posed by Yocto.
>
> There are no special files, just marked-up BSP metadata and kernel
> configuration, .scc, and patch files.
>
> The Yocto BSP markup is meant to be extremely simple and limited to
> pretty much this initial set of items; it is however designed to be
> extensible enough to cover any future need.
>
> Anything within double braces is considered to be a Yocto BSP markup
> tag:
>
> {{ foo }}
>
> Anything within double braces and starting with {{= is considered to
> implement simple variable replacement, where variables are by default
> simply keys in a dictionary representing the BSP and containing values
> for the whole BSP:
>
> {{=machine}}
>
> The above means that the whole expression will be replaced with the
> value of the 'machine' entry in the BSP dictionary. The syntax is
> actually shorthand for the equivalent Python code for accessing the
> value of a given key, but isn't restricted to that e.g. it could
> reference a local variable defined in a previous tag, for instance.
>
> In every other case, the 'yocto-bsp' tool will pre-process the
> contents looking for certain keywords at the beginning. The rest it
> will treat as Python code. As of now, there's only one keyword that's
> meaningful:
>
> {{ input }}
>
> which signals that anything contained is input-related. The
> 'yocto-bsp' implementation will use this information to prompt the
> user for input. Under certain conditions e.g. when the user has
> already supplied this information via a file for instance, anything
> contained in this is considered a no-op and ignored.
>
> Anything else is considered to be Python code and will be directly
> incorporated into the BSP generation code.
>
> {{ if keyboard == False: }}
>
> Note also that any of the above can also be used in file and directory
> names, which will also be processed in the same way; this allows us to
> avoid extraneous files is consistent with the rest of the scheme.
>
> Internally, the way things are envisioned to work is that the tool
> will first read in the set of files that makes up the BSP for the
> chosen architecture. It will then pre-process all the 'input' tags it
> finds and use that to prepare a series of text-based prompts to
> retrieve the needed information from the user. The end result of
> presenting all that to the user will be a completd BSP dictionary
> containing all the key/value pairs needed to fill in the BSP. Once
> that information is available, it will make a second pass over the
> remaining tags and fill them in with the Python code needed to
> retrieve and replace the remaining tags with their corresponding
> values from the BSP dictionary (or any other source possibly
> introduced by other tags). The end result of this second pass will be
> executed as Python code to produce the final BSP metadata.
>
> For the first version, the 'input' tags will be used to construct a
> series of text-based questions for the user. The design, however,
> should make it relatively trivial to create an application that could
> create real GUI that would do the same thing at some later point. For
> example, such an application could use the input tags to generate a
> web-based form, which would return the results gathered from the user
> to a web service in the form of a json blob that could directly be
> converted into the BSP Python dictionary needed. e.g. the browser
> would return something like this to the web service, which would in
> turn send it to the 'bsp-create' library function to create the final
> BSP.
>
> {
> "machine": "foo",
> "kmachine": "/yocto/standard/base",
> "tunefile": "tune-atom",
> "smp": true,
> "xserver": {
> "xserver-choice": "xserver-i915",
> },
> "kfeatures": [
> "kfeature": "features/taskstats",
> "kfeature": "cfg/sound",
> ]
> },
> touchscreen: false,
> keyboard: true
> }
>
> Below are some concrete examples of specific tags and how they're used.
>
> Here's a simple example of markup that describes information needed
> from the user about touchscreen and keyboard capabilities in the
> formfactor 'machconfig' file:
>
> # Assume a USB mouse and keyboard are connected
> {{ input type:"boolean" name:"touchscreen" msg:"Does your BSP have a touchscreen?" }}
> HAVE_TOUCHSCREEN={{=touchscreen}}
> {{ input type:"boolean" name:"keyboard" msg:"Does your BSP have a keyboard?" }}
> HAVE_KEYBOARD={{=keyboard}}
>
> For instance, the touchscreen 'input' tag above describes everything a
> UI would need in order to retrieve an answer from the user as to
> whether the hardware in question has a touchscreen or not.
>
> The 'type' key comprises a small set of possible input types that
> include things like 'boolean' (yes/no) choices and lists of choices
> (type:"choicelist").
>
> The 'name' key provides a means for later markup to access the value
> of the variable containing the user's answer for that input. For
> example, the name:"touchscreen" simply references the value in the BSP
> dictionary that was filled in with the user's response to the question
> which was formed and presented to the user using the "msg" tag in the
> input tag.
>
> Below the input tag is the actual content that will be rendered into
> the 'machconfig' file, but with the {{=}} portion replaced with the
> 'touchscreen' value supplied by the user i.e.:
>
> HAVE_TOUCHSCREEN={{=touchscreen}}
>
> will be rendered as:
>
> HAVE_TOUCHSCREEN=0
>
> Assuming the user answered 'no'.
>
> Another interesting thing to note about this example is that in Yocto
> BSPs, the machconfig files are contained within a filesystem location
> that includes the machine name in the path e.g.
>
> recipes-bsp/formfactor/formfactor/foo/machconfig
>
> To accomplish this, the directory names themselves contain markup that
> gets processed and replaced by the tool:
>
> recipes-bsp/formfactor/formfactor/{{=machine}}/machconfig
>
> Here, the {{=machine} directory is renamed to /foo to match the
> machine name of the BSP.
>
> Here's a more involved example, markup included in the
> i386/conf/{{=machine}}.conf file:
>
> {{ input type:"boolean" name:"xserver" msg:"Do you need support for X?" }}
> {{ if xserver: }}
> XSERVER ?= "${XSERVER_IA32_BASE} \
> ${XSERVER_IA32_EXT} \
> {{ input type:"choicelist" name:"xserver_choice" msg:"Please select an xserver for this machine" }}
> {{ input type:"choice" val:"xserver-vesa" msg:"VESA xserver support" }}
> ${XSERVER_IA32_VESA} \
> {{ input type:"choice" val:"xserver-emgd" msg:"EMGD xserver support (proprietary)" }}
> ${XSERVER_IA32_EMGD} \
> {{ input type:"choice" val:"xserver-i915" desc:"i915 xserver support" }}
> ${XSERVER_IA32_I915} \
> "
>
> In this case, we start off with a boolean 'input' question asking if
> the BSP needs to support X. The following line is straight Python
> code which essentially skips the following sections (until a blank
> line) if the user's answer was 'no', or if the answer was 'yes',
> creates a further query to the user asking which xserver type to use.
> In the latter case, the input type is a list of choices resulting in
> the values specified by the 'val' tag appearing as the value for the
> 'xserver_choice' dictionary key.
>
> Code further down in the .conf file can also make use of this
> information to include or exclude content based on the value of the
> xserver_choice variable:
>
> {{ if xserver_choice == "xserver-emgd": }}
> PREFERRED_VERSION_xserver-xorg ?= "1.9.3"
> PREFERRED_VERSION_mesa-dri ?= "7.11"
> PREFERRED_VERSION_emgd-driver-bin ?= "1.8"
>
> {{ if xserver_choice == "xserver-vesa" or xserver_choice == "xserver-emgd": }}
> APPEND += "video=vesafb vga=0x318"
>
> Note again that the tags used to determine whether or not to include
> content simply contain Python code used to directly implement the
> needed logic. Not only is there no need for the user to learn a
> special-purpose language, but gives the tool the flexibility to add
> essentially any kind of processing logic needed in the future.
>
> In the case of xserver choices, we also need to use the value of
> xserver_choice to determine which xorg.conf file to add to the BSP.
> Again, consistent with the rest of the markup processing, this is
> seamlessly accomplished by adding markup to the filenames themselves:
>
> recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver_choice == xserver_emgd: }} xorg.conf
> recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver_choice == xserver_vesa: }} xorg.conf
> recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver_choice == xserver_i915: }} xorg.conf
>
> In this case, the filenames actually contain Python code that
> describes the conditions needed to include those particular files.
>
> Here's another example, again making use of general-purpose Python in
> the markup to generate an open-ended list of choices for the user, in
> this case in linux-yocto_3.0.bbappend, to generate a list of available
> KERNEL_FEATURES for the user to choose from:
>
> {{ input type:"checklist" name:"kfeatures" gen:"fetch-kernel-features()" msg:"Please choose the kernel features to use in this BSP on" }}
> {{ for kfeature in kfeatures: }}
> KERNEL_FEATURES_append_{{=machine}} += {{=kfeature}}
>
> In this case an 'input' tag calls an internally-defined yocto-bsp
> function named fetch_kernel_feature() that generates the list of
> available KERNEL_FEATURES. The output of that call is a Python list
> containing the features, which is assigned to the 'kfeatures'
> variable. The next tag contains Python code that simply iterates over
> that list, and uses the value of the current feature, 'kfeature', to
> append the chosen kernel features to the list of features used by the
> BSP.
>
> Finally, because the kernel components are actually part of the BSP
> metadata, the user-specified values are also available in the
> processing of those files. For instance, in the below case, we can
> use the xserver_choice to include or exclude specific kernel
> configuration settings in the
> recipes-kernel/linux/linux-yocto/{{=machine}}.scc file associated with
> the BSP:
>
> {{ if xserver_choice == "xserver-emgd": }}
> include features/drm-emgd/drm-emgd.scc
>
> ---
>
> For reference, here's an example of how the tools are expected to be
> laid out. Everything is contained within a 'bsp-tools' directory
> under '/scripts'. The 'common' portion, containing the files common
> to every BSP are:
>
> bsp-tools/
> bsp-tools/yocto-bsp.py
> bsp-tools/yocto-kernel.py
> bsp-tools/bsp-cache
> bsp-tools/bsp-cache/target
> bsp-tools/bsp-cache/target/arch
> bsp-tools/bsp-cache/target/arch/common
> bsp-tools/bsp-cache/target/arch/common/README.sources
> bsp-tools/bsp-cache/target/arch/common/binary
> bsp-tools/bsp-cache/target/arch/common/binary/.gitignore
> bsp-tools/bsp-cache/target/arch/common/recipes-kernel
> bsp-tools/bsp-cache/target/arch/common/recipes-kernel/linux
> bsp-tools/bsp-cache/target/arch/common/COPYING.MIT
> bsp-tools/bsp-cache/target/arch/common/README
> bsp-tools/bsp-cache/target/arch/common/recipes-graphics
> bsp-tools/bsp-cache/target/arch/common/recipes-graphics/xorg-xserver
> bsp-tools/bsp-cache/target/arch/common/recipes-graphics/xorg-xserver/{{ if xserver: }} xserver-xf86-config_0.1.bbappend
> bsp-tools/bsp-cache/target/arch/common/conf
> bsp-tools/bsp-cache/target/arch/common/conf/layer.conf
> bsp-tools/bsp-cache/target/arch/common/recipes-core
> bsp-tools/bsp-cache/target/arch/common/recipes-core/tasks
> bsp-tools/bsp-cache/target/arch/common/recipes-core/tasks/task-core-tools.bbappend
> bsp-tools/bsp-cache/target/arch/common/recipes-bsp
> bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor
> bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor_0.0.bbappend
> bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor
> bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor/{{=machine}}
> bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor/{{=machine}}/machconfig
>
> The BSP-specific files are contained in directories named for the
> architecture. In this case, only the i386 files are expanded as an
> example.
>
> bsp-tools/bsp-cache/target/arch/arm
> bsp-tools/bsp-cache/target/arch/powerpc
> bsp-tools/bsp-cache/target/arch/i386
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt_3.0.bbappend
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto_3.0.bbappend
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}.cfg
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}.scc
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}-preempt-rt.scc
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}-standard.scc
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}.cfg
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}.scc
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}-preempt-rt.scc
> bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}-standard.scc
> bsp-tools/bsp-cache/target/arch/i386/recipes-graphics
> bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config
> bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}
> bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver-choice == xserver-emgd: }} xorg.conf
> bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver-choice == xserver-vesa: }} xorg.conf
> bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver-choice == xserver-i915: }} xorg.conf
> bsp-tools/bsp-cache/target/arch/i386/conf
> bsp-tools/bsp-cache/target/arch/i386/conf/machine
> bsp-tools/bsp-cache/target/arch/i386/conf/machine/{{=machine}}.conf
> bsp-tools/bsp-cache/target/arch/mips
> bsp-tools/bsp-cache/target/arch/x86_64
>
>
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
--
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: RFC: Yocto BSP and kernel usability tools
2012-01-31 22:52 ` Darren Hart
@ 2012-02-01 0:07 ` Tom Zanussi
0 siblings, 0 replies; 7+ messages in thread
From: Tom Zanussi @ 2012-02-01 0:07 UTC (permalink / raw)
To: Darren Hart; +Cc: yocto@yoctoproject.org
On Tue, 2012-01-31 at 14:52 -0800, Darren Hart wrote:
> Hi Tom,
>
> This is really late in coming, my apologies. This looks to be the
> beginnings of a very powerful set of tools, and the detail you started
> with should help ensure a successful initial version. Nice work.
>
> Some general questions that don't belong in any one particular place inline:
>
> o You mention the use of json files. Is there a web UI planned for this?
> Or did you just decide that the requirements for the UI were the same as
> a web UI so you opted to use an established format rather than create a
> new one?
>
> o As a follow-on to the previous, it is always nice if the input files
> to tools such as this can be easily human readable (and IMO this
> excludes things like XML and serialized python objects). A simple INI
> type config seems like it might be more accessible and python has a
> module for this. I imagine you've considered this, I'm curious as to
> your rationale for the method outlined below.
>
As a design point, I've tried to completely decouple the user interface
from the back-end, so that yes, in the future a web UI could easily
replace the current command-line user interface. I decided to use json
because it's pretty standard for web applications, and Python also has
json parsers available. And it's human-readable, unlike XML or
serialized Python. But it really doesn't need to be any of those, if
something else makes sense.
> o I have some concern about yet another description format of BSPs and
> essentially kernel configuration. We already ask users to think in terms
> of SCC and CFG fragments, rather than defconfigs. This effectively asks
> users to start thinking in terms of json name value pairs to map to
> bitbake syntax to describe scc and cfg fragments so we can assemble a
> .config. Having the tools and structure in place eliminates a lot of the
> guesswork, so in the end I think it is probably worth it ... I just
> worry about so many levels of abstraction - and I do not have a better
> alternative in mind.
>
No, users never see json or any description format (UI tool or webapp
developers do though). The json format (or whatever it ends up being)
is only used as a way of feeding user input values into the back-end in
the absence of a user interface (e.g. sent from a browser, or specified
as a group on the command-line). Regardless of where the input comes
from, in any case it's finally translated it into the .scc and .cfg
fragments we all know and love.
Actually, the point of all this is that the end user doesn't even need
to know about .scc files or .cfg fragments, and deals only at a
high-level with 'features' or config items or patches; this whole scheme
allows UI developers to creat UIs that do that.
> o I see the tools for KMACHINE but nothing for KBRANCH. The distinction
> here is subtle and not described in any of our documentations (BSP
> guide, Kernel Arch and Usage, or Yocto Developer guide). Ideally we use
> standard/default/base for new BSPs and only add the meta-data to the
> linux-yocto kernel. Is this workflow supported? I ask because I'm
> failing to accomplish this with the sys940x and seem to have to add the
> sys940x branch in order for kernel_configme to pick up the sys940x meta
> data.
>
In the current implementation of the cmd-line UI, the user can specify
any branch ending with base e.g.
standard/base
standard/default/base
standard/default/common-pc-64/base
standard/default/common-pc/base
standard/preempt-rt/base
'base' is removed from the branch name and replaced with the machine
name e.g.
standard/default/foo
The 'publish' function will end up creating that branch in the repo and
adding the .scc/.cfg files as a bsp/ in the meta branch. Also, new
features (.cfg and .scc contained in subdirs in the SRC_URI) will be
created under features/ and referenced in the bsp .scc.
I've also run into the same thing you describe - the current tools seem
to require a 1-1 mapping between KBRANCH/KMACHINE and the machine's
metadata, but I don't see a problem with that and it seems more
intuitive to the user (and allows a place to commit bsp-specific
branches immediately without explicitly having to force a new branch
first).
Tom
> --
> Darren
>
> On 12/19/2011 08:55 AM, Tom Zanussi wrote:
> > Hi,
> >
> > This is what I hope to implement in the M2 timeframe.
> >
> > Any and all comments welcome...
> >
> > Thanks,
> >
> > Tom
> >
> > ---
> >
> > Getting Yocto running on a new board is often a prerequisite task to
> > the real task of interest for new users, which is creating new
> > applications designed to run on the board, or creating new
> > board-specific embedded systems. In other words, most users just want
> > to get the hardware up and running and not have to think too much
> > about it before moving on to what they're really interested in, which
> > is the stuff running on top of it.
> >
> > While there really can't be any such thing as a magic-bullet
> > application that would do that automatically i.e. generate and
> > maintain a guaranteed-working Yocto BSP for any given piece of
> > hardware, it should at least be possible to provide a tool that would
> > generate a base BSP usable as a starting point for that, and which
> > would additionally allow the user to tweak various BSP settings,
> > including kernel configuration options and simple kernel patches.
> > Such a tool should be usable with only a minimal understanding of the
> > Yocto build system and metadata, and with no knowledge at all of
> > kernel internals or the Yocto kernel recipe bindings, or of the
> > details of the internals of yocto kernel repositories, or in fact any
> > knowledge of git or any other specific build-related tool.
> >
> > This document provides the design for such a tool. At a high level,
> > the goal of the tool is to provide a user the means of creating a new
> > Yocto BSP from which it should be possible (assuming a perfect
> > run-time outcome, admittedly unlikely on a first pass) to submit a
> > 'pull request' for inclusion into any repo(s) that would accept an
> > 'official' Yocto BSP. For example, the output of the tool should
> > produce metadata and kernel configuration that would be directly
> > mergeable into the meta-intel and the linux-yocto kernel repos. The
> > tool should also allow the BSP maintainer to afterwards at any time
> > make changes to the kernel configuration and/or submit kernel patches
> > for the BSP.
> >
> > Note that although knowledge of the Yocto kernel or kernel internals
> > is not expected, the user is expected to know the basics of how to
> > deal with the linux kernel at the 'user' level, specifically
> > understanding and determining the specific kernel config options
> > needed for the user's BSP.
> >
> > Specifically, the tool will allow for the following high-level tasks,
> > which are discussed in more detail below:
> >
> > - create a new BSP from scratch from a set of user-settable parameters
> > - manage individual kernel config settings
> > - create and use groups of kernel config settings as KERNEL_FEATUREs
> > - apply patches to a BSP's machine branch
> >
> > The following capabilities are specifically not provided by the tool:
> >
> > - help in determining a 'correct' set of kernel config options to set.
> > The tool assumes the user already knows that, either through diffs
> > of .config files before and after using 'bitbake -c menuconfig', or
> > some other means.
> >
> > - modification of the BSP metadata after BSP creation. The tool
> > provides an initial 'write-once' BSP-generation capability, but
> > doesn't allow it to be read in and modified after the initial pass.
> > The user will have to make further modifications manually. The one
> > exception to this is kernel features, which can be added to an
> > already existing kernel bbappend file.
> >
> > - a guarantee or even expectation that the generated BSP will work on
> > the actual hardware it's targeted for - it's highly unlikely the BSP
> > will work 'out-of-the-box' and it's the developer's reponsibility to
> > do the often hard work of figuring out what settings and/or code are
> > actually needed to get the hardware to work as expected. The goal
> > of the tool is to make that job easier, not to actually do that job.
> >
> > Also, though it's not explicitly a requirement of this tool, the
> > design should be sufficiently modular to allow for its participation
> > in generic logical pipelines. For instance, while the tool may
> > present to the user a text-based UI for gathering information, it
> > should also be able to operate without any kind of user interaction
> > and retrieve the data it needs from a file, for instance e.g.:
> >
> > $ yocto-bsp create < x86-input.json
> >
> > Another example that the design should not preclude would be its use
> > at the end of a bsp-import pipeline e.g.:
> >
> > $ cat 3rdparty.bsp | 3rdparty2yocto | yocto-bsp create
> >
> > The tool will initially be implemented as a set of command-line tools
> > which will essentially be thin layers on top of a modular Python API;
> > future tools may be GUI-based but would make use of the functionality
> > exposed through the same API. There might also be an opportunity for
> > future integration into existing tools such as hob; this isn't
> > explicitly catered for in the current design, but it should be
> > possible to do that based either on the functionality provided by the
> > command-line interface or equivalently via the Python module API.
> >
> >
> > A Quick Overview
> > ----------------
> >
> > The initial set of tools will comprise 'yocto-bsp', for creating Yocto
> > BSPs, and 'yocto-kernel', for BSP and BSP-related Yocto kernel
> > maintenance tasks on existing BSPs.
> >
> > The interface and detailed description of each tool are described
> > below, and the underlying templating mechanism is described following
> > that, but it would be good for context to first try to give an overall
> > picture of how the tools were designed to work and be used together.
> >
> > The envisioned usage model is that a user would first start out by
> > creating a new BSP for a given architecture by invoking the 'yocto-bsp
> > create' command. Choosing the architecture essentially chooses a set
> > of files that the BSP will be based on. This set of files is a
> > combination of a set of 'common' files that all BSPs inherit and a set
> > of architecture-specific files that match the specified architecture.
> >
> > Neither set of files can be used unmodified on their own as simple
> > static copies of the originals in order to create a working BSP for a
> > real machine - many of those files need to be tailored for the
> > hardware using information supplied by the user. To accomplish this
> > in an automated way, both the specific portions of the BSP files that
> > need to be filled in with user input, and the means of (optionally)
> > getting that input from the user are included as in-line annotations
> > to the BSP files themselves using a simple templating scheme based on
> > conventions used by web2py. Essentially, in the default case, the
> > templating system is used to generate a series of text-based prompts
> > that ask the user questions about the BSP being created. The data
> > supplied by the user is then used to fill in the missing information
> > in the BSP files. The user can also choose to specify all the
> > information needed directly to the 'yocto-bsp create' command
> > invocation itself; the 'yocto-bsp list' command exists to provide the
> > user the list of 'properties' and enumerations of possible values for
> > those properties, which taken together represent the data items needed
> > to generate the BSP. The user can even more directly supply the
> > information as a json object representing a serialized python
> > dictionary, which is what the templating interface uses natively to
> > represent properties and values needed for BSP generation.
> >
> > The end result is a Yocto-compliant BSP that should be directly
> > buildable by the Yocto build system. The generated BSP is completely
> > self-contained - it includes not only the BSP metadata, but also the
> > necessary kernel components, which are normally for BSPs contained in
> > the linux-yocto repo's meta and machine branches for the BSP. In
> > order to help make the tool more efficient for the user, the tool does
> > everything it needs to locally by initially using the linux-yocto
> > SRC_URI; once ready, the user can then migrate the kernel components
> > to an actual linux-yocto repo by using the 'yocto-kernel migrate'
> > command. Once that's done, the user is in a position to submit both
> > the BSP metadata and the linux-yocto commits upstream.
> >
> > Once a BSP is created, the job of 'yocto-bsp' is done (in the initial
> > version of the tools - it may have sub-commands added later that could
> > possibly manage the non-kernel aspects of a BSP) and the
> > 'yocto-kernel' command can be used going forward to maintain the
> > kernel portion of a BSP. 'yocto-kernel' allows both bare kernel
> > config items and higher-level 'kernel features' to be created, used
> > and removed, patches to be added to a BSP's machine branch, and the
> > kernel changes to a BSP migrated to a linux-yocto repo. These
> > have typically been difficult tasks for many users; the hope is that
> > 'yocto-kernel' will make those tasks more manageable for everyone.
> >
> > The sections below describe each tool in detail.
> >
> > Following those sections is a detailed description of the templating
> > mechanism, useful for anyone who needs to create templates for
> > creating new BSPs using the 'yocto-bsp create' tool.
> >
> >
> > The top-level bsp-creation tool, 'yocto-bsp'
> > ============================================
> >
> > All of the functionality of the Yocto bsp-creation tool is available
> > via the single 'yocto-bsp' command. Access to the specific functions
> > provided by the tool is accomplished by specifying sub-commands to the
> > main 'yocto-bsp' command. The design of the command-line interface is
> > based on similar interfaces provided by git and perf, which is
> > familiar to many users and is easily extensible.
> >
> > Usage help is always available for the command itself as well as for
> > all sub-commands. Executing the top-level command displays a list of
> > the available sub-commands, with descriptions:
> >
> > $ yocto-bsp
> >
> > usage: yocto-bsp [--version] [--help] COMMAND [ARGS]
> >
> > The most commonly used 'yocto-bsp' commands are:
> > create Create a new Yocto BSP
> > list List available values for options and BSP properties
> >
> > See 'yocto-bsp help COMMAND' for more information on a specific command.
> >
> >
> > 'yocto-bsp create' - Create a new Yocto BSP
> > -------------------------------------------
> >
> > NAME
> > yocto-bsp-create - Create a new Yocto BSP
> >
> > SYNOPSIS
> > yocto-bsp create <bsp-name> <karch> <kmachine>
> > [--property <NAME:VAL> | -p <NAME:VAL>]
> >
> > DESCRIPTION
> > This command creates a Yocto BSP based on the specified
> > parameters. The new BSP will be a new Yocto BSP layer contained
> > within the top-level directory specified as bsp-name i.e. not
> > 'meta-bsp-name'. The value of the 'karch' parameter determines
> > the set of files that will be generated for the BSP, along with
> > the specific set of 'properties' that will be used to fill out the
> > BSP-specific portions of the BSP.
> >
> > The BSP-specific properties that define the values that will be
> > used to generate a particular BSP can be specified on the
> > comman-line using the --property option for each property, which
> > is a colon-separated name:value pair.
> >
> > The set of properties and enumerations of their possible values
> > can be displayed using the 'yocto-bsp list' command.
> >
> > It isn't necessary however for the user to explicitly specify the
> > property values using --property options - if none are specified
> > in the 'yocto-bsp create' invocation, the user will be
> > interactively prompted for each of the required property values,
> > which will then be used as values for BSP generation.
> >
> > The 'kmachine' parameter names the 'machine branch' that the BSP's
> > machine branch will be based on; the list of branches meaningful
> > for that purpose can also be listed using the 'yocto-bsp list'
> > command.
> >
> > The 'yocto-bsp list' command can also be used to list the possible
> > values for the 'karch' parameter.
> >
> > An typical example that would query the user for property values
> > would be:
> >
> > $ yocto-bsp create meta-foo i386 yocto/standard/common-pc/foo
> >
> > Because the user hasn't specified any values for the BSP
> > properties, yocto-bsp will query the user for each of the
> > unspecified property values required to create the BSP i.e. the
> > properties listed by 'yocto-bsp list properties'.
> >
> > Here's an example using explicitly-specified properties:
> >
> > $ yocto-bsp create meta-foo i386 yocto/standard/common-pc/foo
> > --property tunefile:tune-atom --property touchscreen:true
> > --property kfeature:cfg/smp [...]
> >
> >
> > 'yocto-bsp list' - List available values for options and BSP properties
> > -----------------------------------------------------------------------
> >
> > NAME
> > yocto-bsp-list - List available values for options and BSP properties
> >
> > SYNOPSIS
> > yocto bsp list <karch | kmachine>
> > yocto bsp list <karch> properties
> > yocto bsp list <karch> property <xxx>
> >
> > DESCRIPTION
> > This command enumerates the complete set of possible values for a
> > specified option or property needed by the BSP creation process.
> >
> > The first form enumerates all the possible values that exist and
> > can be specified for the 'karch' or 'kmachine' parameters to the
> > 'yocto bsp create' command. Example output for those options:
> >
> > $ yocto-bsp list karch
> > i386
> > x86_64
> > arm
> > powerpc
> > mips
> >
> > $ yocto-bsp list kmachine
> > yocto
> > yocto/standard
> > yocto/standard/common-pc
> > yocto/standard/common-pc-64
> > yocto/standard/preempt-rt
> >
> > The second form enumerates all the possible properties that exist
> > and must have values specified for them in the 'yocto bsp create'
> > command for the given 'karch'. Example output for that command:
> >
> > $ yocto-bsp list i386 properties
> > "machine": boolean,
> > "kmachine": choice,
> > "tunefile": choicelist,
> > "smp": boolean,
> > "xserver": choicelist
> > ["xserver-choice": choice]
> > "kfeatures": choicelist
> > ["kfeature": choice]
> > "touchscreen": boolean,
> > "keyboard": boolean
> >
> > The third form enumerates all the possible values that exist and
> > can be specified for any of the enumerable properties of the given
> > 'karch' in the 'yocto bsp create' command. Example output for
> > those properties:
> >
> > $ yocto-bsp list i386 property xserver
> > xserver-vesa
> > xserver-i915
> > xserver-emgd
> >
> > $ yocto-bsp list i386 property kfeatures
> > features/taskstats
> > cfg/sound
> >
> > $ yocto-bsp list i386 property tunefile
> > i386
> > tune-atom
> > tune-core2
> >
> > $ yocto-bsp list x86_64 property tunefile
> > x86_64
> > tune-x86_64
> > tune-corei7
> > tune-ivb
> >
> > $ yocto-bsp list powerpc property tunefile
> > powerpc
> > tune-ppc603e
> > tune-ppce300c2
> > tune-ppce500
> > tune-ppce500mc
> > tune-ppce500v2
> > tune-ppce5500-32b
> > tune-ppce5500-64b
> >
> > $ yocto-bsp list arm property tunefile
> > arm
> > tune-arm1136jf-s
> > tune-arm920t
> > tune-arm926ejs
> > tune-arm9tdmi
> > tune-armv7
> >
> > $ yocto-bsp list mips property tunefile
> > mips
> > tune-mips32
> >
> >
> > The top-level kernel usability tool, 'yocto-kernel'
> > ===================================================
> >
> > All of the functionality of the Yocto kernel usability tool is
> > available via the single 'yocto-kernel' command. Access to the
> > specific functions provided by the tool is accomplished by specifying
> > sub-commands to the main 'yocto-kernel' command. The design of the
> > command-line interface is based on similar interfaces provided by git
> > and perf, which is familiar to many users and is easily extensible.
> >
> > Usage help is always available for the command itself as well as for
> > all sub-commands. Executing the top-level command displays a list of
> > the available sub-commands, with descriptions:
> >
> > $ yocto-kernel
> >
> > usage: yocto-kernel [--version] [--help] COMMAND [ARGS]
> >
> > The most commonly used 'yocto-kernel' commands are:
> > feature-list List available Yocto KERNEL_FEATUREs
> > feature-define Define a new Yocto KERNEL_FEATURE
> > feature-add Add Yocto KERNEL_FEATUREs to a BSP
> > feature-rm Remove Yocto KERNEL_FEATUREs from a BSP
> > config-list List the modifiable set of bare kernel config options for a BSP
> > config-add Add or modify bare kernel config options for a BSP
> > config-rm Remove bare kernel config options from a BSP
> > patch-add Patch the Yocto kernel for a specific BSP
> > migrate Move local kernel config/patches to a git repo
> >
> > See 'yocto-kernel help COMMAND' for more information on a specific command.
> >
> >
> > 'yocto-kernel feature list' - List available Yocto KERNEL_FEATUREs
> > ------------------------------------------------------------------
> >
> > NAME
> > yocto-kernel-feature-list - List available Yocto KERNEL_FEATUREs
> >
> > SYNOPSIS
> > yocto-kernel feature list <bsp-name> --used
> >
> > DESCRIPTION
> > This command lists all the kernel features available to a BSP.
> > This includes any features temporarily attached to the named BSP
> > via the 'feature define' command and thus the reason this command
> > takes bsp-name as a parameter.
> >
> > If the --used param is specified, only the KERNEL_FEATURES used by
> > the BSP are listed.
> >
> >
> > 'yocto-kernel feature define' - Define a new Yocto KERNEL_FEATURE
> > -----------------------------------------------------------------
> >
> > NAME
> > yocto-kernel-feature-define - Define a new Yocto KERNEL_FEATURE
> >
> > SYNOPSIS
> > yocto-kernel feature define <bsp-name> <feature-name> <feature-desc>
> > [<CONFIG_XXX=x> ...] [--dirname <dirname>] [--policy <hw|non-hw>]
> >
> > DESCRIPTION
> > This command defines a new kernel feature, resulting in the
> > addition of two new files for feature xxx to the named BSP's
> > SRC_URI. If dirname is specified, the feature files will be named
> > features_dirname_xxx.cfg and features_dirname_xxx.scc. Otherwise,
> > the features will be named cfg_xxx.cfg and cfg_xxx.scc. The idea
> > behind the naming is that the names reflect the user's intent to
> > have the feature migrated to either the kernel-cache/cfg or
> > kernel-cache/features directory if/when the kernel portion of the
> > BSP is migrated into the linux-yocto kernel repo.
> >
> > The policy stating whether the new feature is considered to be a
> > hardware or a non-hardware feature also needs to be specified in
> > order to create a new feature, and a description of the new
> > feature is also required; this will be added as a comment to the
> > .scc file.
> >
> > NOTE: Although features are not BSP-specific, until the user has
> > gone through the steps of migrating a feature to the linux-yocto
> > repo, it needs to live somewhere accessible to the BSP metadata,
> > which logically would be the kernel SRC_URI for the BSP using it.
> >
> > NOTE: It's up to the user to determine whether or not the config
> > options for a feature being added make sense or not - this command
> > does no sanity checking or verification of any kind to ensure that
> > the config options contained in a kernel feature really make sense
> > and will actually be set in in the final config. For example, if
> > a config option depends on other config options, it will be turned
> > off by kconfig if the other options aren't set correctly.
> >
> >
> > 'yocto-kernel feature add' - Add Yocto KERNEL_FEATUREs to a BSP
> > ---------------------------------------------------------------
> >
> > NAME
> > yocto-kernel-feature-add - Add Yocto KERNEL_FEATUREs to a BSP
> >
> > SYNOPSIS
> > yocto-kernel feature add <bsp-name> [<FEATURE> ...]
> >
> > DESCRIPTION
> >
> > This command adds one or more kernel features to a BSP. The set
> > of features available to be added by this command for a BSP can be
> > found via the 'yocto-kernel feature list' command. This command
> > essentially adds a new machine-specific KERNEL_FEATURE_append to
> > the linux-yocto .bbappend for the BSP.
> >
> >
> > 'yocto-kernel feature rm' - Remove Yocto KERNEL_FEATUREs from a BSP
> > -------------------------------------------------------------------
> >
> > NAME
> > yocto-kernel-feature-rm - Remove Yocto KERNEL_FEATUREs from a BSP
> >
> > SYNOPSIS
> > yocto-kernel feature rm <bsp-name> [<FEATURE> ...]
> >
> > DESCRIPTION
> >
> > This command removes one or more kernel features from a BSP. The
> > set of features available to be removed by this command for a BSP
> > can be found via the 'yocto-kernel feature list' command, if the
> > --used option to that command is specified. This command
> > essentially removes a machine-specific KERNEL_FEATURE from the
> > linux-yocto .bbappend for the BSP.
> >
> >
> > 'yocto-kernel config list' - List a BSP's modifiable config options
> > -------------------------------------------------------------------
> >
> > NAME
> > yocto-kernel-config-list - List the modifiable set of bare kernel
> > config options for a BSP
> >
> > SYNOPSIS
> > yocto-kernel config list <bsp-name>
> >
> > DESCRIPTION
> > This command lists the 'modifiable' config items for a BSP
> > i.e. the items which are eligible for modification or removal by
> > other yocto-kernel commands.
> >
> > 'modifiable' config items are the config items contained a BSP's
> > foo.cfg base config.
> >
> >
> > 'yocto-kernel config add' - Add or modify a BSP's kernel config options
> > -----------------------------------------------------------------------
> >
> > NAME
> > yocto-kernel-config-add - Add or modify bare kernel config options
> > for a BSP
> >
> > SYNOPSIS
> > yocto-kernel config add <bsp-name> [<CONFIG_XXX=x> ...]
> >
> > DESCRIPTION
> > This command adds one or more CONFIG_XXX=x items to a BSP's
> > foo.cfg base config. If a config item is already present, the new
> > value will simply overwrite the old value.
> >
> > NOTE: It's up to the user to determine whether or not the config
> > options being added make sense or not - this command does no
> > sanity checking or verification of any kind to ensure that a
> > config option really makes sense and will actually be set in in
> > the final config. For example, if a config option depends on
> > other config options, it will be turned off by kconfig if the
> > other options aren't set correctly.
> >
> >
> > 'yocto-kernel config rm' - Remove BSP kernel config options
> > -----------------------------------------------------------
> >
> > NAME
> > yocto-kernel-config-rm - Remove bare kernel config options from a
> > BSP
> >
> > SYNOPSIS
> > yocto-kernel config rm <bsp-name> [<CONFIG_XXX> ...]
> >
> > DESCRIPTION
> > This command removes (turns off) one or more CONFIG_XXX items to a
> > BSP's foo.cfg base config.
> >
> > The set of config items available to be removed by this command
> > for a BSP can be found via the 'yocto-kernel config list'
> > command.
> >
> >
> > 'yocto-kernel patch add' - Patch the Yocto kernel for a specific BSP
> > --------------------------------------------------------------------
> >
> > NAME
> > yocto-kernel-patch-add - Patch the Yocto kernel for a specific BSP
> >
> > SYNOPSIS
> > yocto-kernel patch add <bsp-name> [<PATCH> ...]
> >
> > DESCRIPTION
> > This command adds one or more patches to a BSP's machine branch.
> > The patch will be added to the BSP's linux-yocto kernel SRC_URI
> > and will be guaranteed to be applied in the order specified.
> >
> > NOTE: It's up to the user to determine whether or not the patches
> > being added makes sense or not - this command does no sanity
> > checking or verification of any kind to ensure that a patch can
> > actually be applied to the BSP's kernel branch; it's assumed that
> > the user has already done that.
> >
> >
> > 'yocto-kernel migrate' - Move local kernel config/patches to a git repo
> > -----------------------------------------------------------------------
> >
> > NAME
> > yocto-kernel-migrate - Move local kernel config/patches to a git repo
> >
> > SYNOPSIS
> > yocto-kernel migrate <bsp-name> <local linux-yocto repo>
> >
> > DESCRIPTION
> > This command converts the kernel configuration changes, patches,
> > and features collected in a BSP's linux-yocto SRC_URI into an
> > equivalent set of commits in a local linux-yocto repo. On
> > success, the local files will named in the SRC_URI will be
> > removed, and the linux-yocto .bbappend will be fixed up to point
> > to the new repo.
> >
> > NOTE: The user is responsible for submitting the generated commits
> > upstream and subsequently modifying the kernel .bbappend to
> > use the upsteam kernel repo.
> >
> >
> > The design of the 'yocto-bsp create' templating mechanism
> > ==========================================================
> >
> > This section describes the details of the templating mechanism used by
> > 'yocto-bsp' to generate BSPs.
> >
> > Generating a new BSP is basically a templating exercise. A given BSP
> > is comprised of a combination of a set of common files that all BSPs
> > share along with a set of files specific to a given BSP type. Many of
> > the files in both sets need to be configured according to the needs of
> > the BSP; this also extends to the naming and existence of the files
> > themselves.
> >
> > The templating scheme used in the Yocto tool is partly inspired by
> > ideas from web2py, but adapted to a more general input mechanism for
> > the specific problems posed by Yocto.
> >
> > There are no special files, just marked-up BSP metadata and kernel
> > configuration, .scc, and patch files.
> >
> > The Yocto BSP markup is meant to be extremely simple and limited to
> > pretty much this initial set of items; it is however designed to be
> > extensible enough to cover any future need.
> >
> > Anything within double braces is considered to be a Yocto BSP markup
> > tag:
> >
> > {{ foo }}
> >
> > Anything within double braces and starting with {{= is considered to
> > implement simple variable replacement, where variables are by default
> > simply keys in a dictionary representing the BSP and containing values
> > for the whole BSP:
> >
> > {{=machine}}
> >
> > The above means that the whole expression will be replaced with the
> > value of the 'machine' entry in the BSP dictionary. The syntax is
> > actually shorthand for the equivalent Python code for accessing the
> > value of a given key, but isn't restricted to that e.g. it could
> > reference a local variable defined in a previous tag, for instance.
> >
> > In every other case, the 'yocto-bsp' tool will pre-process the
> > contents looking for certain keywords at the beginning. The rest it
> > will treat as Python code. As of now, there's only one keyword that's
> > meaningful:
> >
> > {{ input }}
> >
> > which signals that anything contained is input-related. The
> > 'yocto-bsp' implementation will use this information to prompt the
> > user for input. Under certain conditions e.g. when the user has
> > already supplied this information via a file for instance, anything
> > contained in this is considered a no-op and ignored.
> >
> > Anything else is considered to be Python code and will be directly
> > incorporated into the BSP generation code.
> >
> > {{ if keyboard == False: }}
> >
> > Note also that any of the above can also be used in file and directory
> > names, which will also be processed in the same way; this allows us to
> > avoid extraneous files is consistent with the rest of the scheme.
> >
> > Internally, the way things are envisioned to work is that the tool
> > will first read in the set of files that makes up the BSP for the
> > chosen architecture. It will then pre-process all the 'input' tags it
> > finds and use that to prepare a series of text-based prompts to
> > retrieve the needed information from the user. The end result of
> > presenting all that to the user will be a completd BSP dictionary
> > containing all the key/value pairs needed to fill in the BSP. Once
> > that information is available, it will make a second pass over the
> > remaining tags and fill them in with the Python code needed to
> > retrieve and replace the remaining tags with their corresponding
> > values from the BSP dictionary (or any other source possibly
> > introduced by other tags). The end result of this second pass will be
> > executed as Python code to produce the final BSP metadata.
> >
> > For the first version, the 'input' tags will be used to construct a
> > series of text-based questions for the user. The design, however,
> > should make it relatively trivial to create an application that could
> > create real GUI that would do the same thing at some later point. For
> > example, such an application could use the input tags to generate a
> > web-based form, which would return the results gathered from the user
> > to a web service in the form of a json blob that could directly be
> > converted into the BSP Python dictionary needed. e.g. the browser
> > would return something like this to the web service, which would in
> > turn send it to the 'bsp-create' library function to create the final
> > BSP.
> >
> > {
> > "machine": "foo",
> > "kmachine": "/yocto/standard/base",
> > "tunefile": "tune-atom",
> > "smp": true,
> > "xserver": {
> > "xserver-choice": "xserver-i915",
> > },
> > "kfeatures": [
> > "kfeature": "features/taskstats",
> > "kfeature": "cfg/sound",
> > ]
> > },
> > touchscreen: false,
> > keyboard: true
> > }
> >
> > Below are some concrete examples of specific tags and how they're used.
> >
> > Here's a simple example of markup that describes information needed
> > from the user about touchscreen and keyboard capabilities in the
> > formfactor 'machconfig' file:
> >
> > # Assume a USB mouse and keyboard are connected
> > {{ input type:"boolean" name:"touchscreen" msg:"Does your BSP have a touchscreen?" }}
> > HAVE_TOUCHSCREEN={{=touchscreen}}
> > {{ input type:"boolean" name:"keyboard" msg:"Does your BSP have a keyboard?" }}
> > HAVE_KEYBOARD={{=keyboard}}
> >
> > For instance, the touchscreen 'input' tag above describes everything a
> > UI would need in order to retrieve an answer from the user as to
> > whether the hardware in question has a touchscreen or not.
> >
> > The 'type' key comprises a small set of possible input types that
> > include things like 'boolean' (yes/no) choices and lists of choices
> > (type:"choicelist").
> >
> > The 'name' key provides a means for later markup to access the value
> > of the variable containing the user's answer for that input. For
> > example, the name:"touchscreen" simply references the value in the BSP
> > dictionary that was filled in with the user's response to the question
> > which was formed and presented to the user using the "msg" tag in the
> > input tag.
> >
> > Below the input tag is the actual content that will be rendered into
> > the 'machconfig' file, but with the {{=}} portion replaced with the
> > 'touchscreen' value supplied by the user i.e.:
> >
> > HAVE_TOUCHSCREEN={{=touchscreen}}
> >
> > will be rendered as:
> >
> > HAVE_TOUCHSCREEN=0
> >
> > Assuming the user answered 'no'.
> >
> > Another interesting thing to note about this example is that in Yocto
> > BSPs, the machconfig files are contained within a filesystem location
> > that includes the machine name in the path e.g.
> >
> > recipes-bsp/formfactor/formfactor/foo/machconfig
> >
> > To accomplish this, the directory names themselves contain markup that
> > gets processed and replaced by the tool:
> >
> > recipes-bsp/formfactor/formfactor/{{=machine}}/machconfig
> >
> > Here, the {{=machine} directory is renamed to /foo to match the
> > machine name of the BSP.
> >
> > Here's a more involved example, markup included in the
> > i386/conf/{{=machine}}.conf file:
> >
> > {{ input type:"boolean" name:"xserver" msg:"Do you need support for X?" }}
> > {{ if xserver: }}
> > XSERVER ?= "${XSERVER_IA32_BASE} \
> > ${XSERVER_IA32_EXT} \
> > {{ input type:"choicelist" name:"xserver_choice" msg:"Please select an xserver for this machine" }}
> > {{ input type:"choice" val:"xserver-vesa" msg:"VESA xserver support" }}
> > ${XSERVER_IA32_VESA} \
> > {{ input type:"choice" val:"xserver-emgd" msg:"EMGD xserver support (proprietary)" }}
> > ${XSERVER_IA32_EMGD} \
> > {{ input type:"choice" val:"xserver-i915" desc:"i915 xserver support" }}
> > ${XSERVER_IA32_I915} \
> > "
> >
> > In this case, we start off with a boolean 'input' question asking if
> > the BSP needs to support X. The following line is straight Python
> > code which essentially skips the following sections (until a blank
> > line) if the user's answer was 'no', or if the answer was 'yes',
> > creates a further query to the user asking which xserver type to use.
> > In the latter case, the input type is a list of choices resulting in
> > the values specified by the 'val' tag appearing as the value for the
> > 'xserver_choice' dictionary key.
> >
> > Code further down in the .conf file can also make use of this
> > information to include or exclude content based on the value of the
> > xserver_choice variable:
> >
> > {{ if xserver_choice == "xserver-emgd": }}
> > PREFERRED_VERSION_xserver-xorg ?= "1.9.3"
> > PREFERRED_VERSION_mesa-dri ?= "7.11"
> > PREFERRED_VERSION_emgd-driver-bin ?= "1.8"
> >
> > {{ if xserver_choice == "xserver-vesa" or xserver_choice == "xserver-emgd": }}
> > APPEND += "video=vesafb vga=0x318"
> >
> > Note again that the tags used to determine whether or not to include
> > content simply contain Python code used to directly implement the
> > needed logic. Not only is there no need for the user to learn a
> > special-purpose language, but gives the tool the flexibility to add
> > essentially any kind of processing logic needed in the future.
> >
> > In the case of xserver choices, we also need to use the value of
> > xserver_choice to determine which xorg.conf file to add to the BSP.
> > Again, consistent with the rest of the markup processing, this is
> > seamlessly accomplished by adding markup to the filenames themselves:
> >
> > recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver_choice == xserver_emgd: }} xorg.conf
> > recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver_choice == xserver_vesa: }} xorg.conf
> > recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver_choice == xserver_i915: }} xorg.conf
> >
> > In this case, the filenames actually contain Python code that
> > describes the conditions needed to include those particular files.
> >
> > Here's another example, again making use of general-purpose Python in
> > the markup to generate an open-ended list of choices for the user, in
> > this case in linux-yocto_3.0.bbappend, to generate a list of available
> > KERNEL_FEATURES for the user to choose from:
> >
> > {{ input type:"checklist" name:"kfeatures" gen:"fetch-kernel-features()" msg:"Please choose the kernel features to use in this BSP on" }}
> > {{ for kfeature in kfeatures: }}
> > KERNEL_FEATURES_append_{{=machine}} += {{=kfeature}}
> >
> > In this case an 'input' tag calls an internally-defined yocto-bsp
> > function named fetch_kernel_feature() that generates the list of
> > available KERNEL_FEATURES. The output of that call is a Python list
> > containing the features, which is assigned to the 'kfeatures'
> > variable. The next tag contains Python code that simply iterates over
> > that list, and uses the value of the current feature, 'kfeature', to
> > append the chosen kernel features to the list of features used by the
> > BSP.
> >
> > Finally, because the kernel components are actually part of the BSP
> > metadata, the user-specified values are also available in the
> > processing of those files. For instance, in the below case, we can
> > use the xserver_choice to include or exclude specific kernel
> > configuration settings in the
> > recipes-kernel/linux/linux-yocto/{{=machine}}.scc file associated with
> > the BSP:
> >
> > {{ if xserver_choice == "xserver-emgd": }}
> > include features/drm-emgd/drm-emgd.scc
> >
> > ---
> >
> > For reference, here's an example of how the tools are expected to be
> > laid out. Everything is contained within a 'bsp-tools' directory
> > under '/scripts'. The 'common' portion, containing the files common
> > to every BSP are:
> >
> > bsp-tools/
> > bsp-tools/yocto-bsp.py
> > bsp-tools/yocto-kernel.py
> > bsp-tools/bsp-cache
> > bsp-tools/bsp-cache/target
> > bsp-tools/bsp-cache/target/arch
> > bsp-tools/bsp-cache/target/arch/common
> > bsp-tools/bsp-cache/target/arch/common/README.sources
> > bsp-tools/bsp-cache/target/arch/common/binary
> > bsp-tools/bsp-cache/target/arch/common/binary/.gitignore
> > bsp-tools/bsp-cache/target/arch/common/recipes-kernel
> > bsp-tools/bsp-cache/target/arch/common/recipes-kernel/linux
> > bsp-tools/bsp-cache/target/arch/common/COPYING.MIT
> > bsp-tools/bsp-cache/target/arch/common/README
> > bsp-tools/bsp-cache/target/arch/common/recipes-graphics
> > bsp-tools/bsp-cache/target/arch/common/recipes-graphics/xorg-xserver
> > bsp-tools/bsp-cache/target/arch/common/recipes-graphics/xorg-xserver/{{ if xserver: }} xserver-xf86-config_0.1.bbappend
> > bsp-tools/bsp-cache/target/arch/common/conf
> > bsp-tools/bsp-cache/target/arch/common/conf/layer.conf
> > bsp-tools/bsp-cache/target/arch/common/recipes-core
> > bsp-tools/bsp-cache/target/arch/common/recipes-core/tasks
> > bsp-tools/bsp-cache/target/arch/common/recipes-core/tasks/task-core-tools.bbappend
> > bsp-tools/bsp-cache/target/arch/common/recipes-bsp
> > bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor
> > bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor_0.0.bbappend
> > bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor
> > bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor/{{=machine}}
> > bsp-tools/bsp-cache/target/arch/common/recipes-bsp/formfactor/formfactor/{{=machine}}/machconfig
> >
> > The BSP-specific files are contained in directories named for the
> > architecture. In this case, only the i386 files are expanded as an
> > example.
> >
> > bsp-tools/bsp-cache/target/arch/arm
> > bsp-tools/bsp-cache/target/arch/powerpc
> > bsp-tools/bsp-cache/target/arch/i386
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt_3.0.bbappend
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto_3.0.bbappend
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}.cfg
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}.scc
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}-preempt-rt.scc
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto/{{=machine}}-standard.scc
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}.cfg
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}.scc
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}-preempt-rt.scc
> > bsp-tools/bsp-cache/target/arch/i386/recipes-kernel/linux/linux-yocto-rt/{{=machine}}-standard.scc
> > bsp-tools/bsp-cache/target/arch/i386/recipes-graphics
> > bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config
> > bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}
> > bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver-choice == xserver-emgd: }} xorg.conf
> > bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver-choice == xserver-vesa: }} xorg.conf
> > bsp-tools/bsp-cache/target/arch/i386/recipes-graphics/xserver-xf86-config/{{=machine}}/{{ if xserver-choice == xserver-i915: }} xorg.conf
> > bsp-tools/bsp-cache/target/arch/i386/conf
> > bsp-tools/bsp-cache/target/arch/i386/conf/machine
> > bsp-tools/bsp-cache/target/arch/i386/conf/machine/{{=machine}}.conf
> > bsp-tools/bsp-cache/target/arch/mips
> > bsp-tools/bsp-cache/target/arch/x86_64
> >
> >
> > _______________________________________________
> > yocto mailing list
> > yocto@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/yocto
>
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2012-02-01 0:08 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-12-19 16:55 RFC: Yocto BSP and kernel usability tools Tom Zanussi
2011-12-22 20:16 ` Bruce Ashfield
2011-12-22 21:31 ` Tom Zanussi
2011-12-28 0:36 ` McClintock Matthew-B29882
2011-12-29 17:32 ` Tom Zanussi
2012-01-31 22:52 ` Darren Hart
2012-02-01 0:07 ` Tom Zanussi
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.