* How bitbake works(Draft)
@ 2011-12-05 3:27 Robert Yang
2011-12-07 13:01 ` Richard Purdie
0 siblings, 1 reply; 3+ messages in thread
From: Robert Yang @ 2011-12-05 3:27 UTC (permalink / raw)
To: Purdie, Richard; +Cc: bitbake-devel
Hi Richard,
Here is a doc about how bitbake works, it may help the bitbake
newbie to understand how bitbake works, I'm very glad to add it
to the public bitbake repository as documentation if it is useful.
please feel free to give you comments.
How bitbake works
Robert Yang
The document tries to explain a common bitbake build, it is based on
yocto 1.2 and the bitbake version is:
$ bitbake --version
BitBake Build Tool Core version 1.15.0, bitbake version 1.15.0
The entrance of the bitbake is the main() function in bin/bitbake, the
main() function does the following things:
1) Parse command line options
2) Initialize the configuration from the command line options
3) Get the ui and server from the configuration
4) Initialize logger
5) Initialize server
6) Register the idle functions
7) Initialize cooker
8) Parse command line action
9) Add the cooker to the server
10) Start the server and waiting for tasks to run
11) Start the ui
12) The ui runs command on the server
Then the build will start, and stop when the build is done.
1, Parse commandline options
-> options (will be used by BBConfiguration)
/
/
parser.parse_args (return two values)
\
\ -> args (will be saved to configuration.pkgs_to_build)
bitbake uses the python moudule optparse to parse the options:
options, args = parser.parse_args(sys.argv)
For example:
$bitbake gzip -cpatch
After the parse, the "options" is a class' instance, and:
options.cmd = "patch"
There are also many other members in options which would have default
values.
and:
args = ['/buildarea/lyang1/poky/bitbake/bin/bitbake', 'gzip']
The args saves the arguments which has not been matched by the
option.
2, Initialize the configuration from the commandline options
BBConfiguration
options -----------------------> configuration
configuration = BBConfiguration(options)
BBConfiguration will convert the dictionary of "options" to the
member of BBConfiguration, and add a member called pkgs_to_build.
configuration.pkgs_to_build.extend(args[1:])
Add the arguments to configuration.pkgs_to_build except args[0] (which
is the bitbake itself)
3, Get the ui and server from the configuration
-> ui_main (will be used by server.launchUI)
/
/
configuration
\
\-> server (will be used to run tasks)
ui_main = get_ui(configuration)
The get_ui is defined in bin/bitbake, it would use the knotty
(lib/bb/ui/knotty.py) by default, and use __import__ to import the
module at runtime. (The "import" can't import the module
dynamically). The ui_main is the main() function in
lib/bb/ui/knotty.py
The similar to the server, the server is lib/bb/server/process.py by
default.
4) Initialize logger
* set the log level.
---------> verbose ------->
/ \
/ \
configuration ------------> debug --------> bb.msg.init_msgconfig
\ /
\ /
--> debug_domains ------->
bb.msg.init_msgconfig(configuration.verbose, configuration.debug,
configuration.debug_domains)
* set the log handler
handler = bb.event.LogHandler()
logger.addHandler(handler)
There are a lot of log handlers, for example: FileHandler,
SocketHandler, SMTPHandler and so on, but bitbake uses it's own
handler bb.event.LogHandler to handle the messages. The Loghandler
is defined in lib/bb/event.py, it re-defines the function emit which
is used to send the message to the UI(fire(record, None)), and
redefines the function filter, which is used to filter the messages,
determine whether to emit the message(all messages would be emitted
).
5, Initialize server
server = server.BitBakeServer()
server.initServer()
The BitBakeServer() is defined in lib/bb/server/process.py, and the
server.initServer() will init the server:
--> ui_channel
/
Pipe ()
\
--> server_channel ->\ --> command_channel
\ / --> event_queue
----> def waitEvent -> ProcessServer ----> event
/ / \ ---> _idlefunctions
ProcessEventQueue --------->/ ---> keep_running
\
----> def getEvent
The Pipe () from multiprocessing returns two duplex connection
objects connected by a pipe here, The ui_channel is used for ui to
send and receive messages with the server, and the server_channel is
used for cooker to send and receive messages with the server.
The ProcessEventQueue is defined in lib/bb/server/process.py: It is
the subclass of multiprocessing.queues.Queue, the
multiprocessing.queues.Queue has not defined waitEvent and getEvent
which are needed by the UI, so define them here.
The ProcessServer is the real server that we used, it is the
subclass of Process which is from the multiprocessing:
* command_channel = server_channal
* event_queue = ProcessEventQueue
* event = EventAdapter(event_queue)
The EventAdapter is used for adapting the API of the queue, the
bb.event need call a send() method, but our actual queue only has
put(), so convert the send() to put().
* The _idlefunctions is used for saving the functions which would be
run by the server, the register_idle_function will set the value.
* keep_running = Event()
The Event is from multiprocessing, it will run the
keep_running.set(), the set() is to set the internal flag to true.
All processes waiting for it to become true are awakened. Processes
that call wait() once the flag is true will not block at all.
6, Register the idle functions
idle = server.getServerIdleCB()
getServerIdleCB --> register_idle_function --> _idlefunctions
This is used for assigning a value for _idlefunctions in the
server.
The function buildFile and buildTargets will use
server_registration_cb, these are the most important two functions
for building. And the configuration is one of the most important
data which saves the configuration data for a build.
7, Initialize cooker
cooker = bb.cooker.BBCooker(configuration, idle, initialenv)
The BBCooker class is defined in lib/bb/cooker.py, and the
initialization is:
+----------------+
|stateShutdown |
|stateStop |
|getCmdLineAction|
BBCooker -> cmds_sync ->|getVariable |
/ | \ / |setVariable |
/ | \ / |resetCooker |
server_registration_cb | command +----------------+
configuration | \
configuration.event_data| \
| -> cmds_async (see list below)
V
LoadConfiguration
/ | \
/ | \
configuration.data | configuration.cmd
(bb.data.init) | (BB_DEFAULT_TASK or build)
V
parseConfigurationFiles
|
|
V
+---------------------------------------------------+
| bb.parse.init_parser --> bb.siggen.init |
| bblayers.conf (Signature generator)|
| BBLAYERS |
| BBPATH |
| conf/bitbake.conf |
| base.bblcass |
| INHERIT |
| register BBHANDLERS |
| bb.codeparser.parser_cache_init |
| save data to configuration.data |
+---------------------------------------------------+
* loadConfigurationData
Parse the config file(.conf, base.bbclass, and some .bbclass
used by the INHERIT), the data will be saved in configuration.data
(bb.data.init()), we can think it as a dictionary.
* command
Bitbake divides the commands into two kinds: cmds_sync and
cmds_async, cmds_sync has been list in the graph, and cmds_async
list is:
buildFile, buildTargets, generateDepTreeEvent, generateDotGraph,
generateTargetsTree, findConfigFiles, findFilesMatchingInDir,
findConfigFilePath, showVersions, showEnvironmentTarget,
showEnvironment, parseFiles, reparseFiles, compareRevisions
These commands are defined in lib/bb/command.py.
* The configuration.event_data would be used by the function
fire().
8, Parse command line action
cooker.parseCommandLine()
The parseCommandLine() will save the command line action in the
dictionary cook.commandlineAction, the default action is
buildTargets:
self.commandlineAction['action'] = ["buildTargets",
self.configuration.pkgs_to_build, self.configuration.cmd]
The ui will use the server to run "getCmdLineAction", and then
run buildTargets(self.configuration.pkgs_to_build, self.configuration.cmd)
to build the target. The buildTargets is defined in BBCooker.
9, Add the cooker to the server:
server.addcooker(cooker)
server.cooker = cooker
server.server.cooker = cooker
Add the cooker to BitBakeServer and ProcessServer, the first
"sever" is the "BitBakeServer", and the second server is the
real server "ProcessServer".
10, Start the server
server.detach(cooker_logfile)
This will invoke the detach which is defined in
lib/bb/server/process.py::BitBakeServer:
def detach(self, cooker_logfile):
self.server.start()
return
The start() is defined in multiprocessing, which means start the
process’ activity.
11, Start the ui
server_connection = server.establishConnection()
The BitBakeServerConnection will be used to communicate between
the server and the ui
server.launchUI(ui_main, server_connection.connection,
server_connection.events)
+-----------------+
launchUI ----> server_main ----> |main in knotty.py|
+-----------------+
* The default ui is knotty.py
* The main function in knotty.py is running now.
12, The ui runs command on the server
* Get the command line action
* Run the command one the server
* Waiting for the result from the server and handle them.
cmdline = server.runCommand(["command"])
If we run "bitbake target", then the flow is:
+-----------------------------------------------------------+
| server.runCommand(["getCmdLineAction"]) |
| +========+ | |
| | The ui | | return "buildTargets" |
| +========+ | |
| V |
| server.runCommand(["buildTargets"]) |
+-----------------------------------------------------------+
| ^
V |
command.py::buildTargets |
| |
V | return
+------------------------------------------------+ | the
| cooker.py::buildTargets | | result
| +======+ | | |
| |cooker| V | |
| +======+ updateCache | |
| (.bb and .bbclass files are parsed) | |
| taskdata = bb.taskdata.TaskData | |
| rq = bb.runqueue.RunQueue | |
| server_registration_cb(buildTargetsIdle, rq) | |
+------------------------------------------------+ |
| |
| |
V |
+----------------------------------------------------------+
| +===========+ process.py::run |
| |The server | | |
| |is running | | |
| +===========+ V |
| process.py::main |
+----------------------------------------------------------+
| ^
| | return the result
V |
+----------------------------------------------------------+
| +========+ cooker.py::buildTargetsIdle |
| | cooker | | |
| +========+ V |
| rq.execute_runqueue |
| (the tasks are in rq, and begin to run |
+----------------------------------------------------------+
// Robert
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: How bitbake works(Draft)
2011-12-05 3:27 How bitbake works(Draft) Robert Yang
@ 2011-12-07 13:01 ` Richard Purdie
2012-03-08 8:20 ` Robert Yang
0 siblings, 1 reply; 3+ messages in thread
From: Richard Purdie @ 2011-12-07 13:01 UTC (permalink / raw)
To: Robert Yang; +Cc: bitbake-devel, Rifenbark, Scott M
Hi Robert,
On Mon, 2011-12-05 at 11:27 +0800, Robert Yang wrote:
> Here is a doc about how bitbake works, it may help the bitbake
> newbie to understand how bitbake works, I'm very glad to add it
> to the public bitbake repository as documentation if it is useful.
> please feel free to give you comments.
I like this and I do think we need to improve the bitbake documentation.
We do need to ensure there is some consistent style and format for the
documents though so anything going into the repository really needs to
be added to the existing manual.
I'm also a little concerned that in its current form (listing arguments
to functions), this documentation would be high maintenance. If we could
separate it from the code implementation details a little bit, I think
it would make it less likely to bitrot quite as quickly.
Cheers,
Richard
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: How bitbake works(Draft)
2011-12-07 13:01 ` Richard Purdie
@ 2012-03-08 8:20 ` Robert Yang
0 siblings, 0 replies; 3+ messages in thread
From: Robert Yang @ 2012-03-08 8:20 UTC (permalink / raw)
To: Richard Purdie; +Cc: bitbake-devel, Rifenbark, Scott M
Hi Richard,
Here is an updated one.
Changes:
1) Remove the function's arguments as many as possible, try
to make it easy to maintain.
2) Change the name from "How bitbake works" to
"A Brief introduction to bitbake internal"
3) Try to explain rather than put the code there.
4) Some other trivial fix
Please see the document at the end of the email.
// Robert
On 12/07/2011 09:01 PM, Richard Purdie wrote:
> Hi Robert,
>
> On Mon, 2011-12-05 at 11:27 +0800, Robert Yang wrote:
>> Here is a doc about how bitbake works, it may help the bitbake
>> newbie to understand how bitbake works, I'm very glad to add it
>> to the public bitbake repository as documentation if it is useful.
>> please feel free to give you comments.
>
> I like this and I do think we need to improve the bitbake documentation.
> We do need to ensure there is some consistent style and format for the
> documents though so anything going into the repository really needs to
> be added to the existing manual.
>
> I'm also a little concerned that in its current form (listing arguments
> to functions), this documentation would be high maintenance. If we could
> separate it from the code implementation details a little bit, I think
> it would make it less likely to bitrot quite as quickly.
>
> Cheers,
>
> Richard
>
>
A Brief introduction to bitbake internal
The document tries to explain how a common bitbake build runs, it is
based on the yocto project.
The entrance of the bitbake is the main() function in bin/bitbake, the
main() function does the following things:
1, Parse command line options
* options are something like -b, -f, -c and so on.
2, Initialize the configuration
* The configuration saves the options data here, and will save more
later, the metadata(.conf, .inc, .bbclass and so on) will be saved
in configuration.data.
3, Get the ui and server from the configuration
* The ui is the user interface, you can specify which interface you
prefer.
* The server is used for running the tasks.
4, Initialize the logger
* The logger is used for logging the messages.
5, Initialize server
* Prepare the server for running
6, Register the idle functions
* Register the functions to the server, and the server will invoke
the function when it is idle.
7, Initialize cooker
* The cooker manages one bitbake build run, and it will save the
metadata to configuration.data here.
8, Parse command line action
* The actions are: fetch, unpack, patch and so on, the default is
build.
9, Add the cooker to the server
* Let the server can use the data in the cooker
10, Start the server and waiting for tasks to come to run
11, Start the ui
12, The ui submits commands to the server
* Then the build will start, and stop when it is done.
Let's see them one by one:
1, Parse command line options
* parser.parse_args
It returns two values: options and args, the options will be used
by the BBConfiguration
options, args = parser.parse_args
For example:
$bitbake gzip -cpatch
After the parse:
options.cmd = "patch"
The args saves the arguments which has not been matched by the
option:
args = ['/buildarea/lyang1/poky/bitbake/bin/bitbake', 'gzip']
2, Initialize the configuration from the commandline options
BBConfiguration
options -----------------------> configuration
configuration = BBConfiguration
BBConfiguration converts the dictionary of the "options" to the
member of BBConfiguration, and add a member called pkgs_to_build,
which will be used for saving the build pkg:
configuration.pkgs_to_build.extend
The configuration and configuration.data are the most important data
during a build, they save everything of the build.
3, Get the ui and server from the configuration
There are several UI and server in yocto, the default UI and server
are knotty(lib/bb/ui/knotty.py) and (lib/bb/server/process.py), you
can specify another ui by "bitbake -u <ui>", and "bitbake -t <server>"
for server.
The ui will be started by the server: server.launchUI.
The server will be used for running the tasks.
4, Initialize the logger
* set the log level.
bb.msg.init_msgconfig
* set the log handler
handler = bb.event.LogHandler()
logger.addHandler
There are lots of log handlers, for example: FileHandler,
SocketHandler, SMTPHandler and so on, but bitbake uses it's own
handler bb.event.LogHandler to handle the messages. This Loghandler
is defined in lib/bb/event.py, it re-defines the function emit
which is used to send the message to the UI, and redefines the
function filter which is used for determining whether to emit
the message(all messages would be emitted).
5, Initialize server
server = server.BitBakeServer()
server.initServer()
The BitBakeServer() is defined in lib/bb/server/process.py, and the
server.initServer() will init the server:
--> ui_channel
/
Pipe ()
\
--> server_channel ->\ --> command_channel
\ / --> event_queue
----> def waitEvent -> ProcessServer ----> event
/ / \ ---> _idlefunctions
ProcessEventQueue --------->/ ---> keep_running
\
----> def getEvent
The Pipe () from multiprocessing returns two duplex connection
objects connected by a pipe here, The ui_channel is used for ui to
send and receive messages with the server, and the server_channel is
used for cooker to send and receive messages with the server.
The ProcessEventQueue is defined in lib/bb/server/process.py: It is
the subclass of multiprocessing.queues.Queue, the
multiprocessing.queues.Queue has not defined waitEvent and getEvent
which are needed by the UI, so define them here.
The ProcessServer is the real server that we used, it is the
subclass of Process which is from the multiprocessing:
* command_channel = server_channal
To receive the command to run.
* event_queue = ProcessEventQueue
Supply the API waitEvent and getEvent, so that the ui can use the
same API for different servers.
* event = EventAdapter
The EventAdapter is used for adapting the API of the queue, the
bb.event need call a send() method, but our actual queue only has
put(), so convert the send() to put().
* The _idlefunctions is used for saving the functions which would be
run by the server, the register_idle_function will set the value.
* keep_running = Event()
The Event is from multiprocessing, it will run the
keep_running.set(), the set() is to set the internal flag to true.
All processes waiting for it to become true are awakened. Processes
that call wait() once the flag is true will not block at all.
6, Register the idle functions
idle = server.getServerIdleCB()
getServerIdleCB --> register_idle_function --> _idlefunctions
This is used for assigning a value for _idlefunctions in the
server.
The function buildFile and buildTargets will use
server_registration_cb, these are the most important two functions
for building.
7, Initialize cooker
cooker = bb.cooker.BBCooker
The BBCooker class is defined in lib/bb/cooker.py, and the
initialization is:
+----------------+
|stateShutdown |
|stateStop |
|getCmdLineAction|
BBCooker -> cmds_sync ->|getVariable |
/ | \ / |setVariable |
/ | \ / |resetCooker |
server_registration_cb | command +----------------+
configuration | \
configuration.event_data| \
| -> cmds_async (see list below)
V
LoadConfiguration
/ | \
/ | \
configuration.data | configuration.cmd
(bb.data.init) | (BB_DEFAULT_TASK or build)
V
parseConfigurationFiles
|
|
V
+---------------------------------------------------+
| bb.parse.init_parser --> bb.siggen.init |
| bblayers.conf (Signature generator)|
| BBLAYERS |
| BBPATH |
| conf/bitbake.conf |
| base.bblcass |
| INHERIT |
| register BBHANDLERS |
| bb.codeparser.parser_cache_init |
| save metadata to configuration.data |
+---------------------------------------------------+
* loadConfigurationData
Parse the config file(.conf, base.bbclass, and some .bbclass
used by the INHERIT), the data will be saved in configuration.data
(bb.data.init()), we can think it as a dictionary.
* command
Bitbake divides the commands into two kinds: cmds_sync and
cmds_async, cmds_sync has been list in the graph, and cmds_async
list is:
buildFile, buildTargets, generateDepTreeEvent, generateDotGraph,
generateTargetsTree, findConfigFiles, findFilesMatchingInDir,
findConfigFilePath, showVersions, showEnvironmentTarget,
showEnvironment, parseFiles, reparseFiles, compareRevisions
These commands are defined in lib/bb/command.py.
* The configuration.event_data would be used by the function
fire().
8, Parse command line action
cooker.parseCommandLine()
The parseCommandLine() will save the command line action in the
dictionary cook.commandlineAction, the default action is
buildTargets:
self.commandlineAction['action'] = ["buildTargets",
self.configuration.pkgs_to_build, self.configuration.cmd]
The ui will use the server to run "getCmdLineAction", and then
run buildTargets to build the target. The buildTargets is defined
in BBCooker.
9, Add the cooker to the server:
server.cooker = cooker
Add the cooker to BitBakeServer, so the server can use the data in
the cooker
10, Start the server and waiting for tasks to come to run
server.detach
This will invoke the function server.start() which is in
lib/bb/server/process.py::BitBakeServer:
The server.start() is defined in multiprocessing, which means start
the process’ activity.
11, Start the ui
server_connection = server.establishConnection()
The BitBakeServerConnection will be used to communicate between
the server and the ui
server.launchUI
* The main() function in knotty.py is running now.
12, The ui submits commands to the server
* Get the command line action
* Run the command one the server
* Waiting for the result from the server and handle them.
cmdline = server.runCommand
If we run "bitbake target", then the flow is:
+-----------------------------------------------------------+
| server.runCommand |
| +========+ | |
| | The ui | | return "buildTargets" |
| +========+ | |
| V |
| server.runCommand(["buildTargets"]) |
+-----------------------------------------------------------+
| ^
V |
command.py::buildTargets |
| |
V | return
+------------------------------------------------+ | the
| cooker.py::buildTargets | | result
| +======+ | | |
| |cooker| V | |
| +======+ updateCache | |
| (.bb and .bbclass files are parsed) | |
| taskdata = bb.taskdata.TaskData | |
| rq = bb.runqueue.RunQueue | |
| server_registration_cb | |
+------------------------------------------------+ |
| |
| |
V |
+----------------------------------------------------------+
| +===========+ process.py::run |
| |The server | | |
| |is running | | |
| +===========+ V |
| process.py::main |
+----------------------------------------------------------+
| ^
| | return the result
V |
+----------------------------------------------------------+
| +========+ cooker.py::buildTargetsIdle |
| | cooker | | |
| +========+ V |
| rq.execute_runqueue |
| (the tasks are in rq, and begin to run) |
+----------------------------------------------------------+
The build will stop when the tasks in the runqueue has been done.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2012-03-08 8:29 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-12-05 3:27 How bitbake works(Draft) Robert Yang
2011-12-07 13:01 ` Richard Purdie
2012-03-08 8:20 ` Robert Yang
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.