BitBake changes in the Yocto Project 1.5 cycle

BitBake changes in the Yocto Project 1.5 cycle

[Richard Purdie]

I've been giving some thought to where BitBake needs to go in the future
in order to deliver for its users. It started life as a commandline
utility and its grown a lot since it was first created. I think there
are some key decisions that need to be taken to ensure its future
growth.

The first proposal is that we should change the BitBake server so it
becomes memory resident. This means that the first time you run "bitbake
X", the server loads into memory, then subsequent BitBake commands would
just connect to the server and do things. We'd add in some kind of
timeout of say 15 minutes so that it would gracefully exit.

The reason for doing this is simple, it would allow commands to be much
more responsive rather than having the cache/configuration loading each
time which is where our current overhead is. Obviously it would detect
changes to things like MACHINE setting, local.conf and re-parse as
normal in those cases. The intent would be to speed up the interaction
with the system so you don't have the annoying delays/lag.

This would then lead into the second proposal which is to add better
support for new commands. This would be along the lines of some of
Chris' work with bb, the bitbake-env script and so on but with the added
advantage of being able to connect to the server for info. It would
allow us to add query commands for things like the available
PACKAGECONFIG options/settings much more easily than the current
infrastructure. There are some issues about whether these would belong
as part of BitBake or OE and we'd need to figure that out.

The third proposal is to do away with the BitBake wrapper script in
OE-Core for pseudo. Doing this within BitBake is going to be hard and
require some rework but ultimately, I think it will be worth it from a
performance perspective and also stops some of the user confusion about
BitBake running twice (two sets of log files and so on).

These changes should also be in keeping with the expanded UI work and
options such as WebHob and allowing remote use of multiple UIs connected
to servers.

Feedback/comments/suggestions etc./ welcome as always.

I've cc'd the OE-Core list but further discussion should be taken to
bitbake-devel.

Cheers,

Richard

Re: [OE-core] BitBake changes in the Yocto Project 1.5 cycle

[Trevor Woerner]

On Mon, Apr 22, 2013 at 10:16 AM, Richard Purdie <rpurdie@rpsys.net> wrote:
> We'd add in some kind of
> timeout of say 15 minutes so that it would gracefully exit.

Hopefully this time-out will start at the completion of the most
recent bitbake command, rather than from its start?

> The reason for doing this is simple, it would allow commands to be much
> more responsive rather than having the cache/configuration loading each
> time which is where our current overhead is.

This sounds like a very nice feature to have.

> This would then lead into the second proposal which is to add better
> support for new commands.

It would be nice if there was a way to zero-in on an existing
environment variable. For example:

    $ bitbake -e <image>

produces a lot of information which I then have to parse in order to
figure out how a given environment variable is being set. It would be
nice if I could simply:

    $ bitbake -e <image> <variable>

and just get the pertinent parts.

Re: [OE-core] BitBake changes in the Yocto Project 1.5 cycle

[Robert P. J. Day]

On Mon, 22 Apr 2013, Trevor Woerner wrote:

> On Mon, Apr 22, 2013 at 10:16 AM, Richard Purdie <rpurdie@rpsys.net> wrote:
> > We'd add in some kind of
> > timeout of say 15 minutes so that it would gracefully exit.
>
> Hopefully this time-out will start at the completion of the most
> recent bitbake command, rather than from its start?
>
> > The reason for doing this is simple, it would allow commands to be much
> > more responsive rather than having the cache/configuration loading each
> > time which is where our current overhead is.
>
> This sounds like a very nice feature to have.
>
> > This would then lead into the second proposal which is to add better
> > support for new commands.
>
> It would be nice if there was a way to zero-in on an existing
> environment variable. For example:
>
>     $ bitbake -e <image>
>
> produces a lot of information which I then have to parse in order to
> figure out how a given environment variable is being set. It would be
> nice if I could simply:
>
>     $ bitbake -e <image> <variable>
>
> and just get the pertinent parts.

  i think bitbake-env would do this for you.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: [OE-core] BitBake changes in the Yocto Project 1.5 cycle

[Trevor Woerner]

[-- Attachment #1: Type: text/plain, Size: 1709 bytes --]

On Mon, Apr 22, 2013 at 12:57 PM, Robert P. J. Day <rpjday@crashcourse.ca>
wrote:
> On Mon, 22 Apr 2013, Trevor Woerner wrote:
>> On Mon, Apr 22, 2013 at 10:16 AM, Richard Purdie <rpurdie@rpsys.net>
wrote:
>> > This would then lead into the second proposal which is to add better
>> > support for new commands.
>>
>> It would be nice if there was a way to zero-in on an existing
>> environment variable. For example:
>>
>>     $ bitbake -e <image>
>>
>> produces a lot of information which I then have to parse in order to
>> figure out how a given environment variable is being set. It would be
>> nice if I could simply:
>>
>>     $ bitbake -e <image> <variable>
>>
>> and just get the pertinent parts.
>
>   i think bitbake-env would do this for you.
>
> rday

If I use bitbake-env I just get the current value:

    $ bitbake-env OE_TERMINAL
    OE_TERMINAL="screen"

but if I use "bitbake -e" I get a better pictures of why that variable is
what it is
(which I've found very useful):

    $ bitbake -e | less
    [...snip...]
    # $OE_TERMINAL [4 operations]
    #   set /home/trevor/devel/yocto/builds/fullbuild/conf/local.conf:23
    #     "screen"
    #   set?
/home/trevor/devel/yocto/git/poky/meta/classes/terminal.bbclass:1
    #     "auto"
    #   set
/home/trevor/devel/yocto/git/poky/meta/classes/terminal.bbclass:2
    #     [type] "choice"
    #   set
/home/trevor/devel/yocto/git/poky/meta/classes/terminal.bbclass:5
    #     [choices] "auto none                         ${@" ".join(o.name
                                  for o in oe.terminal.prioritized())}"
    # computed:
    #   "screen"
    OE_TERMINAL="screen"
    [...snip...]

[-- Attachment #2: Type: text/html, Size: 2189 bytes --]

Re: [OE-core] BitBake changes in the Yocto Project 1.5 cycle

[Jason Wessel]

On 04/22/2013 11:51 AM, Trevor Woerner wrote:
> On Mon, Apr 22, 2013 at 10:16 AM, Richard Purdie <rpurdie@rpsys.net> wrote:
>> We'd add in some kind of
>> timeout of say 15 minutes so that it would gracefully exit.
> Hopefully this time-out will start at the completion of the most
> recent bitbake command, rather than from its start?


It would be good if it was configurable.   I don't mind leaving and instance running 8 hours for example, so I pay the price once for the project for the day.  Also we will want a way to query the status of the instance that is running and kill it.  If the instance is periodically checking that its local.conf still exists when it is idle, it can shut itself down if the build directory has been removed.   Obviously these things are all in the "nice to have" camp.

Cheers,
Jason.

Re: [OE-core] BitBake changes in the Yocto Project 1.5 cycle

[Marcin Juszkiewicz]

W dniu 22.04.2013 16:16, Richard Purdie pisze:
> I've been giving some thought to where BitBake needs to go in the 
> future in order to deliver for its users. It started life as a 
> commandline utility and its grown a lot since it was first created.
> I think there are some key decisions that need to be taken to ensure 
> its future growth.

There is one thing which kind of bothers me. "ERROR" message which is
just a warning. Like:

ERROR: libelf is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libasm is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libdw is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libdw-dev is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libasm-dev is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libelf-dev is listed in PACKAGES multiple times, this leads to packaging errors.

If it is error then let BitBake exit instead of doing task. If it is
not error but just a warning then let it be renamed?

My recent build failed with:

--------
NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Started
NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Succeeded
NOTE: Tasks Summary: Attempted 2247 tasks of which 915 didn't need to be rerun and all succeeded.

Summary: There were 3 WARNING messages shown.
Summary: There were 6 ERROR messages shown, returning a non-zero exit code.
Build step 'Execute shell' marked build as failure
--------

"all succeeded" does not look like a reason to fail a build.

Re: BitBake changes in the Yocto Project 1.5 cycle

[Marcin Juszkiewicz]

W dniu 22.04.2013 16:16, Richard Purdie pisze:
> I've been giving some thought to where BitBake needs to go in the 
> future in order to deliver for its users. It started life as a 
> commandline utility and its grown a lot since it was first created.
> I think there are some key decisions that need to be taken to ensure 
> its future growth.

There is one thing which kind of bothers me. "ERROR" message which is
just a warning. Like:

ERROR: libelf is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libasm is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libdw is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libdw-dev is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libasm-dev is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libelf-dev is listed in PACKAGES multiple times, this leads to packaging errors.

If it is error then let BitBake exit instead of doing task. If it is
not error but just a warning then let it be renamed?

My recent build failed with:

--------
NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Started
NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Succeeded
NOTE: Tasks Summary: Attempted 2247 tasks of which 915 didn't need to be rerun and all succeeded.

Summary: There were 3 WARNING messages shown.
Summary: There were 6 ERROR messages shown, returning a non-zero exit code.
Build step 'Execute shell' marked build as failure
--------

"all succeeded" does not look like a reason to fail a build.

Re: [OE-core] BitBake changes in the Yocto Project 1.5 cycle

[Richard Purdie]

On Mon, 2013-04-22 at 21:12 +0200, Marcin Juszkiewicz wrote:
> W dniu 22.04.2013 16:16, Richard Purdie pisze:
> > I've been giving some thought to where BitBake needs to go in the 
> > future in order to deliver for its users. It started life as a 
> > commandline utility and its grown a lot since it was first created.
> > I think there are some key decisions that need to be taken to ensure 
> > its future growth.
> 
> There is one thing which kind of bothers me. "ERROR" message which is
> just a warning. Like:
> 
> ERROR: libelf is listed in PACKAGES multiple times, this leads to packaging errors.
> ERROR: libasm is listed in PACKAGES multiple times, this leads to packaging errors.
> ERROR: libdw is listed in PACKAGES multiple times, this leads to packaging errors.
> ERROR: libdw-dev is listed in PACKAGES multiple times, this leads to packaging errors.
> ERROR: libasm-dev is listed in PACKAGES multiple times, this leads to packaging errors.
> ERROR: libelf-dev is listed in PACKAGES multiple times, this leads to packaging errors.
> 
> If it is error then let BitBake exit instead of doing task. If it is
> not error but just a warning then let it be renamed?

What we're trying to do is move everything to use a standard mechanism
for reporting issues of this type. With insane.bbclass, you can elect
whether a given type of error is a warning or errors and fails the task.

There are however several places where bb.error is used without that
mechanism. I'd love to see them fixed, please file a bug for it...

> My recent build failed with:
> 
> --------
> NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Started
> NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Succeeded
> NOTE: Tasks Summary: Attempted 2247 tasks of which 915 didn't need to be rerun and all succeeded.
> 
> Summary: There were 3 WARNING messages shown.
> Summary: There were 6 ERROR messages shown, returning a non-zero exit code.
> Build step 'Execute shell' marked build as failure
> --------
> 
> "all succeeded" does not look like a reason to fail a build.

Agreed, please open a bug for it.

FWIW Bitbake will set a non-zero exit code if any error messages appear,
as the message says. Whilst people find that confusing, I continue to
believe it is the right thing to be doing and we need to tighten up
things in other places as I describe above. That error code does help us
catch problems on the autobuilders and so forth.

Cheers,

Richard

Re: BitBake changes in the Yocto Project 1.5 cycle

[Richard Purdie]

On Mon, 2013-04-22 at 21:12 +0200, Marcin Juszkiewicz wrote:
> W dniu 22.04.2013 16:16, Richard Purdie pisze:
> > I've been giving some thought to where BitBake needs to go in the 
> > future in order to deliver for its users. It started life as a 
> > commandline utility and its grown a lot since it was first created.
> > I think there are some key decisions that need to be taken to ensure 
> > its future growth.
> 
> There is one thing which kind of bothers me. "ERROR" message which is
> just a warning. Like:
> 
> ERROR: libelf is listed in PACKAGES multiple times, this leads to packaging errors.
> ERROR: libasm is listed in PACKAGES multiple times, this leads to packaging errors.
> ERROR: libdw is listed in PACKAGES multiple times, this leads to packaging errors.
> ERROR: libdw-dev is listed in PACKAGES multiple times, this leads to packaging errors.
> ERROR: libasm-dev is listed in PACKAGES multiple times, this leads to packaging errors.
> ERROR: libelf-dev is listed in PACKAGES multiple times, this leads to packaging errors.
> 
> If it is error then let BitBake exit instead of doing task. If it is
> not error but just a warning then let it be renamed?

What we're trying to do is move everything to use a standard mechanism
for reporting issues of this type. With insane.bbclass, you can elect
whether a given type of error is a warning or errors and fails the task.

There are however several places where bb.error is used without that
mechanism. I'd love to see them fixed, please file a bug for it...

> My recent build failed with:
> 
> --------
> NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Started
> NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Succeeded
> NOTE: Tasks Summary: Attempted 2247 tasks of which 915 didn't need to be rerun and all succeeded.
> 
> Summary: There were 3 WARNING messages shown.
> Summary: There were 6 ERROR messages shown, returning a non-zero exit code.
> Build step 'Execute shell' marked build as failure
> --------
> 
> "all succeeded" does not look like a reason to fail a build.

Agreed, please open a bug for it.

FWIW Bitbake will set a non-zero exit code if any error messages appear,
as the message says. Whilst people find that confusing, I continue to
believe it is the right thing to be doing and we need to tighten up
things in other places as I describe above. That error code does help us
catch problems on the autobuilders and so forth.

Cheers,

Richard

Re: [OE-core] BitBake changes in the Yocto Project 1.5 cycle

[Marcin Juszkiewicz]

W dniu 23.04.2013 11:04, Richard Purdie pisze:
>> > "all succeeded" does not look like a reason to fail a build.
> Agreed, please open a bug for it.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=4397

Re: BitBake changes in the Yocto Project 1.5 cycle

[Marcin Juszkiewicz]

W dniu 23.04.2013 11:04, Richard Purdie pisze:
>> > "all succeeded" does not look like a reason to fail a build.
> Agreed, please open a bug for it.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=4397

Re: [OE-core] BitBake changes in the Yocto Project 1.5 cycle

[Robert Yang]

On 04/23/2013 05:27 PM, Marcin Juszkiewicz wrote:
> W dniu 23.04.2013 11:04, Richard Purdie pisze:
>>>> "all succeeded" does not look like a reason to fail a build.
>> Agreed, please open a bug for it.
>
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=4397
>

I marked 4397 as the duplication of this one:-).

// Robert

> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>

Re: BitBake changes in the Yocto Project 1.5 cycle

[Robert Yang]

On 04/23/2013 05:27 PM, Marcin Juszkiewicz wrote:
> W dniu 23.04.2013 11:04, Richard Purdie pisze:
>>>> "all succeeded" does not look like a reason to fail a build.
>> Agreed, please open a bug for it.
>
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=4397
>

I marked 4397 as the duplication of this one:-).

// Robert

> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>

Re: [OE-core] BitBake changes in the Yocto Project 1.5 cycle

[Hauser, Wolfgang (external)]

Hello, 
if there was a plan to divide bitbake into a server and a client part, I
would suggest to add a kind of "fetching proxy" functionality or run
mode to the bitbake sever part.

The reason behind this suggestion is: If you sitting behind a
restrictive proxy/firewall and want to use yocto (OE) you always get
problems to load from git:// svn:// or cvn:// URLs.

Here it may be useful to be able to run a bb fetch proxy outside of the
firewall that tunnels the protocols by https or ssh (via (ssl)port 443
like github does for git), or acts as a flexible download server, do the
fetching part and providing the download archives to an other server
through a common usable protocol tunnel.
May be this was a special issue, but it may ease the use of yocto (OE)
extremely.

Regards
Wolfgang Hauser

Re: BitBake changes in the Yocto Project 1.5 cycle

[Hauser, Wolfgang (external)]

Hello, 
if there was a plan to divide bitbake into a server and a client part, I
would suggest to add a kind of "fetching proxy" functionality or run
mode to the bitbake sever part.

The reason behind this suggestion is: If you sitting behind a
restrictive proxy/firewall and want to use yocto (OE) you always get
problems to load from git:// svn:// or cvn:// URLs.

Here it may be useful to be able to run a bb fetch proxy outside of the
firewall that tunnels the protocols by https or ssh (via (ssl)port 443
like github does for git), or acts as a flexible download server, do the
fetching part and providing the download archives to an other server
through a common usable protocol tunnel.
May be this was a special issue, but it may ease the use of yocto (OE)
extremely.

Regards
Wolfgang Hauser

Re: [OE-core] BitBake changes in the Yocto Project 1.5 cycle

[Robert Yang]

On 04/23/2013 05:04 PM, Richard Purdie wrote:
> On Mon, 2013-04-22 at 21:12 +0200, Marcin Juszkiewicz wrote:
>> W dniu 22.04.2013 16:16, Richard Purdie pisze:
>>> I've been giving some thought to where BitBake needs to go in the
>>> future in order to deliver for its users. It started life as a
>>> commandline utility and its grown a lot since it was first created.
>>> I think there are some key decisions that need to be taken to ensure
>>> its future growth.
>>
>> There is one thing which kind of bothers me. "ERROR" message which is
>> just a warning. Like:
>>
>> ERROR: libelf is listed in PACKAGES multiple times, this leads to packaging errors.
>> ERROR: libasm is listed in PACKAGES multiple times, this leads to packaging errors.
>> ERROR: libdw is listed in PACKAGES multiple times, this leads to packaging errors.
>> ERROR: libdw-dev is listed in PACKAGES multiple times, this leads to packaging errors.
>> ERROR: libasm-dev is listed in PACKAGES multiple times, this leads to packaging errors.
>> ERROR: libelf-dev is listed in PACKAGES multiple times, this leads to packaging errors.
>>
>> If it is error then let BitBake exit instead of doing task. If it is
>> not error but just a warning then let it be renamed?
>
> What we're trying to do is move everything to use a standard mechanism
> for reporting issues of this type. With insane.bbclass, you can elect
> whether a given type of error is a warning or errors and fails the task.
>
> There are however several places where bb.error is used without that
> mechanism. I'd love to see them fixed, please file a bug for it...
>

Filed a bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=4396

>> My recent build failed with:
>>
>> --------
>> NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Started
>> NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Succeeded
>> NOTE: Tasks Summary: Attempted 2247 tasks of which 915 didn't need to be rerun and all succeeded.
>>
>> Summary: There were 3 WARNING messages shown.
>> Summary: There were 6 ERROR messages shown, returning a non-zero exit code.
>> Build step 'Execute shell' marked build as failure
>> --------
>>
>> "all succeeded" does not look like a reason to fail a build.
>
> Agreed, please open a bug for it.
>

Filed another bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=4398

// Robert

> FWIW Bitbake will set a non-zero exit code if any error messages appear,
> as the message says. Whilst people find that confusing, I continue to
> believe it is the right thing to be doing and we need to tighten up
> things in other places as I describe above. That error code does help us
> catch problems on the autobuilders and so forth.
>
> Cheers,
>
> Richard
>
>
>
>
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
>
>

Re: [bitbake-devel] BitBake changes in the Yocto Project 1.5 cycle

[Robert Yang]

On 04/23/2013 05:04 PM, Richard Purdie wrote:
> On Mon, 2013-04-22 at 21:12 +0200, Marcin Juszkiewicz wrote:
>> W dniu 22.04.2013 16:16, Richard Purdie pisze:
>>> I've been giving some thought to where BitBake needs to go in the
>>> future in order to deliver for its users. It started life as a
>>> commandline utility and its grown a lot since it was first created.
>>> I think there are some key decisions that need to be taken to ensure
>>> its future growth.
>>
>> There is one thing which kind of bothers me. "ERROR" message which is
>> just a warning. Like:
>>
>> ERROR: libelf is listed in PACKAGES multiple times, this leads to packaging errors.
>> ERROR: libasm is listed in PACKAGES multiple times, this leads to packaging errors.
>> ERROR: libdw is listed in PACKAGES multiple times, this leads to packaging errors.
>> ERROR: libdw-dev is listed in PACKAGES multiple times, this leads to packaging errors.
>> ERROR: libasm-dev is listed in PACKAGES multiple times, this leads to packaging errors.
>> ERROR: libelf-dev is listed in PACKAGES multiple times, this leads to packaging errors.
>>
>> If it is error then let BitBake exit instead of doing task. If it is
>> not error but just a warning then let it be renamed?
>
> What we're trying to do is move everything to use a standard mechanism
> for reporting issues of this type. With insane.bbclass, you can elect
> whether a given type of error is a warning or errors and fails the task.
>
> There are however several places where bb.error is used without that
> mechanism. I'd love to see them fixed, please file a bug for it...
>

Filed a bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=4396

>> My recent build failed with:
>>
>> --------
>> NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Started
>> NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Succeeded
>> NOTE: Tasks Summary: Attempted 2247 tasks of which 915 didn't need to be rerun and all succeeded.
>>
>> Summary: There were 3 WARNING messages shown.
>> Summary: There were 6 ERROR messages shown, returning a non-zero exit code.
>> Build step 'Execute shell' marked build as failure
>> --------
>>
>> "all succeeded" does not look like a reason to fail a build.
>
> Agreed, please open a bug for it.
>

Filed another bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=4398

// Robert

> FWIW Bitbake will set a non-zero exit code if any error messages appear,
> as the message says. Whilst people find that confusing, I continue to
> believe it is the right thing to be doing and we need to tighten up
> things in other places as I describe above. That error code does help us
> catch problems on the autobuilders and so forth.
>
> Cheers,
>
> Richard
>
>
>
>
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
>
>

Re: [OE-core] BitBake changes in the Yocto Project 1.5 cycle

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> Richard Purdie
> Sent: den 22 april 2013 16:16
> To: bitbake-devel
> Cc: openembedded-core
> Subject: [OE-core] BitBake changes in the Yocto Project 1.5 cycle
> 
> I've been giving some thought to where BitBake needs to go in the
> future in order to deliver for its users. It started life as a 
> commandline utility and its grown a lot since it was first created. 
> I think there are some key decisions that need to be taken to 
> ensure its future growth.
> 
> The first proposal is that we should change the BitBake server so 
> it becomes memory resident. This means that the first time you run
> "bitbake X", the server loads into memory, then subsequent BitBake 
> commands would just connect to the server and do things. We'd add 
> in some kind of timeout of say 15 minutes so that it would 
> gracefully exit.
> 
> The reason for doing this is simple, it would allow commands to be 
> much more responsive rather than having the cache/configuration 
> loading each time which is where our current overhead is. Obviously 
> it would detect changes to things like MACHINE setting, local.conf 
> and re-parse as normal in those cases. The intent would be to speed 
> up the interaction with the system so you don't have the annoying 
> delays/lag.

How will a bitbake command know which server to connect to in case 
of multiple concurrent builds of different products on the same 
computer? Or do you intend for one server to keep track of all builds?

[cut]

> These changes should also be in keeping with the expanded UI work 
> and options such as WebHob and allowing remote use of multiple UIs
> connected to servers.

How should a remote UI know which server to connect to (if there 
are multiple ones)?

> Feedback/comments/suggestions etc./ welcome as always.
> 
> I've cc'd the OE-Core list but further discussion should be taken to
> bitbake-devel.
> 
> Cheers,
> 
> Richard

//Peter

Re: BitBake changes in the Yocto Project 1.5 cycle

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> Richard Purdie
> Sent: den 22 april 2013 16:16
> To: bitbake-devel
> Cc: openembedded-core
> Subject: [OE-core] BitBake changes in the Yocto Project 1.5 cycle
> 
> I've been giving some thought to where BitBake needs to go in the
> future in order to deliver for its users. It started life as a 
> commandline utility and its grown a lot since it was first created. 
> I think there are some key decisions that need to be taken to 
> ensure its future growth.
> 
> The first proposal is that we should change the BitBake server so 
> it becomes memory resident. This means that the first time you run
> "bitbake X", the server loads into memory, then subsequent BitBake 
> commands would just connect to the server and do things. We'd add 
> in some kind of timeout of say 15 minutes so that it would 
> gracefully exit.
> 
> The reason for doing this is simple, it would allow commands to be 
> much more responsive rather than having the cache/configuration 
> loading each time which is where our current overhead is. Obviously 
> it would detect changes to things like MACHINE setting, local.conf 
> and re-parse as normal in those cases. The intent would be to speed 
> up the interaction with the system so you don't have the annoying 
> delays/lag.

How will a bitbake command know which server to connect to in case 
of multiple concurrent builds of different products on the same 
computer? Or do you intend for one server to keep track of all builds?

[cut]

> These changes should also be in keeping with the expanded UI work 
> and options such as WebHob and allowing remote use of multiple UIs
> connected to servers.

How should a remote UI know which server to connect to (if there 
are multiple ones)?

> Feedback/comments/suggestions etc./ welcome as always.
> 
> I've cc'd the OE-Core list but further discussion should be taken to
> bitbake-devel.
> 
> Cheers,
> 
> Richard

//Peter

Re: [OE-core] BitBake changes in the Yocto Project 1.5 cycle

[Richard Purdie]

On Tue, 2013-04-23 at 10:13 +0200, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: openembedded-core-bounces@lists.openembedded.org
> > [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> > Richard Purdie
> > Sent: den 22 april 2013 16:16
> > To: bitbake-devel
> > Cc: openembedded-core
> > Subject: [OE-core] BitBake changes in the Yocto Project 1.5 cycle
> > 
> > I've been giving some thought to where BitBake needs to go in the
> > future in order to deliver for its users. It started life as a 
> > commandline utility and its grown a lot since it was first created. 
> > I think there are some key decisions that need to be taken to 
> > ensure its future growth.
> > 
> > The first proposal is that we should change the BitBake server so 
> > it becomes memory resident. This means that the first time you run
> > "bitbake X", the server loads into memory, then subsequent BitBake 
> > commands would just connect to the server and do things. We'd add 
> > in some kind of timeout of say 15 minutes so that it would 
> > gracefully exit.
> > 
> > The reason for doing this is simple, it would allow commands to be 
> > much more responsive rather than having the cache/configuration 
> > loading each time which is where our current overhead is. Obviously 
> > it would detect changes to things like MACHINE setting, local.conf 
> > and re-parse as normal in those cases. The intent would be to speed 
> > up the interaction with the system so you don't have the annoying 
> > delays/lag.
> 
> How will a bitbake command know which server to connect to in case 
> of multiple concurrent builds of different products on the same 
> computer? Or do you intend for one server to keep track of all builds?

There would continue to be one server per build. I tried to keep away
from specifics of the exact implementation since the answer right now is
"don't know". I do have ideas, for example we may do this through some
kind of file in the build directory like the bitbake.lock file and/or
some kind of variable behaving like DISPLAY. I don't really want to
reinvent the wheel though and am open to other ideas. Dbus falls down
with the "remote" part of what we want to do since its typically only
been used locally and whilst people had ideas about remote parts for it,
I don't think they were ever developed extensively.

> > These changes should also be in keeping with the expanded UI work 
> > and options such as WebHob and allowing remote use of multiple UIs
> > connected to servers.
> 
> How should a remote UI know which server to connect to (if there 
> are multiple ones)?

Right now there are commandline options to bitbake for this. This may
become some kind of DISPLAY equivalent environment variable, not sure
yet.

Cheers,

Richard