Openembedded Core Discussions
 help / color / mirror / Atom feed
* BitBake changes in the Yocto Project 1.5 cycle
@ 2013-04-22 14:16 Richard Purdie
  2013-04-22 19:12 ` Marcin Juszkiewicz
  2013-04-23  8:13 ` Peter Kjellerstedt
  0 siblings, 2 replies; 8+ messages in thread
From: Richard Purdie @ 2013-04-22 14:16 UTC (permalink / raw)
  To: bitbake-devel; +Cc: openembedded-core

I've been giving some thought to where BitBake needs to go in the future
in order to deliver for its users. It started life as a commandline
utility and its grown a lot since it was first created. I think there
are some key decisions that need to be taken to ensure its future
growth.

The first proposal is that we should change the BitBake server so it
becomes memory resident. This means that the first time you run "bitbake
X", the server loads into memory, then subsequent BitBake commands would
just connect to the server and do things. We'd add in some kind of
timeout of say 15 minutes so that it would gracefully exit.

The reason for doing this is simple, it would allow commands to be much
more responsive rather than having the cache/configuration loading each
time which is where our current overhead is. Obviously it would detect
changes to things like MACHINE setting, local.conf and re-parse as
normal in those cases. The intent would be to speed up the interaction
with the system so you don't have the annoying delays/lag.

This would then lead into the second proposal which is to add better
support for new commands. This would be along the lines of some of
Chris' work with bb, the bitbake-env script and so on but with the added
advantage of being able to connect to the server for info. It would
allow us to add query commands for things like the available
PACKAGECONFIG options/settings much more easily than the current
infrastructure. There are some issues about whether these would belong
as part of BitBake or OE and we'd need to figure that out.

The third proposal is to do away with the BitBake wrapper script in
OE-Core for pseudo. Doing this within BitBake is going to be hard and
require some rework but ultimately, I think it will be worth it from a
performance perspective and also stops some of the user confusion about
BitBake running twice (two sets of log files and so on).

These changes should also be in keeping with the expanded UI work and
options such as WebHob and allowing remote use of multiple UIs connected
to servers.

Feedback/comments/suggestions etc./ welcome as always.

I've cc'd the OE-Core list but further discussion should be taken to
bitbake-devel.

Cheers,

Richard




^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: BitBake changes in the Yocto Project 1.5 cycle
  2013-04-22 14:16 BitBake changes in the Yocto Project 1.5 cycle Richard Purdie
@ 2013-04-22 19:12 ` Marcin Juszkiewicz
  2013-04-23  9:04   ` Richard Purdie
  2013-04-23  8:13 ` Peter Kjellerstedt
  1 sibling, 1 reply; 8+ messages in thread
From: Marcin Juszkiewicz @ 2013-04-22 19:12 UTC (permalink / raw)
  To: Richard Purdie; +Cc: bitbake-devel, openembedded-core

W dniu 22.04.2013 16:16, Richard Purdie pisze:
> I've been giving some thought to where BitBake needs to go in the 
> future in order to deliver for its users. It started life as a 
> commandline utility and its grown a lot since it was first created.
> I think there are some key decisions that need to be taken to ensure 
> its future growth.

There is one thing which kind of bothers me. "ERROR" message which is
just a warning. Like:

ERROR: libelf is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libasm is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libdw is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libdw-dev is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libasm-dev is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libelf-dev is listed in PACKAGES multiple times, this leads to packaging errors.

If it is error then let BitBake exit instead of doing task. If it is
not error but just a warning then let it be renamed?

My recent build failed with:

--------
NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Started
NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Succeeded
NOTE: Tasks Summary: Attempted 2247 tasks of which 915 didn't need to be rerun and all succeeded.

Summary: There were 3 WARNING messages shown.
Summary: There were 6 ERROR messages shown, returning a non-zero exit code.
Build step 'Execute shell' marked build as failure
--------

"all succeeded" does not look like a reason to fail a build.



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: BitBake changes in the Yocto Project 1.5 cycle
  2013-04-22 14:16 BitBake changes in the Yocto Project 1.5 cycle Richard Purdie
  2013-04-22 19:12 ` Marcin Juszkiewicz
@ 2013-04-23  8:13 ` Peter Kjellerstedt
  1 sibling, 0 replies; 8+ messages in thread
From: Peter Kjellerstedt @ 2013-04-23  8:13 UTC (permalink / raw)
  To: Richard Purdie, bitbake-devel; +Cc: openembedded-core

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> Richard Purdie
> Sent: den 22 april 2013 16:16
> To: bitbake-devel
> Cc: openembedded-core
> Subject: [OE-core] BitBake changes in the Yocto Project 1.5 cycle
> 
> I've been giving some thought to where BitBake needs to go in the
> future in order to deliver for its users. It started life as a 
> commandline utility and its grown a lot since it was first created. 
> I think there are some key decisions that need to be taken to 
> ensure its future growth.
> 
> The first proposal is that we should change the BitBake server so 
> it becomes memory resident. This means that the first time you run
> "bitbake X", the server loads into memory, then subsequent BitBake 
> commands would just connect to the server and do things. We'd add 
> in some kind of timeout of say 15 minutes so that it would 
> gracefully exit.
> 
> The reason for doing this is simple, it would allow commands to be 
> much more responsive rather than having the cache/configuration 
> loading each time which is where our current overhead is. Obviously 
> it would detect changes to things like MACHINE setting, local.conf 
> and re-parse as normal in those cases. The intent would be to speed 
> up the interaction with the system so you don't have the annoying 
> delays/lag.

How will a bitbake command know which server to connect to in case 
of multiple concurrent builds of different products on the same 
computer? Or do you intend for one server to keep track of all builds?

[cut]

> These changes should also be in keeping with the expanded UI work 
> and options such as WebHob and allowing remote use of multiple UIs
> connected to servers.

How should a remote UI know which server to connect to (if there 
are multiple ones)?

> Feedback/comments/suggestions etc./ welcome as always.
> 
> I've cc'd the OE-Core list but further discussion should be taken to
> bitbake-devel.
> 
> Cheers,
> 
> Richard

//Peter




^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: BitBake changes in the Yocto Project 1.5 cycle
  2013-04-22 19:12 ` Marcin Juszkiewicz
@ 2013-04-23  9:04   ` Richard Purdie
  2013-04-23  9:27     ` Marcin Juszkiewicz
  2013-04-23  9:28     ` [bitbake-devel] " Robert Yang
  0 siblings, 2 replies; 8+ messages in thread
From: Richard Purdie @ 2013-04-23  9:04 UTC (permalink / raw)
  To: Marcin Juszkiewicz; +Cc: bitbake-devel, openembedded-core

On Mon, 2013-04-22 at 21:12 +0200, Marcin Juszkiewicz wrote:
> W dniu 22.04.2013 16:16, Richard Purdie pisze:
> > I've been giving some thought to where BitBake needs to go in the 
> > future in order to deliver for its users. It started life as a 
> > commandline utility and its grown a lot since it was first created.
> > I think there are some key decisions that need to be taken to ensure 
> > its future growth.
> 
> There is one thing which kind of bothers me. "ERROR" message which is
> just a warning. Like:
> 
> ERROR: libelf is listed in PACKAGES multiple times, this leads to packaging errors.
> ERROR: libasm is listed in PACKAGES multiple times, this leads to packaging errors.
> ERROR: libdw is listed in PACKAGES multiple times, this leads to packaging errors.
> ERROR: libdw-dev is listed in PACKAGES multiple times, this leads to packaging errors.
> ERROR: libasm-dev is listed in PACKAGES multiple times, this leads to packaging errors.
> ERROR: libelf-dev is listed in PACKAGES multiple times, this leads to packaging errors.
> 
> If it is error then let BitBake exit instead of doing task. If it is
> not error but just a warning then let it be renamed?

What we're trying to do is move everything to use a standard mechanism
for reporting issues of this type. With insane.bbclass, you can elect
whether a given type of error is a warning or errors and fails the task.

There are however several places where bb.error is used without that
mechanism. I'd love to see them fixed, please file a bug for it...

> My recent build failed with:
> 
> --------
> NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Started
> NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Succeeded
> NOTE: Tasks Summary: Attempted 2247 tasks of which 915 didn't need to be rerun and all succeeded.
> 
> Summary: There were 3 WARNING messages shown.
> Summary: There were 6 ERROR messages shown, returning a non-zero exit code.
> Build step 'Execute shell' marked build as failure
> --------
> 
> "all succeeded" does not look like a reason to fail a build.

Agreed, please open a bug for it.

FWIW Bitbake will set a non-zero exit code if any error messages appear,
as the message says. Whilst people find that confusing, I continue to
believe it is the right thing to be doing and we need to tighten up
things in other places as I describe above. That error code does help us
catch problems on the autobuilders and so forth.

Cheers,

Richard







^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: BitBake changes in the Yocto Project 1.5 cycle
  2013-04-23  9:04   ` Richard Purdie
@ 2013-04-23  9:27     ` Marcin Juszkiewicz
  2013-04-23  9:31       ` Robert Yang
  2013-04-23  9:28     ` [bitbake-devel] " Robert Yang
  1 sibling, 1 reply; 8+ messages in thread
From: Marcin Juszkiewicz @ 2013-04-23  9:27 UTC (permalink / raw)
  To: bitbake-devel; +Cc: openembedded-core

W dniu 23.04.2013 11:04, Richard Purdie pisze:
>> > "all succeeded" does not look like a reason to fail a build.
> Agreed, please open a bug for it.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=4397



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [bitbake-devel] BitBake changes in the Yocto Project 1.5 cycle
  2013-04-23  9:04   ` Richard Purdie
  2013-04-23  9:27     ` Marcin Juszkiewicz
@ 2013-04-23  9:28     ` Robert Yang
  1 sibling, 0 replies; 8+ messages in thread
From: Robert Yang @ 2013-04-23  9:28 UTC (permalink / raw)
  To: Richard Purdie; +Cc: bitbake-devel, openembedded-core



On 04/23/2013 05:04 PM, Richard Purdie wrote:
> On Mon, 2013-04-22 at 21:12 +0200, Marcin Juszkiewicz wrote:
>> W dniu 22.04.2013 16:16, Richard Purdie pisze:
>>> I've been giving some thought to where BitBake needs to go in the
>>> future in order to deliver for its users. It started life as a
>>> commandline utility and its grown a lot since it was first created.
>>> I think there are some key decisions that need to be taken to ensure
>>> its future growth.
>>
>> There is one thing which kind of bothers me. "ERROR" message which is
>> just a warning. Like:
>>
>> ERROR: libelf is listed in PACKAGES multiple times, this leads to packaging errors.
>> ERROR: libasm is listed in PACKAGES multiple times, this leads to packaging errors.
>> ERROR: libdw is listed in PACKAGES multiple times, this leads to packaging errors.
>> ERROR: libdw-dev is listed in PACKAGES multiple times, this leads to packaging errors.
>> ERROR: libasm-dev is listed in PACKAGES multiple times, this leads to packaging errors.
>> ERROR: libelf-dev is listed in PACKAGES multiple times, this leads to packaging errors.
>>
>> If it is error then let BitBake exit instead of doing task. If it is
>> not error but just a warning then let it be renamed?
>
> What we're trying to do is move everything to use a standard mechanism
> for reporting issues of this type. With insane.bbclass, you can elect
> whether a given type of error is a warning or errors and fails the task.
>
> There are however several places where bb.error is used without that
> mechanism. I'd love to see them fixed, please file a bug for it...
>

Filed a bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=4396

>> My recent build failed with:
>>
>> --------
>> NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Started
>> NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Succeeded
>> NOTE: Tasks Summary: Attempted 2247 tasks of which 915 didn't need to be rerun and all succeeded.
>>
>> Summary: There were 3 WARNING messages shown.
>> Summary: There were 6 ERROR messages shown, returning a non-zero exit code.
>> Build step 'Execute shell' marked build as failure
>> --------
>>
>> "all succeeded" does not look like a reason to fail a build.
>
> Agreed, please open a bug for it.
>

Filed another bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=4398

// Robert

> FWIW Bitbake will set a non-zero exit code if any error messages appear,
> as the message says. Whilst people find that confusing, I continue to
> believe it is the right thing to be doing and we need to tighten up
> things in other places as I describe above. That error code does help us
> catch problems on the autobuilders and so forth.
>
> Cheers,
>
> Richard
>
>
>
>
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
>
>



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: BitBake changes in the Yocto Project 1.5 cycle
  2013-04-23  9:27     ` Marcin Juszkiewicz
@ 2013-04-23  9:31       ` Robert Yang
  2013-04-23 10:02         ` Hauser, Wolfgang (external)
  0 siblings, 1 reply; 8+ messages in thread
From: Robert Yang @ 2013-04-23  9:31 UTC (permalink / raw)
  To: Marcin Juszkiewicz; +Cc: bitbake-devel, openembedded-core



On 04/23/2013 05:27 PM, Marcin Juszkiewicz wrote:
> W dniu 23.04.2013 11:04, Richard Purdie pisze:
>>>> "all succeeded" does not look like a reason to fail a build.
>> Agreed, please open a bug for it.
>
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=4397
>

I marked 4397 as the duplication of this one:-).

// Robert

> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: BitBake changes in the Yocto Project 1.5 cycle
  2013-04-23  9:31       ` Robert Yang
@ 2013-04-23 10:02         ` Hauser, Wolfgang (external)
  0 siblings, 0 replies; 8+ messages in thread
From: Hauser, Wolfgang (external) @ 2013-04-23 10:02 UTC (permalink / raw)
  To: Patches and discussions about the oe-core layer, bitbake-devel

Hello, 
if there was a plan to divide bitbake into a server and a client part, I
would suggest to add a kind of "fetching proxy" functionality or run
mode to the bitbake sever part.

The reason behind this suggestion is: If you sitting behind a
restrictive proxy/firewall and want to use yocto (OE) you always get
problems to load from git:// svn:// or cvn:// URLs.

Here it may be useful to be able to run a bb fetch proxy outside of the
firewall that tunnels the protocols by https or ssh (via (ssl)port 443
like github does for git), or acts as a flexible download server, do the
fetching part and providing the download archives to an other server
through a common usable protocol tunnel.
May be this was a special issue, but it may ease the use of yocto (OE)
extremely.

Regards
Wolfgang Hauser



^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2013-04-23 10:33 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-04-22 14:16 BitBake changes in the Yocto Project 1.5 cycle Richard Purdie
2013-04-22 19:12 ` Marcin Juszkiewicz
2013-04-23  9:04   ` Richard Purdie
2013-04-23  9:27     ` Marcin Juszkiewicz
2013-04-23  9:31       ` Robert Yang
2013-04-23 10:02         ` Hauser, Wolfgang (external)
2013-04-23  9:28     ` [bitbake-devel] " Robert Yang
2013-04-23  8:13 ` Peter Kjellerstedt

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox