About PACKAGECONFIG audit

About PACKAGECONFIG audit

[wenzong fan]

Hi All,

We have seen some problems like this:

$ bitbake libcap

ld: warning: libaudit.so.1, needed by .../lib64/libpam.so, not found
.../lib64/libpam.so: undefined reference to `audit_open'
.../lib64/libpam.so: undefined reference to `audit_close'
......

The libaudit has been available and libpam build with it, after we run 
another build with sstate-cache and w/o libaudit, the error occurs. The 
solution in these cases is to use PACKAGECONFIG to lay out dependencies 
on optional packages and make them explicit. We need to run an audit and 
catch all of these issues.

A probable ways maybe:

1) Run world build to generate the sysroots as much as possible;
2) Run 'configure --help' on each package to spot all of the optional 
dependencies, and convert them into PACKAGECONFIG specs/or any other 
prompts.
3) Run #1, #2 as an audit script periodically.

Or could we run this check as part of a QA build step?


Thanks
Wenzong

Re: About PACKAGECONFIG audit

[Burton, Ross]

On 15 August 2013 10:29, wenzong fan <wenzong.fan@windriver.com> wrote:
> A probable ways maybe:
>
> 1) Run world build to generate the sysroots as much as possible;
> 2) Run 'configure --help' on each package to spot all of the optional
> dependencies, and convert them into PACKAGECONFIG specs/or any other
> prompts.
> 3) Run #1, #2 as an audit script periodically.

Step 2 would be hard to to automate and some optional dependencies
don't have configure options.  However Martin Jansa recently
contributed scripts/test-dependencies.sh which does repeated builds to
determine if there are any "floating" or missing dependencies, which
can be ran periodically as an audit.

Ross

Re: About PACKAGECONFIG audit

[Phil Blundell]

On Thu, 2013-08-15 at 17:29 +0800, wenzong fan wrote:
> Or could we run this check as part of a QA build step?

I think that could be done, in principle.  Now that "ld
--no-copy-dt-needed-entries" is the default, there's no reason that the
shlibs code in package.bbclass couldn't be taught to verify that all the
shared library providers it identifies come from recipes that are listed
in DEPENDS.  

The only difficulty is that you'd need to accept DEPENDS-of-DEPENDS
(e.g. a recipe which DEPENDS on gtk+ but also links with libatk or
something) and I don't think there's currently any straightforward way
for package.bbclass to generate the recursive dependency chain for an
arbitrary recipe.  An easy if inelegant workaround would be for it to
just stash the value of ${DEPENDS} along with the other shlibs data that
it writes out for each provider.

p.

Re: About PACKAGECONFIG audit

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1201 bytes --]

On Thu, Aug 15, 2013 at 10:43:16AM +0100, Phil Blundell wrote:
> On Thu, 2013-08-15 at 17:29 +0800, wenzong fan wrote:
> > Or could we run this check as part of a QA build step?
> 
> I think that could be done, in principle.  Now that "ld
> --no-copy-dt-needed-entries" is the default, there's no reason that the
> shlibs code in package.bbclass couldn't be taught to verify that all the
> shared library providers it identifies come from recipes that are listed
> in DEPENDS.  
> 
> The only difficulty is that you'd need to accept DEPENDS-of-DEPENDS
> (e.g. a recipe which DEPENDS on gtk+ but also links with libatk or
> something) and I don't think there's currently any straightforward way
> for package.bbclass to generate the recursive dependency chain for an
> arbitrary recipe.  An easy if inelegant workaround would be for it to
> just stash the value of ${DEPENDS} along with the other shlibs data that
> it writes out for each provider.

Something like that is outlined in WIP patch for
https://bugzilla.yoctoproject.org/show_bug.cgi?id=4628

but I got stuck on generating this recursive dependency chain

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: About PACKAGECONFIG audit

[Richard Purdie]

On Thu, 2013-08-15 at 10:43 +0100, Phil Blundell wrote:
> On Thu, 2013-08-15 at 17:29 +0800, wenzong fan wrote:
> > Or could we run this check as part of a QA build step?
> 
> I think that could be done, in principle.  Now that "ld
> --no-copy-dt-needed-entries" is the default, there's no reason that the
> shlibs code in package.bbclass couldn't be taught to verify that all the
> shared library providers it identifies come from recipes that are listed
> in DEPENDS.  
> 
> The only difficulty is that you'd need to accept DEPENDS-of-DEPENDS
> (e.g. a recipe which DEPENDS on gtk+ but also links with libatk or
> something) and I don't think there's currently any straightforward way
> for package.bbclass to generate the recursive dependency chain for an
> arbitrary recipe.  An easy if inelegant workaround would be for it to
> just stash the value of ${DEPENDS} along with the other shlibs data that
> it writes out for each provider.

Bitbake itself has that information and I suspect we should share it to
tasks. The exact form of that would be the harder thing to determine...

Cheers,

Richard

Re: About PACKAGECONFIG audit

[Phil Blundell]

On Fri, 2013-08-16 at 11:41 +0100, Richard Purdie wrote:
> On Thu, 2013-08-15 at 10:43 +0100, Phil Blundell wrote:
> > I don't think there's currently any straightforward way
> > for package.bbclass to generate the recursive dependency chain for an
> > arbitrary recipe. 
> 
> Bitbake itself has that information and I suspect we should share it to
> tasks. The exact form of that would be the harder thing to determine...

If it's something that bitbake already computes (or could compute
cheaply) then you could just arrange for it to add it to the datastore
as something like ${BB_COMPUTED_DEPENDS} before running each task.  Just
a flat textual list of recipes, de-duped but in arbitrary order, would
probably be good enough for all the use-cases we've talked about so far.
I guess the question is whether you need to do anything clever with
selecting the dependencies for particular tasks, or whether you could
just take the dependencies for all tasks that the recipe defines and
collapse them into a single list.  I think probably the latter.

If it's expensive to compute then I guess there'd need to be some sort
of bb.cooker.get_recursive_dependencies() kind of function that tasks
can call when they want the information.

p.

Re: About PACKAGECONFIG audit

[Paul Eggleton]

Hi Wenzong,

On Thursday 15 August 2013 17:29:46 wenzong fan wrote:
> We have seen some problems like this:
> 
> $ bitbake libcap
> 
> ld: warning: libaudit.so.1, needed by .../lib64/libpam.so, not found
> .../lib64/libpam.so: undefined reference to `audit_open'
> .../lib64/libpam.so: undefined reference to `audit_close'
> ......
> 
> The libaudit has been available and libpam build with it, after we run
> another build with sstate-cache and w/o libaudit, the error occurs. The
> solution in these cases is to use PACKAGECONFIG to lay out dependencies
> on optional packages and make them explicit. We need to run an audit and
> catch all of these issues.
> 
> A probable ways maybe:
> 
> 1) Run world build to generate the sysroots as much as possible;

(See Ross's reply for a pointer to the new test-dependencies script)

> 2) Run 'configure --help' on each package to spot all of the optional
> dependencies, and convert them into PACKAGECONFIG specs/or any other
> prompts.

Actually I have started along this line as well, putting the output of 
configure --help into buildhistory so you can see the differences. I'm not 
totally sure how useful it is at the moment and it will need some smooting out 
to avoid known differences (paths etc.).

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: About PACKAGECONFIG audit

[wenzong fan]

On 08/15/2013 05:44 PM, Paul Eggleton wrote:
> Hi Wenzong,
>
> On Thursday 15 August 2013 17:29:46 wenzong fan wrote:
>> We have seen some problems like this:
>>
>> $ bitbake libcap
>>
>> ld: warning: libaudit.so.1, needed by .../lib64/libpam.so, not found
>> .../lib64/libpam.so: undefined reference to `audit_open'
>> .../lib64/libpam.so: undefined reference to `audit_close'
>> ......
>>
>> The libaudit has been available and libpam build with it, after we run
>> another build with sstate-cache and w/o libaudit, the error occurs. The
>> solution in these cases is to use PACKAGECONFIG to lay out dependencies
>> on optional packages and make them explicit. We need to run an audit and
>> catch all of these issues.
>>
>> A probable ways maybe:
>>
>> 1) Run world build to generate the sysroots as much as possible;
>
> (See Ross's reply for a pointer to the new test-dependencies script)

That would be a good start point, I have run that script once, but looks 
not any failure occurs.

I'll find a box and run it periodically.

Thanks
Wenzong

>
>> 2) Run 'configure --help' on each package to spot all of the optional
>> dependencies, and convert them into PACKAGECONFIG specs/or any other
>> prompts.
>
> Actually I have started along this line as well, putting the output of
> configure --help into buildhistory so you can see the differences. I'm not
> totally sure how useful it is at the moment and it will need some smooting out
> to avoid known differences (paths etc.).
>
> Cheers,
> Paul
>

Re: About PACKAGECONFIG audit

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1353 bytes --]

On Thu, Aug 15, 2013 at 06:14:48PM +0800, wenzong fan wrote:
> On 08/15/2013 05:44 PM, Paul Eggleton wrote:
> > Hi Wenzong,
> >
> > On Thursday 15 August 2013 17:29:46 wenzong fan wrote:
> >> We have seen some problems like this:
> >>
> >> $ bitbake libcap
> >>
> >> ld: warning: libaudit.so.1, needed by .../lib64/libpam.so, not found
> >> .../lib64/libpam.so: undefined reference to `audit_open'
> >> .../lib64/libpam.so: undefined reference to `audit_close'
> >> ......
> >>
> >> The libaudit has been available and libpam build with it, after we run
> >> another build with sstate-cache and w/o libaudit, the error occurs. The
> >> solution in these cases is to use PACKAGECONFIG to lay out dependencies
> >> on optional packages and make them explicit. We need to run an audit and
> >> catch all of these issues.
> >>
> >> A probable ways maybe:
> >>
> >> 1) Run world build to generate the sysroots as much as possible;
> >
> > (See Ross's reply for a pointer to the new test-dependencies script)
> 
> That would be a good start point, I have run that script once, but looks 
> not any failure occurs.

I don't believe it, see "State of bitbake world" thread where I've sent
results from last 3 executions.

There is plenty of issues for everybody :)

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: About PACKAGECONFIG audit

[Randy MacLeod]

Add Joe Slater who owns US17463: PACKAGECONFIG clean-up.


On 13-08-15 05:29 AM, wenzong fan wrote:
> Hi All,
>
> We have seen some problems like this:
>
> $ bitbake libcap
>
> ld: warning: libaudit.so.1, needed by .../lib64/libpam.so, not found
> .../lib64/libpam.so: undefined reference to `audit_open'
> .../lib64/libpam.so: undefined reference to `audit_close'
> ......
>
> The libaudit has been available and libpam build with it, after we run
> another build with sstate-cache and w/o libaudit, the error occurs. The
> solution in these cases is to use PACKAGECONFIG to lay out dependencies
> on optional packages and make them explicit. We need to run an audit and
> catch all of these issues.
>
> A probable ways maybe:
>
> 1) Run world build to generate the sysroots as much as possible;
> 2) Run 'configure --help' on each package to spot all of the optional
> dependencies, and convert them into PACKAGECONFIG specs/or any other
> prompts.

We should try to automate or codify this step so that the output is
a list of oe-core package names.

> 3) Run #1, #2 as an audit script periodically.
>
> Or could we run this check as part of a QA build step?

Do you mean:
   scripts/test-dependencies.sh

// Randy

>
>
> Thanks
> Wenzong


-- 
# Randy MacLeod. SMTS, Linux, Wind River
Direct: 613.963.1350

Re: About PACKAGECONFIG audit

[wenzong fan]

On 08/20/2013 04:46 AM, Randy MacLeod wrote:
> Add Joe Slater who owns US17463: PACKAGECONFIG clean-up.
>
>
> On 13-08-15 05:29 AM, wenzong fan wrote:
>> Hi All,
>>
>> We have seen some problems like this:
>>
>> $ bitbake libcap
>>
>> ld: warning: libaudit.so.1, needed by .../lib64/libpam.so, not found
>> .../lib64/libpam.so: undefined reference to `audit_open'
>> .../lib64/libpam.so: undefined reference to `audit_close'
>> ......
>>
>> The libaudit has been available and libpam build with it, after we run
>> another build with sstate-cache and w/o libaudit, the error occurs. The
>> solution in these cases is to use PACKAGECONFIG to lay out dependencies
>> on optional packages and make them explicit. We need to run an audit and
>> catch all of these issues.
>>
>> A probable ways maybe:
>>
>> 1) Run world build to generate the sysroots as much as possible;
>> 2) Run 'configure --help' on each package to spot all of the optional
>> dependencies, and convert them into PACKAGECONFIG specs/or any other
>> prompts.
>
> We should try to automate or codify this step so that the output is
> a list of oe-core package names.
>
>> 3) Run #1, #2 as an audit script periodically.
>>
>> Or could we run this check as part of a QA build step?
>
> Do you mean:
>    scripts/test-dependencies.sh

No, I mean some check steps that if PACKAGECONFIG should be used, not 
limit to the script.

Thanks
Wenzong

>
> // Randy
>
>>
>>
>> Thanks
>> Wenzong
>
>

Re: About PACKAGECONFIG audit

[Randy MacLeod]

On 13-08-20 01:36 AM, wenzong fan wrote:
> On 08/20/2013 04:46 AM, Randy MacLeod wrote:
>> Add Joe Slater who owns US17463: PACKAGECONFIG clean-up.
>>
>>
>> On 13-08-15 05:29 AM, wenzong fan wrote:
>>> Hi All,
>>>
>>> We have seen some problems like this:
>>>
>>> $ bitbake libcap
>>>
>>> ld: warning: libaudit.so.1, needed by .../lib64/libpam.so, not found
>>> .../lib64/libpam.so: undefined reference to `audit_open'
>>> .../lib64/libpam.so: undefined reference to `audit_close'
>>> ......
>>>
>>> The libaudit has been available and libpam build with it, after we run
>>> another build with sstate-cache and w/o libaudit, the error occurs. The
>>> solution in these cases is to use PACKAGECONFIG to lay out dependencies
>>> on optional packages and make them explicit. We need to run an audit and
>>> catch all of these issues.
>>>
>>> A probable ways maybe:
>>>
>>> 1) Run world build to generate the sysroots as much as possible;
>>> 2) Run 'configure --help' on each package to spot all of the optional
>>> dependencies, and convert them into PACKAGECONFIG specs/or any other
>>> prompts.
>>
>> We should try to automate or codify this step so that the output is
>> a list of oe-core package names.
>>
>>> 3) Run #1, #2 as an audit script periodically.
>>>
>>> Or could we run this check as part of a QA build step?
>>
>> Do you mean:
>>    scripts/test-dependencies.sh
>
> No, I mean some check steps that if PACKAGECONFIG should be used, not
> limit to the script.

thread replace by discussion on:

[Bug 5033] New: Way to audit all optional dependencies
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5033

// Randy

>
> Thanks
> Wenzong
>
>>
>> // Randy
>>
>>>
>>>
>>> Thanks
>>> Wenzong
>>
>>


-- 
# Randy MacLeod. SMTS, Linux, Wind River
Direct: 613.963.1350