Issues with .list files

Issues with .list files

[Michael Morrell]

If I understand the algorithm in package_do_shlibs correctly, each package creates a .list file which is a list of all the shared libraries provided by that package.  It also generates a list of all the libraries that it needs and looks for those in .list files created by other packages so it can create a runtime dependency against that package.

The way it looks for .list files is to simply scan all *.list files in the directory.  This has two problems that I can see.

First, it is inefficient.  It should only have to look through .list files creates by packages created by bb files which were DEPENDed on by this bb file.

Second, when a bitbake run is done with lots of parallelism, it is possible that it tries to open a nonexistent .list file (it was there when it did the os.listdir call, but that package later removed it before it recreates it), causing a crash here.  We are seeing such crashes reasonably often.

Before I try to fix this on my own, I wanted to get some opinions on the best way to handle this.

Thanks,

  Michael

Re: Issues with .list files

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05-02-10 04:24, Michael Morrell wrote:
> If I understand the algorithm in package_do_shlibs correctly, each package creates a .list file which is a list of all the shared libraries provided by that package.  It also generates a list of all the libraries that it needs and looks for those in .list files created by other packages so it can create a runtime dependency against that package.
> 
> The way it looks for .list files is to simply scan all *.list files in the directory.  This has two problems that I can see.
> 
> First, it is inefficient.  It should only have to look through .list files creates by packages created by bb files which were DEPENDed on by this bb file.

Ehm, no. If it did that you'd get missing shlibs in lots of cases. OE
won't error out, but you're stabbing users of the packages in the eye
since they get stuck with the mess.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFLbBEMMkyGM64RGpERAnVlAJ4sLpf6WokOnRUEL+ecgeygyVsndgCdHMIm
P6+zfodeyheiFvYv63SJ6IE=
=Y2X0
-----END PGP SIGNATURE-----

Re: Issues with .list files

[Philip Balister]

On 02/05/2010 07:37 AM, Koen Kooi wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 05-02-10 04:24, Michael Morrell wrote:
>> If I understand the algorithm in package_do_shlibs correctly, each package creates a .list file which is a list of all the shared libraries provided by that package.  It also generates a list of all the libraries that it needs and looks for those in .list files created by other packages so it can create a runtime dependency against that package.
>>
>> The way it looks for .list files is to simply scan all *.list files in the directory.  This has two problems that I can see.
>>
>> First, it is inefficient.  It should only have to look through .list files creates by packages created by bb files which were DEPENDed on by this bb file.
>
> Ehm, no. If it did that you'd get missing shlibs in lots of cases. OE
> won't error out, but you're stabbing users of the packages in the eye
> since they get stuck with the mess.

But it does seem like using a library created by something the recipe 
does not DEPEND on is a problem that should be corrected sooner, rather 
than later.

Philip

Re: Issues with .list files

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05-02-10 13:47, Philip Balister wrote:
> On 02/05/2010 07:37 AM, Koen Kooi wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> On 05-02-10 04:24, Michael Morrell wrote:
>>> If I understand the algorithm in package_do_shlibs correctly, each
>>> package creates a .list file which is a list of all the shared
>>> libraries provided by that package.  It also generates a list of all
>>> the libraries that it needs and looks for those in .list files
>>> created by other packages so it can create a runtime dependency
>>> against that package.
>>>
>>> The way it looks for .list files is to simply scan all *.list files
>>> in the directory.  This has two problems that I can see.
>>>
>>> First, it is inefficient.  It should only have to look through .list
>>> files creates by packages created by bb files which were DEPENDed on
>>> by this bb file.
>>
>> Ehm, no. If it did that you'd get missing shlibs in lots of cases. OE
>> won't error out, but you're stabbing users of the packages in the eye
>> since they get stuck with the mess.
> 
> But it does seem like using a library created by something the recipe
> does not DEPEND on is a problem that should be corrected sooner, rather
> than later.

That's certainly true, but the above approach is the complete wrong way
to go about it
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFLbCIvMkyGM64RGpERArNSAKC2XF79R9ioAzIgmOIngdPct8XQ/ACeLOwt
hfMnO8LI13yt9IfD9W8Xoak=
=JncI
-----END PGP SIGNATURE-----

Re: Issues with .list files

[Michael Smith]

Michael Morrell wrote:

> Second, when a bitbake run is done with lots of parallelism, it is
> possible that it tries to open a nonexistent .list file (it was there
> when it did the os.listdir call, but that package later removed it
> before it recreates it), causing a crash here.  We are seeing such
> crashes reasonably often.

Yeah, package_do_shlibs() in package.bbclass should probably write to a 
temp file and then link it into place. Right now it just removes and 
rewrites.

Mike

Re: Issues with .list files

[Phil Blundell]

On Thu, 2010-02-04 at 19:24 -0800, Michael Morrell wrote:
> If I understand the algorithm in package_do_shlibs correctly, each package creates a .list file which is a list of all the shared libraries provided by that package.  It also generates a list of all the libraries that it needs and looks for those in .list files created by other packages so it can create a runtime dependency against that package.
> 
> The way it looks for .list files is to simply scan all *.list files in the directory.  This has two problems that I can see.
> 
> First, it is inefficient.  It should only have to look through .list files creates by packages created by bb files which were DEPENDed on by this bb file.

That's an interesting point.  I think this probably is true, although it
would lead to some issues with packages that currently have incomplete
DEPENDS (or which rely on the historically-transitive behaviour of
DEPENDS for correct operation); there are probably quite a lot of these.
If you're going to make this change then I think you would also need to
change the current "couldn't find shlib provider for..." warning into a
fatal error.

Do you have any feel for how much time would actually be saved by
eliminating the unnecessary scan of other .list files?

> Second, when a bitbake run is done with lots of parallelism, it is possible that it tries to open a nonexistent .list file (it was there when it did the os.listdir call, but that package later removed it before it recreates it), causing a crash here.  We are seeing such crashes reasonably often.

This one is definitely just a bug (and, I think, unrelated to the first
issue).  Any time you create or modify a file in a live part of staging,
it needs to be done atomically and it sounds like package.bbclass is
currently failing to do that.  Feel free to make a patch, or I can look
at this instead if you want.

p.

Re: Issues with .list files

[Michael Morrell]

>>> Ehm, no. If it did that you'd get missing shlibs in lots of cases. OE
>>> won't error out, but you're stabbing users of the packages in the eye
>>> since they get stuck with the mess.
 
IMO, the missing shlib message should be fatal.  Cases where this message appears need to be fixed.

>> But it does seem like using a library created by something the recipe
>> does not DEPEND on is a problem that should be corrected sooner, rather
>> than later.

Agreed.

> That's certainly true, but the above approach is the complete wrong way
> to go about it

Then what do you suggest?  I'd like to fix the efficiency issue, but I'm more concerned about the parallelism failure.

   Michael

Re: Issues with .list files

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05-02-10 19:12, Michael Morrell wrote:
>>>> Ehm, no. If it did that you'd get missing shlibs in lots of cases. OE
>>>> won't error out, but you're stabbing users of the packages in the eye
>>>> since they get stuck with the mess.
>  
> IMO, the missing shlib message should be fatal.  

See, that's a lot better than "silently" failing and creating known
broken packages.

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFLbSA+MkyGM64RGpERAvAoAKCiJYOMRDoh4CIxVo4s/M8OapsUzQCeI9Ww
ijzHy7MZefNUU+czwUCbfrM=
=Zlbn
-----END PGP SIGNATURE-----