is sstate-cache really deterministic together with shlibs providers?

is sstate-cache really deterministic together with shlibs providers?

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1621 bytes --]

Imagine this process

foo.bb DEPENDS on libabc.bb to provide libabc.1.so

evil recipe bar.bb installs some binary crap in /opt/crap and because it's
picky about libabc version it bundles own compy of libabc.so.1 and
installs it to /opt/crap/lib/libabc.so.1

At the time of first build nobody notices libabc.1.so and foo, bar and
libabc are happily populated to sysroot.

foo.bb is rebuilt because of some unrelated change, but this times it
uses bar as shlibs provider for libabc.so.1

foo does not work in runtime, because it cannot find libabc.so.1 hidden
in /opt/crap/lib.

bar.bb is "fixed" by adding EXCLUDE_FROM_SHLIBS to prevent further
polluting of shlibs providers, but damage is already done.

foo doesn't have any dependency on bar (DEPENDS/RDEPENDS) because bar.bb
is just unrelated binary crap which just happens to bundle libabc.so.1
too (so foo checksum does not include bar checksum)

Now the tricky part:
1) fixing local build is easy
   bitbake -c cleansstate `grep ' bar' buildhistory/images/machine/eglibc/image-with-foo/depends.dot | sed 's/ -> .*//g' | xargs`

but sstate checksums are identical with bar or libabc in package
"Depends:" field, so how to cleanup SSTATE_MIRROR and sstate archives
already distributed to every local builder?

I could bump PR in libabc, but with PR bumps and PRINC going away I
really need to know how to solve such issues in future. Should we fix
do_package to filter shlibs providers to include only recipes which 
are in (R)DEPENDS/RRECOMMENDS/RSUGGESTS?

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: is sstate-cache really deterministic together with shlibs providers?

[Richard Purdie]

On Fri, 2013-06-07 at 21:20 +0200, Martin Jansa wrote:
> Imagine this process
> 
> foo.bb DEPENDS on libabc.bb to provide libabc.1.so
> 
> evil recipe bar.bb installs some binary crap in /opt/crap and because it's
> picky about libabc version it bundles own compy of libabc.so.1 and
> installs it to /opt/crap/lib/libabc.so.1
> 
> At the time of first build nobody notices libabc.1.so and foo, bar and
> libabc are happily populated to sysroot.
> 
> foo.bb is rebuilt because of some unrelated change, but this times it
> uses bar as shlibs provider for libabc.so.1
> 
> foo does not work in runtime, because it cannot find libabc.so.1 hidden
> in /opt/crap/lib.
> 
> bar.bb is "fixed" by adding EXCLUDE_FROM_SHLIBS to prevent further
> polluting of shlibs providers, but damage is already done.
> 
> foo doesn't have any dependency on bar (DEPENDS/RDEPENDS) because bar.bb
> is just unrelated binary crap which just happens to bundle libabc.so.1
> too (so foo checksum does not include bar checksum)
> 
> Now the tricky part:
> 1) fixing local build is easy
>    bitbake -c cleansstate `grep ' bar' buildhistory/images/machine/eglibc/image-with-foo/depends.dot | sed 's/ -> .*//g' | xargs`
> 
> but sstate checksums are identical with bar or libabc in package
> "Depends:" field, so how to cleanup SSTATE_MIRROR and sstate archives
> already distributed to every local builder?
> 
> I could bump PR in libabc, but with PR bumps and PRINC going away I
> really need to know how to solve such issues in future. Should we fix
> do_package to filter shlibs providers to include only recipes which 
> are in (R)DEPENDS/RRECOMMENDS/RSUGGESTS?

PR bumps can still happen for corner cases like this, the intent is just
to rely on the system for the 99.9% of cases which it does cover much
more accurately than a human.

We should perhaps limit shlibs to the default linker search paths?

Wouldn't there be a warning if two things provide the same shlibs or
does that assume the shlibs versioning sorts itself out?

Cheers,

Richard

Re: is sstate-cache really deterministic together with shlibs providers?

[Phil Blundell]

On Fri, 2013-06-07 at 21:20 +0200, Martin Jansa wrote:
> Imagine this process
> 
> foo.bb DEPENDS on libabc.bb to provide libabc.1.so
> 
> evil recipe bar.bb installs some binary crap in /opt/crap and because it's
> picky about libabc version it bundles own compy of libabc.so.1 and
> installs it to /opt/crap/lib/libabc.so.1
> 
> At the time of first build nobody notices libabc.1.so and foo, bar and
> libabc are happily populated to sysroot.
> 
> foo.bb is rebuilt because of some unrelated change, but this times it
> uses bar as shlibs provider for libabc.so.1

It sounds like the real bug here is that the shlibdeps code doesn't
issue a diagnostic when it encounters this kind of ambiguous situation.
It ought to be perfectly capable of figuring out that there are two
copies of libabc.so.1 at large which belong to different packages and,
absent additional guidance, it has no reliable way of knowing which one
of those is going to satisfy the DT_NEEDED in any random binary.

If shlibdeps stopped with an error in that situation, which arguably it
ought to, then the erroneous dependency would never get into sstate and
the scenario that you describe would not be able to happen.

p.

Re: is sstate-cache really deterministic together with shlibs providers?

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3269 bytes --]

On Fri, Jun 07, 2013 at 09:14:09PM +0100, Phil Blundell wrote:
> On Fri, 2013-06-07 at 21:20 +0200, Martin Jansa wrote:
> > Imagine this process
> > 
> > foo.bb DEPENDS on libabc.bb to provide libabc.1.so
> > 
> > evil recipe bar.bb installs some binary crap in /opt/crap and because it's
> > picky about libabc version it bundles own compy of libabc.so.1 and
> > installs it to /opt/crap/lib/libabc.so.1
> > 
> > At the time of first build nobody notices libabc.1.so and foo, bar and
> > libabc are happily populated to sysroot.
> > 
> > foo.bb is rebuilt because of some unrelated change, but this times it
> > uses bar as shlibs provider for libabc.so.1
> 
> It sounds like the real bug here is that the shlibdeps code doesn't
> issue a diagnostic when it encounters this kind of ambiguous situation.
> It ought to be perfectly capable of figuring out that there are two
> copies of libabc.so.1 at large which belong to different packages and,
> absent additional guidance, it has no reliable way of knowing which one
> of those is going to satisfy the DT_NEEDED in any random binary.
> 
> If shlibdeps stopped with an error in that situation, which arguably it
> ought to, then the erroneous dependency would never get into sstate and
> the scenario that you describe would not be able to happen.

I like both suggestions from you and from Richard, there is another
slightly related issue with
https://bugzilla.yoctoproject.org/show_bug.cgi?id=4102

It's probably possible to create test case where I create sstate package
with libabc.so.1 provided by MACHINE_ARCH foo and then try to reuse this
sstate but with libabc.so.1 provided by TUNE_PKGARCH foo (e.g. with
different version) shlibs code will still find it provider by
MACHINE_ARCH foo and there isn't way to clean foo from incremental build
(only manually).

I'll create bug report and try to sum it there:
1) RP1: We should perhaps limit shlibs to the default linker search paths?
2) RP2+PB: Warning/Error if two things provide the same shlibs
   - probably better to error out when bar.bb is being built and tries
     to provide the same libabc libabc.bb, it can be in different order
     as bar does not depends on libabc, so next time it can show error
     when building libabc, but IMHO better then showing error when foo 
     finds out 2 possible libabc providers
3) Restrict possible providers to stuff included in dependencies - 
   I don't know how difficult it will be to implement this (to see all
   dependencies including transitive), but IMHO this one is most 
   important and can also solve #4102, most tricky part would be to 
   prevent shlib providers silently disappearing from RDEPENDS - 
   when something doesn't declare dependency explicitly, but it was
   always there when do_package was executed - now missing providers
   are silently ignored IIRC. Buildhistory can probably help a lot to
   find missing explicit deps for shlibs providers when we restrict it.

   This will also ensure that when shlibs provider is changed, the
   package which is automagically RDEPENDing on it gets rebuilt too.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=4628
-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: is sstate-cache really deterministic together with shlibs providers?

[Phil Blundell]

On Fri, 2013-06-07 at 22:43 +0200, Martin Jansa wrote:
> 2) RP2+PB: Warning/Error if two things provide the same shlibs
>    - probably better to error out when bar.bb is being built and tries
>      to provide the same libabc libabc.bb, it can be in different order
>      as bar does not depends on libabc, so next time it can show error
>      when building libabc, but IMHO better then showing error when foo 
>      finds out 2 possible libabc providers

Raising an error if libabc is already staged when bar is built would be
good as well, but I don't think this is sufficient in the general case.
Consider:

1. bitbake libabc

-> no problem, bar isn't built yet, libabc goes into sstate-cache

2. rm -rf $TMPDIR; bitbake bar

-> also no problem, libabc isn't in the sysroot anymore, bar goes into
sysroot (and sstate-cache)

3. bitbake foo

-> libabc comes back from sstate but bar is still there, disaster
strikes.

Obviously that particular example is a bit contrived, but you can
imagine that if your sstate-cache (or part of it) is shared among
multiple users then this sort of situation could arise fairly easily.

p.