Uninative and sstate

Uninative and sstate

[Joshua Watt]

We've run into a strange issued that turned out to be cased by missing
iconv conversion libraries in our uninative tarball, but is sparked a
larger question that I was hoping someone here could answer.

The problem was particularly baffling. We were attempting to use
Doxygen in a recipe which required some missing iconv conversion files.
If we built doxygen locally, it worked just fine, but if we choose to
allow bitbake to pull doxygen down from a sstate it would fail. Even
more baffling, if we manually extracted the sstate do_populate_sysroot
tarball for doxygen and used the executable there, everything worked.
We surmised that bitbake was modifying the the doxygen executable after
it extracted from the sstate tarball, which was confirmed by comparing
checksums. After some digging, the cause of the change was tracked down
to uninative_changeinterp() running after the sstate tarball was
extracted, changing the program interpreter to the one in the uninative
tarball.

My question is: Why is this coercion of the program interpreter *only*
done when the sysroot is populate from sstate? For consistency, it
would seem appropriate to also coerce the interpreter when doing
do_populate_sysroot from an actual build. If this had been done, we
would have noticed the missing library when building locally, and the
sstate cache would not have caused "random" build breakage.

Does anyone have any insight as to why this isn't done?

Thanks,
Joshua Watt

Re: Uninative and sstate

[Richard Purdie]

On Thu, 2018-01-04 at 14:15 -0600, Joshua Watt wrote:
> We've run into a strange issued that turned out to be cased by
> missing
> iconv conversion libraries in our uninative tarball, but is sparked a
> larger question that I was hoping someone here could answer.
> 
> The problem was particularly baffling. We were attempting to use
> Doxygen in a recipe which required some missing iconv conversion
> files.
> If we built doxygen locally, it worked just fine, but if we choose to
> allow bitbake to pull doxygen down from a sstate it would fail. Even
> more baffling, if we manually extracted the sstate
> do_populate_sysroot
> tarball for doxygen and used the executable there, everything worked.
> We surmised that bitbake was modifying the the doxygen executable
> after
> it extracted from the sstate tarball, which was confirmed by
> comparing
> checksums. After some digging, the cause of the change was tracked
> down
> to uninative_changeinterp() running after the sstate tarball was
> extracted, changing the program interpreter to the one in the
> uninative
> tarball.
> 
> My question is: Why is this coercion of the program interpreter
> *only*
> done when the sysroot is populate from sstate? For consistency, it
> would seem appropriate to also coerce the interpreter when doing
> do_populate_sysroot from an actual build. If this had been done, we
> would have noticed the missing library when building locally, and the
> sstate cache would not have caused "random" build breakage.
> 
> Does anyone have any insight as to why this isn't done?

Which release are you using?

There used to be an optimisation we made where we only did it when we
needed to (using sstate). The move to RSS meant that we now always have
to do it. I'd therefore guess you're using morty or older? On more
recent versions you'll see it always relocating. There are pros and
cons both ways.

Cheers,

Richard

Re: Uninative and sstate

[Joshua Watt]

On Thu, 2018-01-04 at 22:00 +0000, Richard Purdie wrote:
> On Thu, 2018-01-04 at 14:15 -0600, Joshua Watt wrote:
> > We've run into a strange issued that turned out to be cased by
> > missing
> > iconv conversion libraries in our uninative tarball, but is sparked
> > a
> > larger question that I was hoping someone here could answer.
> > 
> > The problem was particularly baffling. We were attempting to use
> > Doxygen in a recipe which required some missing iconv conversion
> > files.
> > If we built doxygen locally, it worked just fine, but if we choose
> > to
> > allow bitbake to pull doxygen down from a sstate it would fail.
> > Even
> > more baffling, if we manually extracted the sstate
> > do_populate_sysroot
> > tarball for doxygen and used the executable there, everything
> > worked.
> > We surmised that bitbake was modifying the the doxygen executable
> > after
> > it extracted from the sstate tarball, which was confirmed by
> > comparing
> > checksums. After some digging, the cause of the change was tracked
> > down
> > to uninative_changeinterp() running after the sstate tarball was
> > extracted, changing the program interpreter to the one in the
> > uninative
> > tarball.
> > 
> > My question is: Why is this coercion of the program interpreter
> > *only*
> > done when the sysroot is populate from sstate? For consistency, it
> > would seem appropriate to also coerce the interpreter when doing
> > do_populate_sysroot from an actual build. If this had been done, we
> > would have noticed the missing library when building locally, and
> > the
> > sstate cache would not have caused "random" build breakage.
> > 
> > Does anyone have any insight as to why this isn't done?
> 
> Which release are you using?
> 
> There used to be an optimisation we made where we only did it when we
> needed to (using sstate). The move to RSS meant that we now always
> have
> to do it. I'd therefore guess you're using morty or older? On more
> recent versions you'll see it always relocating. There are pros and
> cons both ways.

Yes we are using morty. I suppose that at least explains the behavior.

Thanks for clarifying,
Joshua Watt

> 
> Cheers,
> 
> Richard

Re: Uninative and sstate

[Richard Purdie]

On Thu, 2018-01-04 at 16:11 -0600, Joshua Watt wrote:
> On Thu, 2018-01-04 at 22:00 +0000, Richard Purdie wrote:
> > There used to be an optimisation we made where we only did it when
> > we
> > needed to (using sstate). The move to RSS meant that we now always
> > have
> > to do it. I'd therefore guess you're using morty or older? On more
> > recent versions you'll see it always relocating. There are pros and
> > cons both ways.
> Yes we are using morty. I suppose that at least explains the
> behavior.

You'll find:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?h=morty&id=590ef99df7
4581e6b7595e545c12594012653de8

interesting (its on the stable branch now).

That was basically retrofitting some of this behaviour to morty to fix
bugs...

Cheers,

Richard