Weird build dependency issue causing missing symbols

Weird build dependency issue causing missing symbols

[Bills, Jason M]

Hi All,

We are hitting a weird build dependency issue with Yocto and 
phosphor-dbus-interfaces and are looking for any help or insight anyone 
may have on how to fix it.  We have not been able to pinpoint exactly 
when the issue started, but we believe it has come up since the dunfell 
update.

The symptom of this issue is we see an undefined symbol error at runtime:
[  101.733677] Jul 02 10:37:48 intel-obmc phosphor-ledcontroller[461]: 
phosph
or-ledcontroller: symbol lookup error: phosphor-ledcontroller: undefined 
symbol: 
_ZN9sdbusplus3xyz15openbmc_project3Led6server8Physical17setPropertyByNameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt7variantIJtNS4_7PaletteEhNS4_6ActionEEEb

Once we hit this error, it persists across rebuilds until we delete the 
Yocto build directory (likely something in the cache) and start a fresh 
build.

We have narrowed this down to being caused by two separate issues:
1. When phosphor-dbus-interfaces is rebuilt it will sometimes change the 
order of the PropertiesVariant in server.hpp.
2. When the order of PropertiesVariant changes on a rebuild, the recipes 
that already have an old copy of server.hpp are not triggered to rebuild 
and are left with the old copy of server.hpp.

I have a system that is in this state and have found that if I taint 
phosphor-dbus-interfaces by running "bitbake -C fetch 
phosphor-dbus-interfaces", I see many components rebuild and the symbol 
issue goes away.  If I then remove the taint by running "bitbake -c 
clean phosphor-dbus-interfaces" only phosphor-dbus-interfaces and any 
components in my devtool status list rebuild and the symbol issue comes 
back.

We ran an experiment where we compared the contents of 
".../Led/Physical/server.hpp" between components by running this command 
(where the base file came from an existing build):
for fname in $(find . -iname server.hpp|grep -i "led/physical"); do echo 
"$fname"; diff 
"./tmp/work/arm1176jzs-openbmc-linux-gnueabi/phosphor-network/1.0+gitAUTOINC+d0679f9bb4-r1/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp" 
"$fname"; done

With the tainted phosphor-dbus-interfaces, there is no diff in any of 
the server.hpp files.

After cleaning the taint and rebuilding, I get the following results:
./tmp/work/arm1176jzs-openbmc-linux-gnueabi/phosphor-sel-logger/0.1+gitAUTOINC+761bf202ba-r0/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp
./tmp/work/arm1176jzs-openbmc-linux-gnueabi/intel-ipmi-oem/0.1+gitAUTOINC+e4f710d7d9-r0/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp
./tmp/work/arm1176jzs-openbmc-linux-gnueabi/phosphor-dbus-interfaces/1.0+gitAUTOINC+26ff1c8446-r1/sysroot-destdir/usr/include/xyz/openbmc_project/Led/Physical/server.hpp
66,67c66
<                 Action,
<                 uint16_t,
---
 >                 uint8_t,
69c68,69
<                 uint8_t>;
---
 >                 Action,
 >                 uint16_t>;
./tmp/work/arm1176jzs-openbmc-linux-gnueabi/phosphor-ipmi-ipmb/0.1+gitAUTOINC+a86059348f-r0/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp
./tmp/work/arm1176jzs-openbmc-linux-gnueabi/phosphor-host-postd/0.1+gitAUTOINC+bf002b46d5-r1/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp
./tmp/work/arm1176jzs-openbmc-linux-gnueabi/phosphor-network/1.0+gitAUTOINC+d0679f9bb4-r1/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp
./tmp/work/arm1176jzs-openbmc-linux-gnueabi/x86-power-control/1.0+gitAUTOINC+b0c613aa88-r0/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp
66,67c66
<                 Action,
<                 uint16_t,
---
 >                 uint8_t,
69c68,69
<                 uint8_t>;
---
 >                 Action,
 >                 uint16_t>;
./tmp/work/arm1176jzs-openbmc-linux-gnueabi/phosphor-post-code-manager/1.0+gitAUTOINC+9d91a39a3a-r0/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp
./tmp/work/arm1176jzs-openbmc-linux-gnueabi/obmc-ikvm/1.0+gitAUTOINC+861337e8ec-r0/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp
./tmp/work/arm1176jzs-openbmc-linux-gnueabi/service-config-manager/0.1+gitAUTOINC+83241c09ec-r0/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp
./tmp/work/arm1176jzs-openbmc-linux-gnueabi/phosphor-ipmi-kcs/1.0+gitAUTOINC+d8594e9a62-r1/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp
./tmp/sysroots-components/arm1176jzs/phosphor-dbus-interfaces/usr/include/xyz/openbmc_project/Led/Physical/server.hpp
66,67c66
<                 Action,
<                 uint16_t,
---
 >                 uint8_t,
69c68,69
<                 uint8_t>;
---
 >                 Action,
 >                 uint16_t>;

The order of the variant changed in server.hpp in 
phosphor-dbus-interfaces.  I had x86-power-control in my devtool status 
list, so it rebuilt and got the new copy of server.hpp, but everything 
else still had the old copy.

Does anyone have any ideas on what could be happening or if we're 
missing something to properly trigger the rebuilds?

Thanks for your help!
-Jason

Re: Weird build dependency issue causing missing symbols

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 1083 bytes --]

On Thu, Jul 02, 2020 at 12:58:43PM -0700, Bills, Jason M wrote:
> We have narrowed this down to being caused by two separate issues:
> 1. When phosphor-dbus-interfaces is rebuilt it will sometimes change the 
> order of the PropertiesVariant in server.hpp.

This sounds like a bug in sdbus++.  We should be sorting the variant
parameters or issuing them in array order.  I'll look into it.

> 2. When the order of PropertiesVariant changes on a rebuild, the recipes 
> that already have an old copy of server.hpp are not triggered to rebuild 
> and are left with the old copy of server.hpp.

This isn't surprising if what is triggering the rebuild is not a Yocto
variable change (or git revision).  Yocto doesn't cache the contents of
the packages, but caches the variables that went into a build step.  A
hash of the variables are used to look up the potential 'sstate-cache'
files so that it can skip build steps.

If you think a variable or a git-revision should have changed with what
you were doing, then maybe it is something else.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Weird build dependency issue causing missing symbols

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 613 bytes --]

On Thu, Jul 02, 2020 at 04:33:28PM -0500, Patrick Williams wrote:
> On Thu, Jul 02, 2020 at 12:58:43PM -0700, Bills, Jason M wrote:
> > We have narrowed this down to being caused by two separate issues:
> > 1. When phosphor-dbus-interfaces is rebuilt it will sometimes change the 
> > order of the PropertiesVariant in server.hpp.
> 
> This sounds like a bug in sdbus++.  We should be sorting the variant
> parameters or issuing them in array order.  I'll look into it.

sdbus++: server: fix PropertyVariant order
https://gerrit.openbmc-project.xyz/c/openbmc/sdbusplus/+/34344

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Weird build dependency issue causing missing symbols

[Bills, Jason M]

On 7/2/2020 2:42 PM, Patrick Williams wrote:
> On Thu, Jul 02, 2020 at 04:33:28PM -0500, Patrick Williams wrote:
>> On Thu, Jul 02, 2020 at 12:58:43PM -0700, Bills, Jason M wrote:
>>> We have narrowed this down to being caused by two separate issues:
>>> 1. When phosphor-dbus-interfaces is rebuilt it will sometimes change the
>>> order of the PropertiesVariant in server.hpp.
>>
>> This sounds like a bug in sdbus++.  We should be sorting the variant
>> parameters or issuing them in array order.  I'll look into it.
> 
> sdbus++: server: fix PropertyVariant order
> https://gerrit.openbmc-project.xyz/c/openbmc/sdbusplus/+/34344
> 
Thanks for the quick response!  I tested this change with tainting and 
cleaning phosphor-dbus-interfaces.  It still shows this diff after the 
clean build, but I don't know what is different and it doesn't trigger 
the symbol error:
./tmp/sysroots-components/arm1176jzs/phosphor-dbus-interfaces/usr/include/xyz/openbmc_project/Led/Physical/server.hpp
67d66
<                 uint8_t,
68a68
 >                 uint8_t,

This looks good!  Thanks again!

Re: Weird build dependency issue causing missing symbols

[Bills, Jason M]

On 7/2/2020 2:33 PM, Patrick Williams wrote:
> On Thu, Jul 02, 2020 at 12:58:43PM -0700, Bills, Jason M wrote:
>> We have narrowed this down to being caused by two separate issues:
>> 1. When phosphor-dbus-interfaces is rebuilt it will sometimes change the
>> order of the PropertiesVariant in server.hpp.
> 
> This sounds like a bug in sdbus++.  We should be sorting the variant
> parameters or issuing them in array order.  I'll look into it.
> 
>> 2. When the order of PropertiesVariant changes on a rebuild, the recipes
>> that already have an old copy of server.hpp are not triggered to rebuild
>> and are left with the old copy of server.hpp.
> 
> This isn't surprising if what is triggering the rebuild is not a Yocto
> variable change (or git revision).  Yocto doesn't cache the contents of
> the packages, but caches the variables that went into a build step.  A
> hash of the variables are used to look up the potential 'sstate-cache'
> files so that it can skip build steps.
> 
> If you think a variable or a git-revision should have changed with what
> you were doing, then maybe it is something else.
> 
It seems like a header file change should trigger a rebuild, though?  If 
I manually edited something like a library header file, I'd expect 
everything that includes that library to be rebuilt with the new header 
change.

I tried to devtool modify boost to check the behavior, but that causes 
boost to rebuild every time and correctly triggers the dependent builds. 
  Maybe the case above of modifying a header file is invalid?

RE: Weird build dependency issue causing missing symbols

[Ren, Zhikui]

Maybe the problem is that this header server.hpp is generated not  a source. 
Artifact created from the same source *should* be the same (except timestamp)
If the source did not update, just a forced rebuild to create new binaries,  I can see Yocto choose not to rebuild things depend on the package.
In the case of boost, since it is devtool modified and the header is a source and not build artifact, it make sense to trigger all the rebuild. 


-----Original Message-----
From: openbmc <openbmc-bounces+zhikui.ren=intel.com@lists.ozlabs.org> On Behalf Of Bills, Jason M
Sent: Thursday, July 2, 2020 5:00 PM
To: openbmc@lists.ozlabs.org
Subject: Re: Weird build dependency issue causing missing symbols



On 7/2/2020 2:33 PM, Patrick Williams wrote:
> On Thu, Jul 02, 2020 at 12:58:43PM -0700, Bills, Jason M wrote:
>> We have narrowed this down to being caused by two separate issues:
>> 1. When phosphor-dbus-interfaces is rebuilt it will sometimes change 
>> the order of the PropertiesVariant in server.hpp.
> 
> This sounds like a bug in sdbus++.  We should be sorting the variant 
> parameters or issuing them in array order.  I'll look into it.
> 
>> 2. When the order of PropertiesVariant changes on a rebuild, the 
>> recipes that already have an old copy of server.hpp are not triggered 
>> to rebuild and are left with the old copy of server.hpp.
> 
> This isn't surprising if what is triggering the rebuild is not a Yocto 
> variable change (or git revision).  Yocto doesn't cache the contents 
> of the packages, but caches the variables that went into a build step.  
> A hash of the variables are used to look up the potential 'sstate-cache'
> files so that it can skip build steps.
> 
> If you think a variable or a git-revision should have changed with 
> what you were doing, then maybe it is something else.
> 
It seems like a header file change should trigger a rebuild, though?  If I manually edited something like a library header file, I'd expect everything that includes that library to be rebuilt with the new header change.

I tried to devtool modify boost to check the behavior, but that causes boost to rebuild every time and correctly triggers the dependent builds. 
  Maybe the case above of modifying a header file is invalid?

Re: Weird build dependency issue causing missing symbols

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 2903 bytes --]

On Fri, Jul 03, 2020 at 12:18:34AM +0000, Ren, Zhikui wrote:
> Maybe the problem is that this header server.hpp is generated not  a source. 
> Artifact created from the same source *should* be the same (except timestamp)
> If the source did not update, just a forced rebuild to create new binaries,  I can see Yocto choose not to rebuild things depend on the package.
> In the case of boost, since it is devtool modified and the header is a source and not build artifact, it make sense to trigger all the rebuild. 
> 
> 
> -----Original Message-----
> From: openbmc <openbmc-bounces+zhikui.ren=intel.com@lists.ozlabs.org> On Behalf Of Bills, Jason M
> Sent: Thursday, July 2, 2020 5:00 PM
> To: openbmc@lists.ozlabs.org
> Subject: Re: Weird build dependency issue causing missing symbols
> 
> 
> 
> On 7/2/2020 2:33 PM, Patrick Williams wrote:
> > On Thu, Jul 02, 2020 at 12:58:43PM -0700, Bills, Jason M wrote:
> >> We have narrowed this down to being caused by two separate issues:
> >> 1. When phosphor-dbus-interfaces is rebuilt it will sometimes change 
> >> the order of the PropertiesVariant in server.hpp.
> > 
> > This sounds like a bug in sdbus++.  We should be sorting the variant 
> > parameters or issuing them in array order.  I'll look into it.
> > 
> >> 2. When the order of PropertiesVariant changes on a rebuild, the 
> >> recipes that already have an old copy of server.hpp are not triggered 
> >> to rebuild and are left with the old copy of server.hpp.
> > 
> > This isn't surprising if what is triggering the rebuild is not a Yocto 
> > variable change (or git revision).  Yocto doesn't cache the contents 
> > of the packages, but caches the variables that went into a build step.  
> > A hash of the variables are used to look up the potential 'sstate-cache'
> > files so that it can skip build steps.
> > 
> > If you think a variable or a git-revision should have changed with 
> > what you were doing, then maybe it is something else.
> > 
> It seems like a header file change should trigger a rebuild, though?  If I manually edited something like a library header file, I'd expect everything that includes that library to be rebuilt with the new header change.
> 
> I tried to devtool modify boost to check the behavior, but that causes boost to rebuild every time and correctly triggers the dependent builds. 
>   Maybe the case above of modifying a header file is invalid?

It doesn't matter what the content is: header, library, executable, data
file.  Yocto does not use contents in the decision of "does this need to
be rebuilt".  It only uses variables from recipes.  If the variables do
not change, the package is not rebuilt (unless explicitly tainted).

See https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#dev-invalidating-shared-state-to-force-a-task-to-run for example.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Weird build dependency issue causing missing symbols

[Bills, Jason M]

On 7/2/2020 10:20 PM, Patrick Williams wrote:
> On Fri, Jul 03, 2020 at 12:18:34AM +0000, Ren, Zhikui wrote:
>> Maybe the problem is that this header server.hpp is generated not  a source.
>> Artifact created from the same source *should* be the same (except timestamp)
>> If the source did not update, just a forced rebuild to create new binaries,  I can see Yocto choose not to rebuild things depend on the package.
>> In the case of boost, since it is devtool modified and the header is a source and not build artifact, it make sense to trigger all the rebuild.
>>
>>
>> -----Original Message-----
>> From: openbmc <openbmc-bounces+zhikui.ren=intel.com@lists.ozlabs.org> On Behalf Of Bills, Jason M
>> Sent: Thursday, July 2, 2020 5:00 PM
>> To: openbmc@lists.ozlabs.org
>> Subject: Re: Weird build dependency issue causing missing symbols
>>
>>
>>
>> On 7/2/2020 2:33 PM, Patrick Williams wrote:
>>> On Thu, Jul 02, 2020 at 12:58:43PM -0700, Bills, Jason M wrote:
>>>> We have narrowed this down to being caused by two separate issues:
>>>> 1. When phosphor-dbus-interfaces is rebuilt it will sometimes change
>>>> the order of the PropertiesVariant in server.hpp.
>>>
>>> This sounds like a bug in sdbus++.  We should be sorting the variant
>>> parameters or issuing them in array order.  I'll look into it.
>>>
>>>> 2. When the order of PropertiesVariant changes on a rebuild, the
>>>> recipes that already have an old copy of server.hpp are not triggered
>>>> to rebuild and are left with the old copy of server.hpp.
>>>
>>> This isn't surprising if what is triggering the rebuild is not a Yocto
>>> variable change (or git revision).  Yocto doesn't cache the contents
>>> of the packages, but caches the variables that went into a build step.
>>> A hash of the variables are used to look up the potential 'sstate-cache'
>>> files so that it can skip build steps.
>>>
>>> If you think a variable or a git-revision should have changed with
>>> what you were doing, then maybe it is something else.
>>>
>> It seems like a header file change should trigger a rebuild, though?  If I manually edited something like a library header file, I'd expect everything that includes that library to be rebuilt with the new header change.
>>
>> I tried to devtool modify boost to check the behavior, but that causes boost to rebuild every time and correctly triggers the dependent builds.
>>    Maybe the case above of modifying a header file is invalid?
> 
> It doesn't matter what the content is: header, library, executable, data
> file.  Yocto does not use contents in the decision of "does this need to
> be rebuilt".  It only uses variables from recipes.  If the variables do
> not change, the package is not rebuilt (unless explicitly tainted).
> 
> See https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#dev-invalidating-shared-state-to-force-a-task-to-run for example.
> 
Thanks, Patrick! I think I get it.  I was stuck on the behavior that I 
see when I'm working with repo using 'devtool modify'.  I guess 'devtool 
modify' must have some additional magic to force rebuilds on file 
changes that doesn't apply to a standard build?