Hashserve OEEquivHash unihash issue

Hashserve OEEquivHash unihash issue

[Harro Verton]

[-- Attachment #1: Type: text/plain, Size: 4002 bytes --]

Hello Community,

First post, and a relative OE newbie, so please bear with me.

I've inherited a build environment of an open source project ( https://github.com/OpenPLi/openpli-oe-core ). The project builds images for some 120 different devices, with MIPSEL or ARM SoCs, covering 5 different ARCHs.

All are build from a single repository, containing the submodules for Bitbake, Openembedded-Core and meta-openembedded, and all required BSP's for these devices. They all share sstate, a PR server, and a Hashserve server.

In order to avoid packages build for the ARCH feed rebuilding for all subsequent MACHINEs using that same ARCH, my predecessor has introduced HashServe, using OEEquivHash.

This works fine in our previous release (which is based on Hardknott), and worked fine in our current release (which is based on Scarthgap), until a few months ago.
We want to start with the next release, based on Wrynose, but we can't until we can make a final Scarthgap release, which we can't because of this issue.

I've run a  test, building two different MACHINEs with the same ARCH (identical hardware, different brands, different drivers, so separate BSPs), with an empty build directory (so no sstate cache, new PR database, no hash data). For the second machine, 230 tasks are recorded as having a changed unihash, which triggers a recipe rebuild for the second MACHINE.

I've enabled hashserve debugging, and these recipes log something like:
log.do_package_write_ipk.212153:DEBUG: Reported task efe6373f8d36e03f92f0198b5c2891fba8b459bd1e16ee9043f3e3d85a47c65e as unihash efe6373f8d36e03f92f0198b5c2891fba8b459bd1e16ee9043f3e3d85a47c65e to unix:///data/openpli/oe/scarthgap/build/hashserve.sock log.do_package_write_ipk.2223688:DEBUG: Task 8da90b016d0a41cac6ce46c20d8ee64d5e7af35687de43c306544508050227e8 unihash changed 8da90b016d0a41cac6ce46c20d8ee64d5e7af35687de43c306544508050227e8 -> efe6373f8d36e03f92f0198b5c2891fba8b459bd1e16ee9043f3e3d85a47c65e by server unix:///data/openpli/oe/scarthgap/build/hashserve.sock
so the second build records a changed unihash, which triggers a rebuild of the package, and leads to an r0.1 version of the ipk.

All of them but 8 report a unihash change for do_package_qa , those 8 also for do_package_write_ipk.

The downside of this is that after device A is flashed with the result of the first build, and you do an "opkg list-upgradable", you're presented with 182 packages from the ARCH feed shared between the two devices that are supposedly changed (but the only change is r0.0 to r0.1 due to the rebuild). Apart from the fact that most of our users are not tech savvy, several devices are limited on flash memory space, so having to update 180+ packages every time we have a build run is quite a bit of an issue.

I can't find a reason why this suddenly happens, some of these packages are quite basic and don't contain binary code, packages like python3-requests for example. And I have no idea how to debug this further.

For this test build, Bitbake is at yocto-5.0.19, openembedded-core and meta-openembedded are on the scarthgap branch, pulled yesterday.

I've looked at the commit history in our repo, the only time this trio was updated was on July 30th. If I revert that, bitbake goes back to yocto-5.0.15 and both OE submodules to a commit in december 2025, which is when we started with our Scarthgap branch.
When I repeat the test build after the revert, only two recipes for the ARCH feed are rebuild for the second machine, ffmpeg and gstreamer, both are recipes that we override in our own meta layer ( version upgrades to those in openembedded ).

So somewhere along the line, something got broken (or how it worked changed, and we missed that).

re there fixes in later versions for this issue, that are not backported to Scarthgap? How can I debug what exactly causes the unihash to change? Or better, how do I fix this?

TIA,

Harro Verton, member of the PLi Association
https://openpli.org

[-- Attachment #2: Type: text/html, Size: 6250 bytes --]

Re: [yocto] Hashserve OEEquivHash unihash issue

[Harro Verton]

[-- Attachment #1: Type: text/plain, Size: 450 bytes --]

Note that what I find very strange, is that the first MACHINE comes up with efe6373f8d36e03f92f0198b5c2891fba8b459bd1e16ee9043f3e3d85a47c65e as unihash.

But for the second MACHINE it says the hash was 8da90b016d0a41cac6ce46c20d8ee64d5e7af35687de43c306544508050227e8 (why? how?) and is changed to efe6373f8d36e03f92f0198b5c2891fba8b459bd1e16ee9043f3e3d85a47c65e (which it should have been in the first place).

Does this suggest a lookup issue?

[-- Attachment #2: Type: text/html, Size: 517 bytes --]

Re: [yocto] Hashserve OEEquivHash unihash issue

[Joshua Watt]

[-- Attachment #1: Type: text/plain, Size: 1812 bytes --]

On Thu, Sep 10, 2026 at 1:50 PM Harro Verton via lists.yoctoproject.org
<wanwizard=openpli.org@lists.yoctoproject.org> wrote:

> Note that what I find very strange, is that the first MACHINE comes up
> with efe6373f8d36e03f92f0198b5c2891fba8b459bd1e16ee9043f3e3d85a47c65e as
> unihash.
>
> But for the second MACHINE it says the hash was
> 8da90b016d0a41cac6ce46c20d8ee64d5e7af35687de43c306544508050227e8 (why?
> how?) and is changed to
> efe6373f8d36e03f92f0198b5c2891fba8b459bd1e16ee9043f3e3d85a47c65e (which it
> should have been in the first place).
>
> Does this suggest a lookup issue?
>

Probably not. The unihash is (basically) the taskhash first reported to the
server with a given output hash (a hash of the output the task produced).
When a second taskhash is reported to the server with the same output, it
gets mapped to the same unihash (which is the message you see above).

What this usually means is that you have two recipes that have different
task hashes, but produce the same output so the hashserver is mapping them
together as expected. You might want to figure out why the taskhashes are
different; if you have the .sigdata files for those hashes (look for e.g.
"sigdata.8da90b016d0a41cac6ce46c20d8ee64d5e7af35687de43c306544508050227e8"
in the stamps directory) you can diff them with bitbake-diffsigs and see
what is actually different.


>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#66716):
> https://lists.yoctoproject.org/g/yocto/message/66716
> Mute This Topic: https://lists.yoctoproject.org/mt/121187356/3616693
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [
> JPEWhacker@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 2848 bytes --]

Re: [yocto] Hashserve OEEquivHash unihash issue

[Harro Verton]

[-- Attachment #1: Type: text/plain, Size: 1675 bytes --]

Thanks for the quick reply, much appreciated !

> 
> What this usually means is that you have two recipes that have different
> task hashes, but produce the same output so the hashserver is mapping them
> together as expected. You might want to figure out why the taskhashes are
> different; if you have the .sigdata files for those hashes (look for e.g.
> "sigdata.8da90b016d0a41cac6ce46c20d8ee64d5e7af35687de43c306544508050227e8"
> in the stamps directory) you can diff them with bitbake-diffsigs and see
> what is actually different.

This file doesn't exist in the stamps directory, and neither does an entry for the hash it was changed to.

The (relevant?) build config contains:

INHERIT += "rm_work"
PRSERV_HOST = "localhost:0"
BB_SIGNATURE_HANDLER = "OEEquivHash"
BB_HASHSERVE = "auto"

INSANE_SKIP += "version-going-backwards"
ERROR_QA:remove = "version-going-backwards"
BUILDHISTORY_CHECKVERBACKWARDS = "0"
INHERIT:remove = "buildhistory"

If I flash the image built on both devices, I see that on the first device, ffmpeg - 6.1.4-r0.0 is installed, and on the second, ffmpeg - 6.1.4-r0.1.  And as a result, the first machine shows in the "opkg list-upgradable" output "ffmpeg - 6.1.4-r0.0 - 6.1.4-r0.1". In deploy/ipks/cortexa15hf-neon-vfpv4 (same for both devices), I only have the ipk for 6.1.4-r0.1.

If I build both MACHINE's a second time, new rootfs files are generated, but to my surprise, for the first device the rootfs still contains ffmpeg - 6.1.4-r0.0, even though that does not exist anywhere in the build tree.

I have no clue where to start looking, but something changed along the way in Scarthgap that "broke" this.

[-- Attachment #2: Type: text/html, Size: 1865 bytes --]

Re: [yocto] Hashserve OEEquivHash unihash issue

[Harro Verton]

[-- Attachment #1: Type: text/plain, Size: 333 bytes --]

The PR database contains two entries for ffmpeg:

sqlite> SELECT * from PRMAIN_nohist WHERE version LIKE 'ffmpeg%';
ffmpeg-6.1.4-r0|cortexa15hf-neon-vfpv4|fc22c7d30187c90305b3aa0bc761e3e4203655822f618753019b8779a8c0ca68|0
ffmpeg-6.1.4-r0|cortexa15hf-neon-vfpv4|4f51d61f7509b9bc26440b0e0639af96fe7893070e516e63c523bd8b49e3003d|1

[-- Attachment #2: Type: text/html, Size: 383 bytes --]

Re: Hashserve OEEquivHash unihash issue

[Harro Verton]

[-- Attachment #1: Type: text/plain, Size: 304 bytes --]

Took a lot of digging but the issue was found.

Turned out someone added a bbappend for openssl.bb, and included a PACKAGE_ARCH = $MACHINE_ARCH.

The caused openssl to be rebuild for every MACHINE, which in turn triggered a rebuild of every package dependend on openssl.

Thanks for the pointers!

[-- Attachment #2: Type: text/html, Size: 399 bytes --]

Re: [yocto] Hashserve OEEquivHash unihash issue

[Joshua Watt]

[-- Attachment #1: Type: text/plain, Size: 5037 bytes --]

On Thu, Sep 10, 2026 at 1:42 PM Harro Verton via lists.yoctoproject.org
<wanwizard=openpli.org@lists.yoctoproject.org> wrote:

> Hello Community,
>
> First post, and a relative OE newbie, so please bear with me.
>
> I've inherited a build environment of an open source project (
> https://github.com/OpenPLi/openpli-oe-core ). The project builds images
> for some 120 different devices, with MIPSEL or ARM SoCs, covering 5
> different ARCHs.
>
> All are build from a single repository, containing the submodules for
> Bitbake, Openembedded-Core and meta-openembedded, and all required BSP's
> for these devices. They all share sstate, a PR server, and a Hashserve
> server.
>
> In order to avoid packages build for the ARCH feed rebuilding for all
> subsequent MACHINEs using that same ARCH, my predecessor has introduced
> HashServe, using OEEquivHash.
>
> This works fine in our previous release (which is based on Hardknott), and
> worked fine in our current release (which is based on Scarthgap), until a
> few months ago.
> We want to start with the next release, based on Wrynose, but we can't
> until we can make a final Scarthgap release, which we can't because of this
> issue.
>
> I've run a  test, building two different MACHINEs with the same ARCH
> (identical hardware, different brands, different drivers, so separate
> BSPs), with an empty build directory (so no sstate cache, new PR database,
> no hash data). For the second machine, 230 tasks are recorded as having a
> changed unihash, which triggers a recipe rebuild for the second MACHINE.
>
> I've enabled hashserve debugging, and these recipes log something like:
> log.do_package_write_ipk.212153:DEBUG: Reported task
> efe6373f8d36e03f92f0198b5c2891fba8b459bd1e16ee9043f3e3d85a47c65e as unihash
> efe6373f8d36e03f92f0198b5c2891fba8b459bd1e16ee9043f3e3d85a47c65e to
> unix:///data/openpli/oe/scarthgap/build/hashserve.sock
> log.do_package_write_ipk.2223688:DEBUG: Task
> 8da90b016d0a41cac6ce46c20d8ee64d5e7af35687de43c306544508050227e8 unihash
> changed 8da90b016d0a41cac6ce46c20d8ee64d5e7af35687de43c306544508050227e8 ->
> efe6373f8d36e03f92f0198b5c2891fba8b459bd1e16ee9043f3e3d85a47c65e by server
> unix:///data/openpli/oe/scarthgap/build/hashserve.sock
> so the second build records a changed unihash, which triggers a rebuild of
> the package, and leads to an r0.1 version of the ipk.
>
> All of them but 8 report a unihash change for  do_package_qa, those 8
> also for do_package_write_ipk.
>
> The downside of this is that after device A is flashed with the result of
> the first build, and you do an "opkg list-upgradable", you're presented
> with 182 packages from the ARCH feed shared between the two devices that
> are supposedly changed (but the only change is r0.0 to r0.1 due to the
> rebuild). Apart from the fact that most of our users are not tech savvy,
> several devices are limited on flash memory space, so having to update 180+
> packages every time we have a build run is quite a bit of an issue.
>
> I can't find a reason why this suddenly happens, some of these packages
> are quite basic and don't contain binary code, packages like
> python3-requests for example. And I have no idea how to debug this further.
>
> For this test build, Bitbake is at yocto-5.0.19, openembedded-core and
> meta-openembedded are on the scarthgap branch, pulled yesterday.
>
> I've looked at the commit history in our repo, the only time this trio was
> updated was on July 30th. If I revert that, bitbake goes back to
> yocto-5.0.15 and both OE submodules to a commit in december 2025, which is
> when we started with our Scarthgap branch.
> When I repeat the test build after the revert, only two recipes for the
> ARCH feed are rebuild for the second machine, ffmpeg and gstreamer, both
> are recipes that we override in our own meta layer ( version upgrades to
> those in openembedded ).
>
> So somewhere along the line, something got broken (or how it worked
> changed, and we missed that).
>
> re there fixes in later versions for this issue, that are not backported
> to Scarthgap? How can I debug what exactly causes the unihash to change? Or
> better, how do I fix this?
>

Are you using a PR server to automatically bump the PR? There could be some
strange interaction happening between that and the hash server (and TBH,
it's not entirely clear how they should co-exist).

Also, are you running an external hash server? If so, is it the latest
version?


>
> TIA,
>
> Harro Verton, member of the PLi Association
> https://openpli.org
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#66715):
> https://lists.yoctoproject.org/g/yocto/message/66715
> Mute This Topic: https://lists.yoctoproject.org/mt/121187356/3616693
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [
> JPEWhacker@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 7776 bytes --]