Took a lot of digging but the issue was found.

 

Turned out someone added a bbappend for openssl.bb, and included a PACKAGE_ARCH = $MACHINE_ARCH.

 

The caused openssl to be rebuild for every MACHINE, which in turn triggered a rebuild of every package dependend on openssl.

 

Thanks for the pointers!