Conditional DAG recalculation at build time

Conditional DAG recalculation at build time

[Daniel Levin]

[-- Attachment #1: Type: text/plain, Size: 998 bytes --]

In short: does bitbake support dependency graph recalculation at build time?

I am looking for a dependency resolution similar to BYPRODUCTS in CMake and
restat in Ninja.

In other words when task _after_ execution might tell that output did not
actually change. So all dependencies of that target should not be updated.
Even when initially they were treated as (potentially) to be updated.

Usually in CMake that results in remaining task count decreased instantly
during build time, because in fact they must not be running.

I found documentation in bitbake about task _input_ checksum, which is used
to evaluate whether task need to be executed.
So I am looking for the similar checksum for the task _output_. And when
task does not change that output checksum means that dependencies of that
output does not need to be triggered.

If this topic is well known and documented then could you please point me
to the appropriate section in the documentation?

Thanks,
Daniel

[-- Attachment #2: Type: text/html, Size: 1161 bytes --]

Re: Conditional DAG recalculation at build time

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1629 bytes --]

Afaik Bitbake does not currently have that capability. It's a feature I've
wanted to add for a long time, but it hasn't been a priority for us up to
now. I believe patches to add such a feature would be welcomed. I'd suggest
opening a bug in bugzilla for tracking potential future development of such
a feature.

On Mon, Apr 2, 2018 at 3:23 PM Daniel Levin <dendy.ua@gmail.com> wrote:

> In short: does bitbake support dependency graph recalculation at build
> time?
>
> I am looking for a dependency resolution similar to BYPRODUCTS in CMake
> and restat in Ninja.
>
> In other words when task _after_ execution might tell that output did not
> actually change. So all dependencies of that target should not be updated.
> Even when initially they were treated as (potentially) to be updated.
>
> Usually in CMake that results in remaining task count decreased instantly
> during build time, because in fact they must not be running.
>
> I found documentation in bitbake about task _input_ checksum, which is
> used to evaluate whether task need to be executed.
> So I am looking for the similar checksum for the task _output_. And when
> task does not change that output checksum means that dependencies of that
> output does not need to be triggered.
>
> If this topic is well known and documented then could you please point me
> to the appropriate section in the documentation?
>
> Thanks,
> Daniel
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>

[-- Attachment #2: Type: text/html, Size: 2186 bytes --]

Re: Conditional DAG recalculation at build time

[Richard Purdie]

On Mon, 2018-01-29 at 23:06 -0800, Daniel Levin wrote:
> In short: does bitbake support dependency graph recalculation at
> build time?
> 
> I am looking for a dependency resolution similar to BYPRODUCTS in
> CMake and restat in Ninja.
> 
> In other words when task _after_ execution might tell that output did
> not actually change. So all dependencies of that target should not be
> updated. Even when initially they were treated as (potentially) to be
> updated.
> 
> Usually in CMake that results in remaining task count decreased
> instantly during build time, because in fact they must not be
> running.
> 
> I found documentation in bitbake about task _input_ checksum, which
> is used to evaluate whether task need to be executed.
> So I am looking for the similar checksum for the task _output_. And
> when task does not change that output checksum means that
> dependencies of that output does not need to be triggered.
> 
> If this topic is well known and documented then could you please
> point me to the appropriate section in the documentation?

There are mailing list posts from me a long time ago when we added
checksum support and sstate explaining why we use input values rather
than outputs and the pros/cons of either.

Also, more recently, have a read of the thread "[yocto] RFC: Backwards
compatibility checking sstate-cache" on the yocto list. Its a similar
request.

My own thoughts are that the best way forward would be some kind of
equivalence server so that we could remap sstate values back to a
master value.

The first step would be to allow such a server to exist and to allow
resolution of a hash to a master base value.

Allowing bitbake to recalculate its taskgraph and dynamically adapt to
this would then be a secondary step in the interests of efficiency.

The first piece would be minimally invasive. The second piece would
mean huge changes to the way bitbake operates.

I'm not sure if anyone is pursuing this but its an interesting topic
I've given a bit of thought to.

Cheers,

Richard