Re: [RFC PATCH 0/3] Build once, test everywhere

[Chuck Lever <cel@kernel.org>]

On 4/23/25 1:28 PM, Daniel Gomez wrote:
> On Wed, Apr 23, 2025 at 09:36:37AM +0100, Chuck Lever wrote:
>> On 4/23/25 8:34 AM, Daniel Gomez wrote:
>>> On Tue, Apr 22, 2025 at 11:49:03AM +0100, cel@kernel.org wrote:
>>>> From: Chuck Lever <chuck.lever@oracle.com>
>>>>
>>>> With apologies to the Java folks.
>>>>
>>>> I've been looking for ways to make our workflow runs more efficient
>>>> because my lab resources are limited, and moving to the cloud costs
>>>> money per CPU second.
>>>>
>>>> One thing that seems like an obvious win would be to build the test
>>>> kernel one time (for example, after a merge/pull request), then
>>>> re-use that kernel binary for all the workflows we want to run on
>>>> it.
>>>
>>> What about rebuilds between iterations?
>>> I think an extra step to improve that build time could be adding support for
>>> ccache in our kernel builds. Would it make sense to extend this in the future to
>>> enable a redis node?
>>>
>>> https://github.com/ccache/ccache/wiki/Redis-storage
>>>
>>> With the workflow suggestion below, I'd then do:
>>>
>>> 1. Bring up a redis node guest
>>> 2. Bring up a kernel builder node that pulls from redis node ccache
>>> 3. Push redis artifacts to control host (for later sync up) and destroy redis guest
>>> 4. Push kernel package to control host
>>
>> So I've asked our internal testing folks about using ccache to speed up
>> kernel builds. There were various reasons not to do this, one of which
>> is we want to have a clean trustworthy and repeatable kernel build to
>> test with.
>>
>> I'm not sure I have a specific technical argument against using ccache
>> except that our builder node is ephemeral, making its ccache usable for
>> only a single build. A Redis storage node might solve that issue.
> 
> Build artifacts can also be shared between the build nodes instances as long as
> they are the same following reproducible builds [1]
> 
> [1]
> https://docs.kernel.org/kbuild/reproducible-builds.html#reproducible-builds
> 
> But all right, I understand this is a separate topic and having ccache/redis/
> reproducible build features is something we can work with and extend as opt-in
> features in kdevops and/or this new workflow.
> 
> Thanks for the feedback.
> 
>> However I wonder if the time it takes to bring up the node, reload its
>> storage, then after the build save the redis artifacts and destroy it
>> might wipe out any benefit.
> 
> I guess the benefits will depend on every user, kernel config and how the cache
> is exposed.
> 
>>
>> With libvirt guests we can save several minutes by doing the guest bring
>> up in parallel (via Ansible) instead of serially via a shell script.
>> I've got some ideas about that.
>>
>>
>>> That said, all this new workflow makes sense to me. I guess instead of
>>> duplicating the bootlinux role into build_linux I'd like to see a linux
>>> role that handles build location (localhost/specific guest), output package
>>> generation or not, deployment, etc. Does that make sense to you too?
>>
>> The bootlinux role has grown a slew of special cases over the years, so
>> I'm in the mood to help reorganize it a little.
> 
> That would be great.
> 
>>
>> At least split it into a role that handles the kernel build, and one
>> that handles the minutiae of installing the builds on test runners.
>>
>> I'm not quite following your suggestion, but I'm interested in hearing
>> more detail.
> 
> I'm usually more in favor of having standalone playbooks. But I see here
> the potential of merging the features rather than having them separate. My
> suggestion is to refactor bootlinux role into separate tasks/<task name/
> feature>. Where the features are build_linux, artifacts, package creation, etc.

I did something similar to playbooks/roles/gen_nodes. I can experiment
and post some ideas.


-- 
Chuck Lever