Re: [yocto] Switching between multiple DISTROs without "contamination"

[Mike Looijmans <mike.looijmans@topic.nl>]

Hi Nicolas,

I guess part of your problem is that DISTRO does not end up in any 
package grouping, unlike MACHINE and ARCH and so.

So if OE/Yocto does The Right Thing, this means that it will have to 
remove a bunch of packages from your feed and sysroot, and add them 
again (from sstate-cache or by building) every time you switch DISTRO 
(which, on a build server, would be each and every build).
Hence my advice to use a separate build directory for each, otherwise 
Yocto will have to juggle with several packages on each and every build. 
With accidents waiting to happen.

Hence my advice remains, you can have "my-image-dev" and even 
"my-machine-dev" in the same build directory, but don't share 
"my-distro-dev" that way, it's an invitation to trouble, even though It 
Should Work In Theory...

Mike.


Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topic.nl

Please consider the environment before printing this e-mail
On 13-07-2022 15:32, Nicolas Jeker wrote:
> Thanks Martin and Mike for your explanations and tips.
>
> So, I've done a lot of testing today and it seems I simplified the
> example in my first email a bit too much. The example as-is works fine
> when switching DISTROs as far as I can tell. The problem only arises
> when wildcards are used.
>
> Changing my initial example like this should trigger the behaviour I've
> initially described:
>
> SRC_URI:append:mydistro-dev = " file://application-dbg.service"
>
> do_install {
>      # ...snip...
>      # systemd service
>      install -d ${D}${systemd_system_unitdir}
>      install -m 0644 ${WORKDIR}/*.service ${D}${systemd_system_unitdir}
> }
>
> do_install:append:mydistro-dev() {
>      # debug systemd services
>      install -d ${D}${systemd_system_unitdir}
>      install -m 0644 ${WORKDIR}/application-dbg.service
> ${D}${systemd_system_unitdir}
> }
>
> Notice the *.service in do_install.
>
>  From my testing, this is how contamination happens:
>
> 1) Build with 'DISTRO=mydistro bitbake application'. All tasks for the
> recipe are run and the directories in WORKDIR are populated, including
> the "application.service" file.
> 2) Build with 'DISTRO=mydistro-dev bitbake application'. do_unpack is
> rerun and places the additional "application-dbg.service" file in
> WORKDIR.
> 3) Switching back to 'mydistro' will get the recipe from sstate cache,
> which works fine.
> 4) Changing application.bb and rebuilding with 'DISTRO=mydistro bitbake
> application' reruns do_install (as expected). This leads to the
> packages do_install picking up the additional "application-dbg.service"
> file left behind by the invocation in step 2).
>
> Mike, Martin: Do you remember in which cases you encountered problems
> when sharing the build directory?
>
> On Tue, 2022-07-12 at 09:15 -0700, Khoi Dinh Trinh wrote:
>> Thank you Nicolas for asking this question since I will probably run
>> into this issue soon if not for this email thread. The answers so far
>> have been very helpful but I just want to clarify a bit more on why
>> doesn't the package get rebuilt? From my understanding, Yocto should
>> rerun a task when the signature of the task changes and since
>> do_install has an override on mydistro-dev, shouldn't the content and
>> thus the signature of do_install change when switching distro and so
>> Yocto should rerun it?
> As far as I can tell, all the relevant tasks are rerun correctly when
> something is changed. Relevant tasks meaning only the tasks that are
> actually different.
>
> The specific issue I experienced is due to the WORKDIR not being
> cleaned between different task invocations and the recipe (probably
> wrongfully) relying on wildcards to gather files, see example above.
>
>> I have a lot of tasks with override not just on DISTRO but other
>> things like MACHINE or custom variables so I want to understand the
>> rebuild mechanism as best I can.
>>
> There's surely someone more knowledgeable here that could clarify the
> inner workings of this mechanism a lot better than me.
>
>> Best,
>> Khoi Trinh
>>
>> On Tue, Jul 12, 2022 at 8:05 AM Mike Looijmans
>> <mike.looijmans@topic.nl> wrote:
>>> Quick answer: Don't build multiple distros in one build directory.
>>>
> It's really a bummer that it's not reliably possible to switch between
> DISTROs inside one build directory.
>
>>> You might get away with setting TMPDIR = "tmp-${DISTRO}" to give
>>> each
>>> its own.
>>>
>>> But I'd rather advice to set up two separate builds and just point
>>> the
>>> downloads and sstate-cache to the same location. It'll be faster
>>> than
>>> the TMPDIR option.
>>>
>>> Or figure out how to put the difference in the IMAGE only. Then you
>>> can
>>> just build both images (in parallel, woot), which is faster, more
>>> convenient and saves on diskspace.
> As I'm currently working with something close to what you describe, I
> think I'll try to stay away from multiple DISTROs if possible and
> improve on what I'm already doing.
>
>>> What I often do is have my-application.bb generate a
>>> my-application-utils package that only gets installed in the "dev"
>>> image
>>> but not in the production one, which only installs "my-
>>> application".
> This is probably what Martin meant with his A.bb and A-xx.bb example.
> It's so far one of the best approaches I've seen, thanks.
>
>>> You could also create a "my-application-dev.bb" recipe that
>>> includes
>>> my-application.bb and just changes what it needs to be different.
>>>
>>>
>>>
>>>
>>> Met vriendelijke groet / kind regards,
>>>
>>> Mike Looijmans
>>> System Expert
>>>
>>>
>>> TOPIC Embedded Products B.V.
>>> Materiaalweg 4, 5681 RJ Best
>>> The Netherlands
>>>
>>> T: +31 (0) 499 33 69 69
>>> E: mike.looijmans@topicproducts.com
>>> W: www.topic.nl
>>>
>>> Please consider the environment before printing this e-mail
>>> On 12-07-2022 15:37, Nicolas Jeker via lists.yoctoproject.org
>>> wrote:
>>>> Hi all,
>>>>
>>>> I'm currently using an additional layer and image to
>>>> differentiate
>>>> between a release and development build (enabling NFS, SSH, root
>>>> login,
>>>> etc.). To create a development build, I manually add the layer to
>>>> bblayers.conf. This works quite well, but feels a bit clumsy to
>>>> integrate into a CI/CD pipeline.
>>>>
>>>> Per these past discussions here [1][2], I'm now trying to migrate
>>>> to
>>>> multiple DISTROs, something like "mydistro" and "mydistro-dev".
>>>>
>>>> While migrating some of the changes, I discovered that I run into
>>>> caching(?) issues. I have a recipe for an internal application
>>>> and want
>>>> to include additional systemd service files in the development
>>>> image.
>>>>
>>>> What I did was this:
>>>>
>>>> Added "application-dbg.service" to recipes-
>>>> internal/application/files
>>>>
>>>> Adapted application.bb recipe:
>>>>
>>>> SRC_URI:append:mydistro-dev = " file://application-dbg.service"
>>>>
>>>> do_install {
>>>>        # ...snip...
>>>>        # systemd service
>>>>        install -d ${D}${systemd_system_unitdir}
>>>>        install -m 0644 ${WORKDIR}/application.service
>>>> ${D}${systemd_system_unitdir}
>>>> }
>>>>
>>>> do_install:append:mydistro-dev() {
>>>>        # debug systemd services
>>>>        install -d ${D}${systemd_system_unitdir}
>>>>        install -m 0644 ${WORKDIR}/application-dbg.service
>>>> ${D}${systemd_system_unitdir}
>>>> }
>>>>
>>>>
>>>> When I run "DISTRO=mydistro-dev bitbake application" followed by
>>>> "DISTRO=mydistro bitbake application", the debug service file is
>>>> still
>>>> present in the package. This seems to be caused by the "image"
>>>> directory in the recipe WORKDIR not being cleaned between
>>>> subsequent
>>>> do_install runs. Is this expected behaviour? What's the best
>>>> solution?
>>>>
>>>> Kind regards,
>>>> Nicolas
>>>>
>>>> [1]:
>>>> https://lore.kernel.org/yocto/CAH9dsRiArf_9GgQS4hCg5=J_Jk6cd3eiGaOiQd788+iSLTuU+g@mail.gmail.com/
>>>> [2]:
>>>> https://lore.kernel.org/yocto/VI1PR0602MB3549F83AC93A53785DE48677D3FD9@VI1PR0602MB3549.eurprd06.prod.outlook.com/
>>>>
>>>>
>>>>
>>> -- 
>>> Mike Looijmans
>>>
>>>
>>>
>>>

-- 
Mike Looijmans