Re: [docs] [PATCH v2 3/3] tools/build-docs-container: build in separate directory for each distro

["Antonin Godard" <antonin.godard@bootlin.com>]

Hi,

On Wed Oct 15, 2025 at 1:37 PM CEST, Quentin Schulz via lists.yoctoproject.org wrote:
[...]
>> With this, additional runs with different SHAs will just stack next to this,
>> re-creating the symlink only when it is not a directory. This way, I know that
>> for any previous run of build-docs-container, I can open the "_build" symlink
>> to get the latest output. Of course I need to be careful when running concurrent
>> builds, but that is another use-case, a bit different from day-to-day
>> development IMO.
>> 
>
> Mmmmm...
>
> So I think we could make this concurrent-safe by using an flock on some 
> file?
>
> e.g. surround any Make target that modifies the source directory with an 
> flock, something like shown here: https://unix.stackexchange.com/a/382916
>
> And actually... I think we already have some issue with concurrency 
> without calling make multiple times from the same tree.
>
> I believe when you run make all, make will run make html, make epub and 
> make latexpdf in parallel which will all call set_versions.py which 
> modifies the source tree. Same would happen if passing multiple targets 
> to make (e.g. make latexpdf html). I don't think the f.write() and 
> w.write() in set_versions.py are concurrent-safe.
>
>> What do you think?
>> 
>
> With the flock above, we could simply add the creation of a 
> _build/latest symlink which points to the last make target that ran. 
> Maybe we want a _build/latest/html for the last make html target, 
> otherwise imagine you run make all once and then make html one more time 
> for another distro, then _build/latest will now point to _build/distroB/ 
> which didn't generate the html part). But that could also mean we should 
> split the build directories for latexpdf, html and epub so this works... 
> but that means not being able to reuse the sphinx "cache" (I believe 
> there's one to do incremental builds? maybe it's also used for different 
> output types?).
>
> I think we (I) may be overengineering this a tiny bit :)
>
> I think we probably should add an flock at least for the writing of the 
> files by set_versions.py if that proves to be an issue (testing needs to 
> be done :) )

Sorry for the delay answering to this, I was caught up with the Whinlatter
release and never had the time to properly look at this.

I've sent a respin of this with what we were mentioning here, copied you to it:
https://lore.kernel.org/r/20251222-concurrent-safety-v1-0-e3d86e44cd38@bootlin.com

Thanks,
Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com