Re: [Buildroot] [PATCH RFC 1/1] boot/edk2: unset MAKEFLAGS in build environment

["Yann E. MORIN" <yann.morin.1998@free.fr>]

Julien, All,

On 2023-12-31 19:14 +0100, Julien Olivain spake thusly:
> Yann reported in [1] that edk2 build could sometimes fail. The issue
> can be reproduced when per-package directories is enabled, or also
> when building on a system with GNU Make >= 4.4 using the
> "--shuffle=reverse" option (such as Fedora 39). Those are pointing
> toward a Makefile dependency issue.
> 
> The issue can be reproduced with commands:
> 
>     cat > .config <<EOF
>     BR2_riscv=y
>     BR2_RISCV_64=y
>     BR2_TOOLCHAIN_EXTERNAL=y
>     BR2_TARGET_EDK2=y
>     EOF
>     make olddefconfig
> 
> Then, building either with:
> 
>     make --shuffle=reverse
> 
> Or:
> 
>     utils/config -e BR2_PER_PACKAGE_DIRECTORIES
>     make olddefconfig
>     make -j$(nproc)
> 
> It is interesting to mention that when using "make --shuffle=reverse"
> to build, the build can be completed if restarted only with "make". It
> will not pull any other Buildroot package. This fact hints toward a
> Makefile dependency issue internal to the EDK2 build system, rather
> than in the Buildroot recipe.
> 
> The EDK2 build system is quite unique. See [2]. It generates files,
> makefiles and internally uses GNU Make to compile code. This system is
> likely not tested as being a sub-Make process in a complex Makefile
> such as Buildroot.
> 
> In order to prevent Buildroot to pass unexpected Make flags to the
> EDK2 sub-Make, this commit unset the MAKEFLAGS variable in the EDK2
> build environment. This will put the EDK2 build script in a more
> common and tested state. See GNU Make documentation about recursive use
> of Make, more specifically [3].
> 
> Note: as mentioned, the build failure is likely due to an internal
> issue of the EDK2 build system. The failure points to a missing
> dependency in the EDK2 generator itself. This commit does not fix this
> issue, but rather put the EDK2 build system in a normalized
> environment, avoiding Buildroot flags being passed to the internal
> EDK2 sub-Make invocation. The upstream EDK2 build system most likely
> need a fix too.
> 
> Fixes:
> 
>     make[2]: *** No rule to make target '/buildroot/output/build/edk2-edk2-stable202308/Build/RiscVVirtQemu/RELEASE_GCC5/RISCV64/MdeModulePkg/Application/UiApp/UiApp/DEBUG/UiApp.efi', needed by '/buildroot/output/build/edk2-edk2-stable202308/Build/RiscVVirtQemu/RELEASE_GCC5/FV/Ffs/462CAA21-7614-4503-836E-8AB6F4662331UiApp/UiApp.offset'.  Stop.
> 
>     build.py...
>      : error 7000: Failed to execute command
> 	    make tbuild [/buildroot/output/build/edk2-edk2-stable202308/Build/RiscVVirtQemu/RELEASE_GCC5/RISCV64/MdeModulePkg/Application/UiApp/UiApp]
> 
>     build.py...
>      : error F002: Failed to build module
> 	    /buildroot/output/build/edk2-edk2-stable202308/MdeModulePkg/Application/UiApp/UiApp.inf [RISCV64, GCC5, RELEASE]
> 
> [1] https://lists.buildroot.org/pipermail/buildroot/2023-December/681507.html
> [2] https://tianocore-docs.github.io/edk2-BuildSpecification/draft/4_edk_ii_build_process_overview/42_build_process_overview.html
> [3] https://www.gnu.org/software/make/manual/make.html#Options_002fRecursion
> 
> Reported-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Signed-off-by: Julien Olivain <ju.o@free.fr>

Thank you very much for this very detailed explanations! I see you
marked is as RFC, but I believe it is good to go as-is: the workaround
is perfectly sensible.

Of course, if upstream were to fix that issue, it would be better, but
given the complexity of it, we are probably not in a position to
properly fix it in a timely manner.

Applied to master, thanks.

Regards,
Yann E. MORIN.

> ---
>  boot/edk2/edk2.mk | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/boot/edk2/edk2.mk b/boot/edk2/edk2.mk
> index efdb4d02ce..87ac80c88f 100644
> --- a/boot/edk2/edk2.mk
> +++ b/boot/edk2/edk2.mk
> @@ -150,7 +150,13 @@ EDK2_BASETOOLS_OPTS = \
>  
>  EDK2_PACKAGES_PATH = $(subst $(space),:,$(strip $(EDK2_PACKAGES_PATHS)))
>  
> +# EDK2 "build" script internally uses and calls "make", which controls
> +# its own flags. It is mainly tested while not being a sub-make. In
> +# order to stay in that configuration, we avoid leaking top-level
> +# Buildroot make flags into EDK2 build by clearing the MAKEFLAGS
> +# environment variable.
>  EDK2_BUILD_ENV += \
> +	MAKEFLAGS= \
>  	WORKSPACE=$(@D) \
>  	PACKAGES_PATH=$(EDK2_PACKAGES_PATH) \
>  	PYTHON_COMMAND=$(HOST_DIR)/bin/python3 \
> -- 
> 2.43.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot