Re: [PATCH v3 1/3] scripts: kconfig: merge_config.sh: refactor from shell/sed/grep to awk

[Mikko Rapeli <mikko.rapeli@linaro.org>]

Hi,

On Mon, Mar 09, 2026 at 10:09:04AM -0700, Nathan Chancellor wrote:
> On Mon, Mar 09, 2026 at 05:38:58PM +0100, Andreas Larsson wrote:
> > On 2026-01-22 11:57, Mikko Rapeli wrote:
> > > From: Anders Roxell <anders.roxell@linaro.org>
> > > 
> > > merge_config.sh shell/sed/grep loop scales poorly and is slow.
> > > With Yocto genericarm64 kernel and around 190 config fragments
> > > the script takes more than 20 minutes to run on a fast build machine.
> > > Re-implementation with awk does the same job in 10 seconds.
> > > Using awk since it is likely available in the build environments
> > > and using perl, python etc would introduce more complex runtime
> > > dependencies. awk is good enough and lot better than shell/sed/grep.
> > > 
> > > Output stays the same but changed execution time means that
> > > parallel job output may be ordered differently.
> > > 
> > > Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
> > > Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> > > ---
> > >  scripts/kconfig/merge_config.sh | 168 ++++++++++++++++++++++++--------
> > >  1 file changed, 128 insertions(+), 40 deletions(-)
> > 
> > Hi,
> > 
> > Commit 5fa9b82cbcfc ("scripts: kconfig: merge_config.sh: refactor from
> > shell/sed/grep to awk") breaks merge_config.sh for me:
> > 
> > ---------->%----------
> > $ make tiny.config
> >   HOSTCC  scripts/basic/fixdep
> >   HOSTCC  scripts/kconfig/conf.o
> >   HOSTCC  scripts/kconfig/confdata.o
> >   HOSTCC  scripts/kconfig/expr.o
> >   LEX     scripts/kconfig/lexer.lex.c
> >   YACC    scripts/kconfig/parser.tab.[ch]
> >   HOSTCC  scripts/kconfig/lexer.lex.o
> >   HOSTCC  scripts/kconfig/menu.o
> >   HOSTCC  scripts/kconfig/parser.tab.o
> >   HOSTCC  scripts/kconfig/preprocess.o
> >   HOSTCC  scripts/kconfig/symbol.o
> >   HOSTCC  scripts/kconfig/util.o
> >   HOSTLD  scripts/kconfig/conf
> > The base file '.config' does not exist. Creating one...
> > Using .config as base
> > Merging ./kernel/configs/tiny.config
> > awk: cannot open ./.tmp.config.U9SROCKTBj.new (No such file or directory)
> > mv: cannot stat './.tmp.config.U9SROCKTBj.new': No such file or directory
> > make[2]: *** [scripts/kconfig/Makefile:112: tiny.config] Error 1
> > make[1]: *** [<kernel-source-top-dir>/Makefile:744: tiny.config] Error 2
> > make: *** [Makefile:248: __sub-make] Error 2
> > ---------->%----------
> > 
> > with this underlying call to merge_config.sh, an empty .config and where
> > my awk is GNU Awk 5.1.0:
> > 
> > ---------->%----------
> > $ make tiny.config V=1
> > make --no-print-directory -C <kernel-source-top-dir> \
> > -f <kernel-source-top-dir>/Makefile tiny.config
> > make -f ./scripts/Makefile.build obj=scripts/basic
> > make -f ./scripts/Makefile.build obj=scripts/kconfig tiny.config
> > # cmd_merge_fragments tiny.config
> >   ./scripts/kconfig/merge_config.sh -m .config ./kernel/configs/tiny.config ./arch/x86/configs/tiny.config
> > ...
> > 
> > $ awk --version
> > GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0, GNU MP 6.2.1)
> > ---------->%----------
> > 
> > At 5fa9b82cbcfc~ things works as expected.

Hmm, all these steps work for me with GNU awk 5.1.0 so something else
is different here. On line 267 awk is given input file args
"$ORIG_MERGE_FILE" "$TMP_FILE" "$TMP_FILE.new" and it fails to
create the output file "$TMP_FILE.new" with >> and plain print statement.

> Hmmm, not sure how I have not seen this error myself since I test
> tiny.config but I can reproduce with a clean output directory. Something
> like this seems like a simple fix but the only instances of ARGV[3] in
> the awk script that I can see use '>>', which should create the file if
> it does not exist. Mikko, any ideas?
> 
> diff --git a/scripts/kconfig/merge_config.sh b/scripts/kconfig/merge_config.sh
> index 735e1de450c6..070ecae87a1c 100755
> --- a/scripts/kconfig/merge_config.sh
> +++ b/scripts/kconfig/merge_config.sh
> @@ -123,6 +123,7 @@ fi
>  MERGE_LIST=$*
>  
>  TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX)
> +touch "$TMP_FILE.new"
>  
>  echo "Using $INITFILE as base"

This could help but I fail to understand why this would be needed. Why is awk
not able to create this file on line 256?

awk manual says:

https://www.gnu.org/software/gawk/manual/html_node/Redirection.html

    print items >> output-file

    This redirection prints the items into the preexisting output file named
    output-file. The difference between this and the single-‘>’ redirection
    is that the old contents (if any) of output-file are not erased. Instead,
    the awk output is appended to the file. If output-file does not exist,
    then it is created.

And in all my testing this works. Which distro is this? Is something else like
file system setup in some way unusual?

Cheers,

-Mikko