Re: [Buildroot] [PATCH] utils/brmake: filter output for parallel build

["Yann E. MORIN via buildroot" <buildroot@buildroot.org>]

[-- Attachment #1: Type: text/plain, Size: 2527 bytes --]

Vincent, Marcus, All,

On 2025-10-17 16:39 +0200, Vincent Stehlé spake thusly:
> On Fri, Oct 17, 2025 at 04:12:34PM +0200, Marcus Hoffmann wrote:
> > I have seen the following thing happen once during a CI run, where 2 of the
> > lines where truncated in the middle of the timestamp. I have looked through
> > a bunch of build logs now but haven't seen this again, so not sure if that's
> > an actual problem we need to care about
[--SNIP--]
> > And secondly, also in CI:
> > The "Done in Xmin Ys" text that is printed in the end is still always
> > displayed in the middle of a line during the end of the build output:
> > 
> > 2025-10-17T08:52:28 >>> owfs 3.2p4 Installing to staging directory
> > 2025-10-17T08:52:29 >>> owfs 3.2p4 Fixing libtool fDone in 7min 37s
> > iles
> > 2025-10-17T08:52:30 >>> owfs 3.2p4 Installing to target
> > 2025-10-17T08:52:38 >>> python-pydantic-core 2.33.2 Installing to target
> > 2025-10-17T08:52:52 >>> e2fsprogs 1.47.2 Building
> > [...]
[--SNIP--]
> What you described is a valid reason I think to add the --unbuffered argument to
> sed; I will respin a v2.

For what it's worth, I've locally extensively tweaked and almost
rewritten brmake over the years since I initially submitted the original
one, and which addresses all the issues I've seen mentionned in this
thread (but I only skimmed over0.

Basically, it inverses the filtering and the tee, tee-ing before
filtering. It also uses a bash coproc to beter catch-and-terminate the
process in case of Ctrl-C (but it is not 100% perfect).

The most important thing is that is forces -Otarget (unless -O was
already specified on the command line) to group per-target output and
avoid mixing output of multiple recipes.

It got quite a bit more complex, of course, but hey... :-]

With this, I can no longer observe any mangled or mixed-up line.

Feel free to properly submit it if you like it better. ;-)

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[-- Attachment #2: brmake --]
[-- Type: text/plain, Size: 3098 bytes --]

#!/bin/bash

trap_ctrl-c() {
    if [ -n "${MAKE_PID}" ]; then
        printf '\rInterrupted, waiting for jobs to finish...\n' >&6
        kill -INT ${MAKE_PID}
        wait ${MAKE_PID}
        exit ${?}
    fi
}

disp_time() {
    local start="${1}"
    local S h m s

    if [ "${BRMAKE_ABS_TIME}" ]; then
        printf '%(%T)T' -1
    else
        S=$((${EPOCHSECONDS:-$(date +%s)}-start))
        h=$((S/3600))
        m=$(((S%3600)/60))
        s=$((S%60))
        [ "${h}" -eq 0 ] || printf '%d:' "${h}"
        printf '%02d:%02d' "${m}" "${s}"
    fi
}

main() {
    local ret start dir_found dir O log_file outsync

    start=${BR_START:-${EPOCHSECONDS:-$(date +%s)}}

    # Crude parsing to find the directory with the .config
    dir="$(pwd)"
    dir_found=false
    outsync=-Otarget
    for i in "${@}"; do
        ${dir_found} && { dir="${i}"; dir_found=false; continue; }
        case "${i}" in
        (-C|--directory)    dir_found=true;;
        (-C*)               dir="${i#-C}";;
        (--directory=*)     dir="${i#--directory=}";;
        (O=*)               O="${i#O=}";;
        (-O|-O*)            outsync=;;
        esac
    done

    if [ -n "${O}" -a -e "${O}/.config" ]; then
        log_file="${O}/br.log"
    elif [ -z "${O}" -a -e "${dir}/.config" ]; then
        log_file="${dir}/br.log"
    else
        # No .config, don't log, and let Buildroot do its magic (i.e., whine)
        exec make "${@}"
    fi

    {   coproc LOGGER \
        {
            printf '\r%s' "$(disp_time "${start}")"
            IFS="${CR}"
            tee -a "${log_file}" \
            |sed -u -r -e '/^.{4}(>>> .+)/!d; s//\1/' \
            |while true; do
                if read -r -t 0.25 line; then
                    printf '\r%s %s%s%s\n' \
                        "$(disp_time "${start}")" \
                        "${BRMAKE_PREFIX:+${BRMAKE_PREFIX} }" \
                        "${prev}" \
                        "${line}"
                    disp_time "${start}"
                    prev=""
                elif [ ${?} -gt 128 ]; then
                    prev="${prev}${line}"
                    printf '\r%s' "$(disp_time "${start}")"
                else
                    break
                fi
            done
            exit 0
        } >&3
    } 3>&1

    trap trap_ctrl-c INT

    exec 5>&1 16>&1 6>&2 >&${LOGGER[1]} 2>&1 4>&1
    exec 16>&-
    ${MAKE:-make} ${outsync} "${@}" 2>&1 &
    MAKE_PID="${!}"
    wait ${MAKE_PID}
    ret="${?}"
    MAKE_PID=""
    exec >&5 2>&6 4>&-

    # bash seems to have issues closing an fd when it is an indexed-array
    # expansion, while it has no problem opening it (see above).
    eval exec "${LOGGER[1]}>&-"
    wait ${LOGGER_PID}

    printf '\r%s %s>>> ' "$(disp_time "${start}")" "${BRMAKE_PREFIX:+${BRMAKE_PREFIX} }"
    if [ "${BRMAKE_ABS_TIME}" ]; then
        printf 'Done in %s' "$(BRMAKE_ABS_TIME= disp_time "${start}")"
    else
        printf 'Finished at %(%T)T' -1
    fi
    if [ ${ret} -ne 0 ]; then
        printf ' (error %d)' ${ret}
    fi
    printf '\n'

    return ${ret}
}

CR="
"

main "${@}"

[-- Attachment #3: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot