Re: [Buildroot] [PATCH] utils/brmake: filter output for parallel build

["Vincent Stehlé" <vincent.stehle@arm.com>]

On Thu, Oct 09, 2025 at 07:42:17PM +0200, Marcus Hoffmann via buildroot wrote:
> Hi Vincent!

Hi Marcus,

Thanks for having a look.

> 
> On 10/9/25 17:28, Vincent Stehlé wrote:
> > When building in parallel with per-package directories
> > (BR2_PER_PACKAGE_DIRECTORIES=y), brmake output is often garbled:
> 
> This has bothered me for sooo long, but I never found out why this is going
> wrong. Do you know why this specifically fails with PPDs (which I always
> enable, so I never knew this was not broken without them)?

Yes, this is a limitation of make parallel builds. If we look at the brmake
"pipeline":

    make "${@}" \

At this point we already have all the messages from all the threads, often mixed
on the same line:

  checking for stdint.h... >>> host-dtc 1.7.2 Installing to host directory

The rest of the pipeline just adds the timestamp, copies all the messages to the
br.log and keeps only the messages with the ">>>" markers for the console
output:

    &> >( while read -r line; do
             printf "%(%Y-%m-%dT%H:%M:%S)T %s\n" -1 "${line}"
         done \
         |tee -a br.log \
         |grep --colour=never -E '>>>'
       )

The simplest thing we can do at this point is to cleanup a bit "after the fact".

This is not perfect, but this is far simpler than what aiaiai is doing for
example, with intermediate logfiles and a lock:

https://git.infradead.org/?p=users/dedekind/aiaiai.git;a=blob;f=helpers/aiaiai-make-kernel;h=4338a2764a094f29f4feb38cc3ccdd26336b1ba6;hb=eadfb4d5c88d2dec2842ba8986df42f111d4a7a8#l119

> 
> > 
> >    2025-10-08T18:39:10 >>> host-dtc 1.7.2 Building
> >    2025-10-08T18:39:11 checking for stdint.h... >>> host-dtc 1.7.2 Installing to host directory
> >    2025-10-08T18:39:12 checking for limits.h... >>> host-gmp 6.3.0 Installing to host directory
> > 
> > Remove the spurious string between the timestamp and the ">>>" marker to
> > fix this.
> > We need some extra care to preserve the preceding "term bold" special
> > characters sequence.
> > 
> > Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>
> > Cc: Yann E. MORIN <yann.morin.1998@free.fr>
> > ---
> >   utils/brmake | 3 ++-
> >   1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > diff --git a/utils/brmake b/utils/brmake
> > index a4b9e7fa71..2a74d58167 100755
> > --- a/utils/brmake
> > +++ b/utils/brmake
> > @@ -12,7 +12,8 @@ main() {
> >                printf "%(%Y-%m-%dT%H:%M:%S)T %s\n" -1 "${line}"
> >            done \
> >            |tee -a br.log \
> > -         |grep --colour=never -E '>>>'
> > +         |grep --line-buffered --colour=never -E '>>>' \
> 
> Could you explain what --line-buffered does here and why we need this? (The
> man page wasn't too helpful in just specifying "Use line buffering on
> output".

This tells grep to output its results line by line instead of buffering for too long.
Without this option, brmake output is printed in "bursts", and only from times
to times; not very "live".

You can just remove it and try to see the difference, as this is otherwise
functionally identical.

Best regards,
Vincent.

> 
> > +         |sed -E 's/^([0-9T:-]{19}) [[:print:]]*(.*>>> )/\1 \2/'
> >          )
> >       ret=${?}
> 
> Best,
> Marcus
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot