Non-recursive makefiles, and static pattern rules

Non-recursive makefiles, and static pattern rules

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 2077 bytes --]

Hi Miguel,


As we talked earlier today, if you're trying to improve performance of
the kernel makefiles, I recommend checking a few things we've done in
the man-pages.git repo.

In the man-pages, we have a huge set of makefiles (not as large as the
kernel, but 120k is already huge for the scale of this project).  The
performance seems nice, and it's a non-recursive set of makefiles.

(If you're curious, the reason we have so much makefile code is that
 I set up a large number of linters to run in the build system, with
 'make -R lint'.  See 'make -R help' and 'make -R help-targets' if you
 want to look at what we have available there.)

We use static pattern rules:
<https://www.gnu.org/software/make/manual/html_node/Static-Pattern.html>

That is:

	targets …: target-pattern: prereq-patterns …
		recipe
		…

This results in a huge database when make(1) runs, being 10x larger than
what we would get with regular pattern rules.

However, recently I tried moving to pattern rules, since I thought the
smaller database could improve performace, but the experience was
surprising: the run time degraded by several orders of magnitude.
I don't remember the exact numbers, but a full
	'make -R -j24 build-all lint check'
(which is building almost everything that the build system can build)
would take a few minutes with the huge database, and several hours with
the size-optimized database using regular pattern rules.

I suspect it's because by having to guess for each file which rule to
apply (and being non-recursive, it has the whole list of thousands of
files, and around a hundred different patterns, so the combination is
explosive) is too expensive.  By using pattern rules, make(1) doens't
have to guess, and thus goes directly to the rule that applies to each
file.

I think this might solve the issues you were seeing in your experiment.
(Or it might not.)  :-)

Please let me know if this changes anything in your experiments!


Have a lovely night!
Alex


-- 
<https://www.alejandro-colomar.es>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Non-recursive makefiles, and static pattern rules

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 2386 bytes --]

> Date: 2026-10-07 22:07:46+0200
> From: Alejandro Colomar <alx@kernel.org>
>
> Hi Miguel,
> 
> 
> As we talked earlier today, if you're trying to improve performance of
> the kernel makefiles, I recommend checking a few things we've done in
> the man-pages.git repo.
> 
> In the man-pages, we have a huge set of makefiles (not as large as the
> kernel, but 120k is already huge for the scale of this project).  The
> performance seems nice, and it's a non-recursive set of makefiles.
> 
> (If you're curious, the reason we have so much makefile code is that
>  I set up a large number of linters to run in the build system, with
>  'make -R lint'.  See 'make -R help' and 'make -R help-targets' if you

Oops, it's s/help-targets/help-list-targets/


Cheers,
Alex

>  want to look at what we have available there.)
> 
> We use static pattern rules:
> <https://www.gnu.org/software/make/manual/html_node/Static-Pattern.html>
> 
> That is:
> 
> 	targets …: target-pattern: prereq-patterns …
> 		recipe
> 		…
> 
> This results in a huge database when make(1) runs, being 10x larger than
> what we would get with regular pattern rules.
> 
> However, recently I tried moving to pattern rules, since I thought the
> smaller database could improve performace, but the experience was
> surprising: the run time degraded by several orders of magnitude.
> I don't remember the exact numbers, but a full
> 	'make -R -j24 build-all lint check'
> (which is building almost everything that the build system can build)
> would take a few minutes with the huge database, and several hours with
> the size-optimized database using regular pattern rules.
> 
> I suspect it's because by having to guess for each file which rule to
> apply (and being non-recursive, it has the whole list of thousands of
> files, and around a hundred different patterns, so the combination is
> explosive) is too expensive.  By using pattern rules, make(1) doens't
> have to guess, and thus goes directly to the rule that applies to each
> file.
> 
> I think this might solve the issues you were seeing in your experiment.
> (Or it might not.)  :-)
> 
> Please let me know if this changes anything in your experiments!
> 
> 
> Have a lovely night!
> Alex
> 
> 
> -- 
> <https://www.alejandro-colomar.es>



-- 
<https://www.alejandro-colomar.es>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]