[RFT PATCH -perfbook] Enable parallel runs of pdflatex

[RFT PATCH -perfbook] Enable parallel runs of pdflatex

[Akira Yokosawa]

Parallel build of PDF figures and .fcv snippets by "make -jN" worked
when a single PDF target was given to "make", such as
"make neatfreak; make -j4 eb".
On the other hand, "make -j3 2c 1c eb" ended up in various errors
during parallel runs of pdflatex.

It turns out that the cause of the error is the \include{} commands
in appendix/appendix.tex and perfbook-lt.tex.
\include{} switches to its own .aux file, in this case,
appendix/questions.aux, appendix/toyrcu.aux, and so on.
As their names are common to different PDF targets, those .aux files
can be overwritten/corrupted by parallel runs of pdflatex.

\include{} implies \cleardoublepage both in front of and next to it.
In perfbook, included LaTeX sources have \chapter{} commands and
the implied page breaks are redundant.

By replacing \include{} with \input{}, parallel runs of pdflatex can
be enabled.

There were a couple of minor issues in Makefile WRT parallel runs of
runfirstlatex.sh.

When some of perfbook-xxx.tex files already existed, runfirstlatex.sh
for other main perfbook-yyy.tex could be invoked prematurely before
that .tex file was ready.

Fix it by adding stricter dependencies of perfbook-xxx.aux on
perfbook-xxx.tex.
Also remove some redundant dependencies around here.

Note: LATEXSOURCES contains perfbook-lt.tex

Signed-off-by: Akira Yokosawa <akiyks@gmail.com>
---
Hi Paul,

For quite a while, I was wondering why "make -j3 2c 1c eb"
didn't work as expected.

I think I have managed to get it work.

But changes in Makefile need extra testing.  Especially,
those involving parallelizing.

So I'd like you to give this a fair amount of testing before
pushing it.

        Thanks, Akira
--
 Makefile              | 6 +++---
 appendix/appendix.tex | 8 ++++----
 perfbook-lt.tex       | 4 ++--
 3 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/Makefile b/Makefile
index 6ee849fd..9946be3a 100644
--- a/Makefile
+++ b/Makefile
@@ -222,14 +222,14 @@ $(PDFTARGETS): %.pdf: %.tex %.bbl
 $(PDFTARGETS:.pdf=.bbl): %.bbl: %.aux $(BIBSOURCES)
 	bibtex $(basename $@)
 
-$(PDFTARGETS:.pdf=.aux): $(LATEXGENERATED) $(LATEXSOURCES) $(LST_SOURCES)
+$(PDFTARGETS:.pdf=.aux): %.aux: %.tex $(LATEXGENERATED) $(LST_SOURCES)
 ifeq ($(NEWTXTEXT),)
 	$(error Font package 'newtx' not found. See #9 in FAQ-BUILD.txt)
 endif
 	sh utilities/runfirstlatex.sh $(basename $@)
 
-autodate.tex: perfbook-lt.tex $(LATEXSOURCES) $(BIBSOURCES) \
-    $(PDFTARGETS_OF_EPS) $(PDFTARGETS_OF_SVG) $(FCVSNIPPETS) $(FCVSNIPPETS_VIA_LTMS) \
+autodate.tex: $(LATEXSOURCES) $(BIBSOURCES) $(PDFTARGETS_OF_EPS) \
+    $(PDFTARGETS_OF_SVG) $(FCVSNIPPETS) $(FCVSNIPPETS_VIA_LTMS) \
     $(GITREFSTAGS) utilities/autodate.sh
 	sh utilities/autodate.sh
 
diff --git a/appendix/appendix.tex b/appendix/appendix.tex
index f2640157..edcfe7ef 100644
--- a/appendix/appendix.tex
+++ b/appendix/appendix.tex
@@ -2,9 +2,9 @@
 % mainfile: ../perfbook.tex
 % SPDX-License-Identifier: CC-BY-SA-3.0
 
-\include{appendix/questions/questions}
-\include{appendix/toyrcu/toyrcu}
-\include{appendix/whymb/whymemorybarriers}
-\include{appendix/styleguide/styleguide}
+\input{appendix/questions/questions}
+\input{appendix/toyrcu/toyrcu}
+\input{appendix/whymb/whymemorybarriers}
+\input{appendix/styleguide/styleguide}
 \renewcommand{\bottomtitlespace}{.08\textheight}
 \QuickQuizAnswers
diff --git a/perfbook-lt.tex b/perfbook-lt.tex
index 085ce55d..d70dde66 100644
--- a/perfbook-lt.tex
+++ b/perfbook-lt.tex
@@ -667,12 +667,12 @@
 
 % Credits
 \setcounter{secnumdepth}{-1} % surpress section numbering in backmatter
-\include{ack}
+\input{ack}
 
 % Index if enabled
 \IfIndexOn{
 \IfIndexHier{
-\include{indexsee}
+\input{indexsee}
 }{}
 \phantomsection
 \IfEbookSize{

base-commit: f24ccfbd345eab8a70d6c08e71c84f18df46a3a1
-- 
2.17.1

Re: [RFT PATCH -perfbook] Enable parallel runs of pdflatex

[Paul E. McKenney]

On Thu, Jan 27, 2022 at 10:51:25AM +0900, Akira Yokosawa wrote:
> Parallel build of PDF figures and .fcv snippets by "make -jN" worked
> when a single PDF target was given to "make", such as
> "make neatfreak; make -j4 eb".
> On the other hand, "make -j3 2c 1c eb" ended up in various errors
> during parallel runs of pdflatex.
> 
> It turns out that the cause of the error is the \include{} commands
> in appendix/appendix.tex and perfbook-lt.tex.
> \include{} switches to its own .aux file, in this case,
> appendix/questions.aux, appendix/toyrcu.aux, and so on.
> As their names are common to different PDF targets, those .aux files
> can be overwritten/corrupted by parallel runs of pdflatex.
> 
> \include{} implies \cleardoublepage both in front of and next to it.
> In perfbook, included LaTeX sources have \chapter{} commands and
> the implied page breaks are redundant.
> 
> By replacing \include{} with \input{}, parallel runs of pdflatex can
> be enabled.
> 
> There were a couple of minor issues in Makefile WRT parallel runs of
> runfirstlatex.sh.
> 
> When some of perfbook-xxx.tex files already existed, runfirstlatex.sh
> for other main perfbook-yyy.tex could be invoked prematurely before
> that .tex file was ready.
> 
> Fix it by adding stricter dependencies of perfbook-xxx.aux on
> perfbook-xxx.tex.
> Also remove some redundant dependencies around here.
> 
> Note: LATEXSOURCES contains perfbook-lt.tex
> 
> Signed-off-by: Akira Yokosawa <akiyks@gmail.com>
> ---
> Hi Paul,
> 
> For quite a while, I was wondering why "make -j3 2c 1c eb"
> didn't work as expected.
> 
> I think I have managed to get it work.
> 
> But changes in Makefile need extra testing.  Especially,
> those involving parallelizing.
> 
> So I'd like you to give this a fair amount of testing before
> pushing it.

OK, I am applying it to an experimental repo.  Let's give it a go:

	$ time make -j32 2c 1c eb a4 nq sf sfnq
	[ . . . ]
	'perfbook-sf.pdf' is ready.
	'perfbook-1c.pdf' is ready.
	'perfbook.pdf' is ready.
	'perfbook-sfnq.pdf' is ready.
	'perfbook-a4.pdf' is ready.
	'perfbook-eb.pdf' is ready.
	'perfbook-nq.pdf' is ready.

	real    2m31.412s
	user    16m45.778s
	sys     1m48.385s

Not bad, actually!  Might be worth redoing my release scripts to take
advantage of this.  ;-)

I (very) quickly glanced at the PDFs, and they look OK.

I then did a "make clean" and:

	$ time make
	[ . . . ]
	'perfbook.pdf' is ready.
	utilities/punctcheck.sh
	utilities/cleverefcheck.sh

	real    1m47.823s
	user    1m45.715s
	sys     0m1.226s

Another "make clean" and:

	$ time make -j32
	[ . . . ]
	'perfbook.pdf' is ready.
	utilities/punctcheck.sh
	utilities/cleverefcheck.sh

	real    1m49.484s
	user    1m47.206s
	sys     0m1.437s

So, as one would expect, the sweet spot for -j is when rebuilding
all the images from scratch or when building multiple PDFs.

Nice!

Huh.  It would be good to have a way of diff-ing PDF files.  Then
a single-threaded-generated PDF could be automatically compared to
its counterpart from a concurrent build.  But the search engines
were not kind to me.

Thoughts?

							Thanx, Paul

>         Thanks, Akira
> --
>  Makefile              | 6 +++---
>  appendix/appendix.tex | 8 ++++----
>  perfbook-lt.tex       | 4 ++--
>  3 files changed, 9 insertions(+), 9 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 6ee849fd..9946be3a 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -222,14 +222,14 @@ $(PDFTARGETS): %.pdf: %.tex %.bbl
>  $(PDFTARGETS:.pdf=.bbl): %.bbl: %.aux $(BIBSOURCES)
>  	bibtex $(basename $@)
>  
> -$(PDFTARGETS:.pdf=.aux): $(LATEXGENERATED) $(LATEXSOURCES) $(LST_SOURCES)
> +$(PDFTARGETS:.pdf=.aux): %.aux: %.tex $(LATEXGENERATED) $(LST_SOURCES)
>  ifeq ($(NEWTXTEXT),)
>  	$(error Font package 'newtx' not found. See #9 in FAQ-BUILD.txt)
>  endif
>  	sh utilities/runfirstlatex.sh $(basename $@)
>  
> -autodate.tex: perfbook-lt.tex $(LATEXSOURCES) $(BIBSOURCES) \
> -    $(PDFTARGETS_OF_EPS) $(PDFTARGETS_OF_SVG) $(FCVSNIPPETS) $(FCVSNIPPETS_VIA_LTMS) \
> +autodate.tex: $(LATEXSOURCES) $(BIBSOURCES) $(PDFTARGETS_OF_EPS) \
> +    $(PDFTARGETS_OF_SVG) $(FCVSNIPPETS) $(FCVSNIPPETS_VIA_LTMS) \
>      $(GITREFSTAGS) utilities/autodate.sh
>  	sh utilities/autodate.sh
>  
> diff --git a/appendix/appendix.tex b/appendix/appendix.tex
> index f2640157..edcfe7ef 100644
> --- a/appendix/appendix.tex
> +++ b/appendix/appendix.tex
> @@ -2,9 +2,9 @@
>  % mainfile: ../perfbook.tex
>  % SPDX-License-Identifier: CC-BY-SA-3.0
>  
> -\include{appendix/questions/questions}
> -\include{appendix/toyrcu/toyrcu}
> -\include{appendix/whymb/whymemorybarriers}
> -\include{appendix/styleguide/styleguide}
> +\input{appendix/questions/questions}
> +\input{appendix/toyrcu/toyrcu}
> +\input{appendix/whymb/whymemorybarriers}
> +\input{appendix/styleguide/styleguide}
>  \renewcommand{\bottomtitlespace}{.08\textheight}
>  \QuickQuizAnswers
> diff --git a/perfbook-lt.tex b/perfbook-lt.tex
> index 085ce55d..d70dde66 100644
> --- a/perfbook-lt.tex
> +++ b/perfbook-lt.tex
> @@ -667,12 +667,12 @@
>  
>  % Credits
>  \setcounter{secnumdepth}{-1} % surpress section numbering in backmatter
> -\include{ack}
> +\input{ack}
>  
>  % Index if enabled
>  \IfIndexOn{
>  \IfIndexHier{
> -\include{indexsee}
> +\input{indexsee}
>  }{}
>  \phantomsection
>  \IfEbookSize{
> 
> base-commit: f24ccfbd345eab8a70d6c08e71c84f18df46a3a1
> -- 
> 2.17.1
> 

Re: [RFT PATCH -perfbook] Enable parallel runs of pdflatex

[Akira Yokosawa]

On Wed, 26 Jan 2022 20:45:30 -0800,
Paul E. McKenney wrote:
> On Thu, Jan 27, 2022 at 10:51:25AM +0900, Akira Yokosawa wrote:
>> Parallel build of PDF figures and .fcv snippets by "make -jN" worked
>> when a single PDF target was given to "make", such as
>> "make neatfreak; make -j4 eb".
>> On the other hand, "make -j3 2c 1c eb" ended up in various errors
>> during parallel runs of pdflatex.
>>
>> It turns out that the cause of the error is the \include{} commands
>> in appendix/appendix.tex and perfbook-lt.tex.
>> \include{} switches to its own .aux file, in this case,
>> appendix/questions.aux, appendix/toyrcu.aux, and so on.
>> As their names are common to different PDF targets, those .aux files
>> can be overwritten/corrupted by parallel runs of pdflatex.
>>
>> \include{} implies \cleardoublepage both in front of and next to it.
>> In perfbook, included LaTeX sources have \chapter{} commands and
>> the implied page breaks are redundant.
>>
>> By replacing \include{} with \input{}, parallel runs of pdflatex can
>> be enabled.
>>
>> There were a couple of minor issues in Makefile WRT parallel runs of
>> runfirstlatex.sh.
>>
>> When some of perfbook-xxx.tex files already existed, runfirstlatex.sh
>> for other main perfbook-yyy.tex could be invoked prematurely before
>> that .tex file was ready.
>>
>> Fix it by adding stricter dependencies of perfbook-xxx.aux on
>> perfbook-xxx.tex.
>> Also remove some redundant dependencies around here.
>>
>> Note: LATEXSOURCES contains perfbook-lt.tex
>>
>> Signed-off-by: Akira Yokosawa <akiyks@gmail.com>
>> ---
>> Hi Paul,
>>
>> For quite a while, I was wondering why "make -j3 2c 1c eb"
>> didn't work as expected.
>>
>> I think I have managed to get it work.
>>
>> But changes in Makefile need extra testing.  Especially,
>> those involving parallelizing.
>>
>> So I'd like you to give this a fair amount of testing before
>> pushing it.
> 
> OK, I am applying it to an experimental repo.  Let's give it a go:
> 
> 	$ time make -j32 2c 1c eb a4 nq sf sfnq
> 	[ . . . ]
> 	'perfbook-sf.pdf' is ready.
> 	'perfbook-1c.pdf' is ready.
> 	'perfbook.pdf' is ready.
> 	'perfbook-sfnq.pdf' is ready.
> 	'perfbook-a4.pdf' is ready.
> 	'perfbook-eb.pdf' is ready.
> 	'perfbook-nq.pdf' is ready.
> 
> 	real    2m31.412s
> 	user    16m45.778s
> 	sys     1m48.385s
> 
> Not bad, actually!  Might be worth redoing my release scripts to take
> advantage of this.  ;-)

Nice!

> 
> I (very) quickly glanced at the PDFs, and they look OK.
> 
> I then did a "make clean" and:
> 
> 	$ time make
> 	[ . . . ]
> 	'perfbook.pdf' is ready.
> 	utilities/punctcheck.sh
> 	utilities/cleverefcheck.sh
> 
> 	real    1m47.823s
> 	user    1m45.715s
> 	sys     0m1.226s
> 
> Another "make clean" and:
> 
> 	$ time make -j32
> 	[ . . . ]
> 	'perfbook.pdf' is ready.
> 	utilities/punctcheck.sh
> 	utilities/cleverefcheck.sh
> 
> 	real    1m49.484s
> 	user    1m47.206s
> 	sys     0m1.437s
> 
> So, as one would expect, the sweet spot for -j is when rebuilding
> all the images from scratch or when building multiple PDFs.
> 
> Nice!
> 
> Huh.  It would be good to have a way of diff-ing PDF files.  Then
> a single-threaded-generated PDF could be automatically compared to
> its counterpart from a concurrent build.  But the search engines
> were not kind to me.
> 
> Thoughts?

How about comparing .log files of two builds?

In my trial, "diff -u perfbook-single.log perfbook-multi.log"
yields only a hunk of build time difference as shown below:

    --- perfbook-single.log	2022-01-27 14:30:01.659959433 +0900

    +++ perfbook-multi.log	2022-01-27 14:33:57.556196036 +0900

    @@ -1,4 +1,4 @@

    -This is pdfTeX, Version 3.141592653-2.6-1.40.23 (TeX Live 2021) (preloaded format=pdflatex 2022.1.24)  27 JAN 2022 14:28

    +This is pdfTeX, Version 3.141592653-2.6-1.40.23 (TeX Live 2021) (preloaded format=pdflatex 2022.1.24)  27 JAN 2022 14:32

     entering extended mode

      restricted \write18 enabled.

      %&-line parsing enabled.


Not a direct comparison, but the log has a summary near the
end:

    Here is how much of TeX's memory you used:
     74049 strings out of 479599
     1597759 string characters out of 5876039
     1950680 words of memory out of 5000000
     80683 multiletter control sequences out of 15000+600000
     692091 words of font info for 721 fonts, out of 8000000 for 9000
     1816 hyphenation exceptions out of 8191
     105i,19n,133p,1187b,1660s stack positions out of 5000i,500n,10000p,200000b,80000s

     [long list of referenced font files]

    Output written on perfbook.pdf (638 pages, 8031215 bytes).
    PDF statistics:
     25510 PDF objects out of 26593 (max. 8388607)
     24061 compressed objects within 241 object streams
     7119 named destinations out of 7423 (max. 500000)
     166134 words of extra memory for PDF output out of 184870 (max. 10000000)

I think any visible difference in two PDFs is likely to be
reflected in this summary.

Indirect hints they may be, at least they provide partial
assurance.

        Thanks, Akira

> 
> 							Thanx, Paul
> 
>>         Thanks, Akira
>> --
>>  Makefile              | 6 +++---
>>  appendix/appendix.tex | 8 ++++----
>>  perfbook-lt.tex       | 4 ++--
>>  3 files changed, 9 insertions(+), 9 deletions(-)
>>
[...]