Re: Linux man-pages Makefile portability

Alejandro Colomar
2022-06-19T22:24:01Z

[-- Attachment #1.1: Type: text/plain, Size: 11326 bytes --]

Hi Ingo,

On 6/19/22 23:06, Ingo Schwarze wrote:
> Hi Alejandro,
>
> Alejandro Colomar wrote on Sun, Jun 19, 2022 at 07:52:33PM +0200:
>
>> Would you mind checking if the current Makefile works on your OpenBSD?
>> I think it depends on gmake(1),
>
> After doing
>
> $ git clone https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/
> $ cd man-pages

Thanks for trying!

>
> i have a hard time finding any targets in the ./Makefile that i could
> possibly run for testing...

Sorry, I need to document the targets.

>
> I'm certainly not going to run targets like "all" or "install".

The only interesting targets for users are 'dist', 'install', and
uninstall'. You could test the install target with DESTDIR=/tmp/somedir

You don't even need to be root to test that in /tmp.

<https://www.gnu.org/prep/standards/html_node/DESTDIR.html>

> I don't think testing the "dist" target makes much sense because
> nobody is going to build your distribution tarballs on OpenBSD.

Makes sense.

> I'm not interested in testing the "src" group of targets;
> i think we discussed in the past that i dislike that whole idea.

I remember.

>
> The html target could potentially be interesting, but
> https://savannah.nongnu.org/projects/man2html/
> and the repo linked to from there,
> https://cvs.savannah.nongnu.org/viewvc/man2html/
> gives me the impression this is abandonware that did not receive
> any maintenance in 19 years.

I used that because the archaic Makefile present in the tree when I
joined was using it (and I didn't know any other tools):

<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/Makefile?id=b32a6935592118ce4acfddf95e52fca8fbd3aded>

But that Makefile was clearly unused since no-one knows.

If you know a better tool, I could start using it. Maybe I could use
groff(1) directly, with grohtml(1).

> Besides, an OpenBSD port does not
> currently exist and i don't feel like creating one just for testing.
>
> The only remaining target i can readily see that might be fit for
> testing is "lint", which does seem quite interesting indeed.

Thanks :-)

>
> But look at this:
>
> $ make lint
> $
>
> It does absolutely nothing.
>
> Running "make -d g1 lint" to show the input graph tells me there
> is a rule
>
> lint:
> @:
>
> so i'm not surprised it does nothing.
>
> Next, i tried
>
> $ make lint-mandoc
> make: don't know how to make lint-mandoc
> Stop in /co/man-pages
>
> The target simply doesn't exist.

I guess it's due to the use of $(foreach ). I guess it's a GNU
extension and make(1POSIX) ignores that creating an empty string. Since
the Makefile uses a lot of functions[1], I guess it's not easy to make
it portable.

The lint part is mostly implemented between lines 432 and 486.

[1]: addsuffix, wildcard, foreach, filter, patsubst, sort, shell,
basename, notdir, info. Not sure how many of those are supported by
your make(1); maybe none?

>
> My impression is your Makefile is full of POSIX violations and
> utterly non-portable and gmake(1)-only. But that's probably OK.
> I mean, the purpose of your project is to document parts of Linux.
> So requiring a tool that almost all Linux users have readily
> available and that users of other systems can install it they
> want to does not seem unreasonable to me.

Yeah, I guess that it's fine, since the extensions are useful, and they
are virtually everywhere in Linux.

I was wondering if I could make it portable with little changes, but it
seems not, and not worth even trying.

> (It's a completely
> different story for a project like GNU troff that is about the
> opposite of Linux only and that basically all UNIX systems want
> to be able to use in production.)

Yes, although they could definitely simplify the build system a little
bit. It's one of the most complex builds I've ever seen.

>
> If i use GNU make instead of our native make, i do get output
> that might be potentially meaningful, even on OpenBSD-current:
>
> $ gmake lint-mandoc
> INSTALL tmp/lint/
> INSTALL tmp/lint/man0/
> LINT (mandoc) tmp/lint/man0/sysexits.h.0.lint.mandoc.touch
> INSTALL tmp/lint/man1/
> LINT (mandoc) tmp/lint/man1/getent.1.lint.mandoc.touch
> LINT (mandoc) tmp/lint/man1/iconv.1.lint.mandoc.touch
> LINT (mandoc) tmp/lint/man1/intro.1.lint.mandoc.touch
> LINT (mandoc) tmp/lint/man1/ldd.1.lint.mandoc.touch
> LINT (mandoc) tmp/lint/man1/locale.1.lint.mandoc.touch
> LINT (mandoc) tmp/lint/man1/localedef.1.lint.mandoc.touch
> LINT (mandoc) tmp/lint/man1/memusage.1.lint.mandoc.touch
> mandoc: man1/memusage.1:187:2: WARNING: empty block: UR
> gmake: *** [Makefile:468: tmp/lint/man1/memusage.1.lint.mandoc.touch] Error 2
>
> That makes sense to me given that i also get
>
> $ mandoc -T lint man1/memusage.1
> mandoc: man1/memusage.1:187:2: WARNING: empty block: UR
>
> when running the mandoc linter manually.
>
>> but I'd like to state that in the dependencies
>
> That sounds reasonable to me.
>
>> (which I need to write in a new INSTALL file).
>
> Yes, i think if you want people to test and use that Makefile,
> then documentating in some way what it is supposed to do and
> how it is supposed to be used would make sense. Sorry if i somehow
> missed the documentation, i looked at README and MAINTAINER_NOTES
> to no avail.

No, you didn't. I expected autocomplete to help, and targets to be
obvious in what they mean (as I followed GNU conventions as close as
possible so that they would be intuitive), but if your make doesn't
understand my Makefile, autocomplete is obviously not going to help.

Sorry :/

>
>
> By the way, this surprised me in the README file:
>
> Consider using multiple threads (at least 2) when installing
> these man pages, as the Makefile is optimized for multiple threads:
> "make -j install".
>
> I did not do measurements, but are you sure this is sane advice?
> For an installation procedure, i would have expected execution
> speed to be limited by disk I/O and i would have expected CPU
> consumption to be negligible. Did you do measurements to confirm
> the relevance of this advice?

Yes, I did:

[
$ sudo make uninstall >/dev/null
$ time sudo make install >/dev/null

real 0m7.722s
user 0m0.004s
sys 0m0.004s
$ sudo make uninstall >/dev/null
$ time sudo make -j2 install >/dev/null

real 0m4.634s
user 0m0.005s
sys 0m0.004s
$ sudo make uninstall >/dev/null
$ time sudo make -j install >/dev/null

real 0m3.170s
user 0m0.000s
sys 0m0.009s
$ touch man2/membarrier.2
$ time sudo make install >/dev/null

real 0m1.766s
user 0m0.005s
sys 0m0.004s
$ touch man2/membarrier.2
$ time sudo make -j2 install >/dev/null

real 0m1.735s
user 0m0.004s
sys 0m0.004s
$ touch man2/membarrier.2
$ time sudo make -j install >/dev/null

real 0m1.737s
user 0m0.001s
sys 0m0.008s
]

You can see that just for the $(shell ...) and all the text functions
that process the filenames to be installed it is taking almost 1.7 s,
even if it only needs to update a single page. That's a lot, but it has
it's advantages (generating the file list on-the-fly; no ./configure).

Then, the actual installation of the ~2.5k pages (most of them are link
pages), takes another 1.4 s in multi-process mode, and 6 s in
single-process mode (so at least 4.6 s that are not I/O). Maybe it's
make(1) that has a hard time traversing the tree... I don't know where
the bottleneck is, but it's clearly there.

>
> Finally, i don't think it is particularly good style that "make all"
> implies "make install", even if it is clearly documented, which it is.

I completely agree. That predates me, and I wanted to change it for a
long time, but I was worried that I might be breaking others' stuff.

But, since you called it, I feel supported to remove that behavior.

> That seems very unusual and not at all what people would expect.
> Usually, "make all" does specifically *not* modify the production
> parts of the system but only *builds* the software such that it can
> later be installed with a separate command. Arguably, having "make all"
> imply "make uninstall" is even worse. I don't think anybody would
> expect that running "make all" would attempt to delete stuff from
> their production system!

I guess that the Makefile before I touched is was so basic that you
could only uninstall everything or install everything, whoever wrote
that in the Makefile wanted to just type those two with just one command:

<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/Makefile?id=b32a6935592118ce4acfddf95e52fca8fbd3aded>

But that Makefile had one good thing: it was very readable, especially
compared to mine; but it was no more than a couple of sh(1) scripts
embedded in it.

>
>> Maybe, if it's not too hard, we can tweak the Makefile to make it
>> work with make(1POSIX).
>
> I see lots and lots of syntax in the Makefile where i don't even
> know what it means. I think if you want to make it POSIX compatible,
> that would likely amount to throwing it all away and starting over
> with a complete rewrite from scratch. It doesn't look like there is
> much you would be able to keep.

Nah, I think I'll keep it. Although if I'm shown a simpler POSIX
Makefile that achieves the same behavior, I'd consider it.

>
>
> There is yet another reason why i don't consider it a serious
> issue if the Makefile is for GNU make(1) only. It is not a rare
> occurence that i use your work - i have repeatedly looked at it
> when working on stuff in OpenBSD to find out how Linux defines
> some features for comparison.

Thanks! I do check yours too :)

BTW, did you check the changes to queue.3? I guess you could improve
yours in a similar manner.

<https://linux-man-pages.blogspot.com/2020/11/man-pages-509-is-released.html>

Also, if you have been following the addition of pages about types, and
would like to comment, you'll be welcome!

<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=178eaf37e2e971cae88bd4d3f124ede0afbb1015>

> But i never even noticed there
> is a Makefile in your distribution.

Heh!

> Commands like
>
> $ man -M /co/man-pages open
>
> work perfectly fine on my system to view the Linux open(2) manual,
> nicely formatted, with no need for installation or a Makefile.
> Even when i put up a copy at
>
> https://man.bsd.lv/Linux-5.13/open

How do you generate your HTML pages? mandoc(1)? They are nice.

>
> just copying the eight man* directories and running the makewhatis(8)
> command distributed with mandoc is completely sufficient, no further
> steps are needed for installation, not even on the production web
> server.
>
> It seems likely to me that other non-Linux users interested in your
> work don't need the Makefile either.

Makes sense. I'm happy that I didn't break anything by making heavy use
of GNU extensions.

Cheers,
Alex

--
Alejandro Colomar
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]