Linux man-pages Makefile portability

Linux man-pages Makefile portability

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 455 bytes --]

Hi Ingo,

Would you mind checking if the current Makefile works on your OpenBSD?
I think it depends on gmake(1), but I'd like to state that in the 
dependencies (which I need to write in a new INSTALL file).  Maybe, if 
it's not too hard, we can tweak the Makefile to make it work with 
make(1POSIX).

<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/>

Thanks,

Alex

-- 
Alejandro Colomar
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Linux man-pages Makefile portability

[Ingo Schwarze]

Hi Alejandro,

Alejandro Colomar wrote on Sun, Jun 19, 2022 at 07:52:33PM +0200:

> Would you mind checking if the current Makefile works on your OpenBSD?
> I think it depends on gmake(1),

After doing

   $ git clone https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/
   $ cd man-pages

i have a hard time finding any targets in the ./Makefile that i could
possibly run for testing...

I'm certainly not going to run targets like "all" or "install".
I don't think testing the "dist" target makes much sense because
nobody is going to build your distribution tarballs on OpenBSD.
I'm not interested in testing the "src" group of targets;
i think we discussed in the past that i dislike that whole idea.

The html target could potentially be interesting, but
  https://savannah.nongnu.org/projects/man2html/
and the repo linked to from there,
  https://cvs.savannah.nongnu.org/viewvc/man2html/
gives me the impression this is abandonware that did not receive
any maintenance in 19 years.  Besides, an OpenBSD port does not
currently exist and i don't feel like creating one just for testing.

The only remaining target i can readily see that might be fit for
testing is "lint", which does seem quite interesting indeed.

But look at this:

   $ make lint
   $

It does absolutely nothing.

Running "make -d g1 lint" to show the input graph tells me there
is a rule

  lint:
  	@:

so i'm not surprised it does nothing.

Next, i tried

   $ make lint-mandoc
  make: don't know how to make lint-mandoc
  Stop in /co/man-pages

The target simply doesn't exist.

My impression is your Makefile is full of POSIX violations and
utterly non-portable and gmake(1)-only.  But that's probably OK.
I mean, the purpose of your project is to document parts of Linux.
So requiring a tool that almost all Linux users have readily
available and that users of other systems can install it they
want to does not seem unreasonable to me.  (It's a completely
different story for a project like GNU troff that is about the
opposite of Linux only and that basically all UNIX systems want
to be able to use in production.)

If i use GNU make instead of our native make, i do get output
that might be potentially meaningful, even on OpenBSD-current:

   $ gmake lint-mandoc
  INSTALL tmp/lint/
  INSTALL tmp/lint/man0/
  LINT (mandoc)   tmp/lint/man0/sysexits.h.0.lint.mandoc.touch
  INSTALL tmp/lint/man1/
  LINT (mandoc)   tmp/lint/man1/getent.1.lint.mandoc.touch
  LINT (mandoc)   tmp/lint/man1/iconv.1.lint.mandoc.touch
  LINT (mandoc)   tmp/lint/man1/intro.1.lint.mandoc.touch
  LINT (mandoc)   tmp/lint/man1/ldd.1.lint.mandoc.touch
  LINT (mandoc)   tmp/lint/man1/locale.1.lint.mandoc.touch
  LINT (mandoc)   tmp/lint/man1/localedef.1.lint.mandoc.touch
  LINT (mandoc)   tmp/lint/man1/memusage.1.lint.mandoc.touch
  mandoc: man1/memusage.1:187:2: WARNING: empty block: UR
  gmake: *** [Makefile:468: tmp/lint/man1/memusage.1.lint.mandoc.touch] Error 2

That makes sense to me given that i also get

   $ mandoc -T lint man1/memusage.1                               
  mandoc: man1/memusage.1:187:2: WARNING: empty block: UR

when running the mandoc linter manually.

> but I'd like to state that in the dependencies

That sounds reasonable to me.

> (which I need to write in a new INSTALL file).

Yes, i think if you want people to test and use that Makefile,
then documentating in some way what it is supposed to do and
how it is supposed to be used would make sense.  Sorry if i somehow
missed the documentation, i looked at README and MAINTAINER_NOTES
to no avail.


By the way, this surprised me in the README file:

  Consider using multiple threads (at least 2) when installing
  these man pages, as the Makefile is optimized for multiple threads:
  "make -j install".

I did not do measurements, but are you sure this is sane advice?
For an installation procedure, i would have expected execution
speed to be limited by disk I/O and i would have expected CPU
consumption to be negligible.  Did you do measurements to confirm
the relevance of this advice?

Finally, i don't think it is particularly good style that "make all"
implies "make install", even if it is clearly documented, which it is.
That seems very unusual and not at all what people would expect.
Usually, "make all" does specifically *not* modify the production
parts of the system but only *builds* the software such that it can
later be installed with a separate command.  Arguably, having "make all"
imply "make uninstall" is even worse.  I don't think anybody would
expect that running "make all" would attempt to delete stuff from
their production system!

> Maybe, if it's not too hard, we can tweak the Makefile to make it
> work with make(1POSIX).

I see lots and lots of syntax in the Makefile where i don't even
know what it means.  I think if you want to make it POSIX compatible,
that would likely amount to throwing it all away and starting over
with a complete rewrite from scratch.  It doesn't look like there is
much you would be able to keep.


There is yet another reason why i don't consider it a serious
issue if the Makefile is for GNU make(1) only.  It is not a rare
occurence that i use your work - i have repeatedly looked at it
when working on stuff in OpenBSD to find out how Linux defines
some features for comparison.  But i never even noticed there
is a Makefile in your distribution.  Commands like

   $ man -M /co/man-pages open

work perfectly fine on my system to view the Linux open(2) manual,
nicely formatted, with no need for installation or a Makefile.
Even when i put up a copy at

  https://man.bsd.lv/Linux-5.13/open

just copying the eight man* directories and running the makewhatis(8)
command distributed with mandoc is completely sufficient, no further
steps are needed for installation, not even on the production web
server.

It seems likely to me that other non-Linux users interested in your
work don't need the Makefile either.

Yours,
  Ingo

Re: Linux man-pages Makefile portability

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 11326 bytes --]

Hi Ingo,

On 6/19/22 23:06, Ingo Schwarze wrote:
> Hi Alejandro,
> 
> Alejandro Colomar wrote on Sun, Jun 19, 2022 at 07:52:33PM +0200:
> 
>> Would you mind checking if the current Makefile works on your OpenBSD?
>> I think it depends on gmake(1),
> 
> After doing
> 
>     $ git clone https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/
>     $ cd man-pages

Thanks for trying!

> 
> i have a hard time finding any targets in the ./Makefile that i could
> possibly run for testing...

Sorry, I need to document the targets.

> 
> I'm certainly not going to run targets like "all" or "install".

The only interesting targets for users are 'dist', 'install', and 
uninstall'.  You could test the install target with DESTDIR=/tmp/somedir

You don't even need to be root to test that in /tmp.

<https://www.gnu.org/prep/standards/html_node/DESTDIR.html>


> I don't think testing the "dist" target makes much sense because
> nobody is going to build your distribution tarballs on OpenBSD.

Makes sense.

> I'm not interested in testing the "src" group of targets;
> i think we discussed in the past that i dislike that whole idea.

I remember.

> 
> The html target could potentially be interesting, but
>    https://savannah.nongnu.org/projects/man2html/
> and the repo linked to from there,
>    https://cvs.savannah.nongnu.org/viewvc/man2html/
> gives me the impression this is abandonware that did not receive
> any maintenance in 19 years.

I used that because the archaic Makefile present in the tree when I 
joined was using it (and I didn't know any other tools):

<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/Makefile?id=b32a6935592118ce4acfddf95e52fca8fbd3aded>

But that Makefile was clearly unused since no-one knows.

If you know a better tool, I could start using it.  Maybe I could use 
groff(1) directly, with grohtml(1).

>  Besides, an OpenBSD port does not
> currently exist and i don't feel like creating one just for testing.
> 
> The only remaining target i can readily see that might be fit for
> testing is "lint", which does seem quite interesting indeed.

Thanks :-)

> 
> But look at this:
> 
>     $ make lint
>     $
> 
> It does absolutely nothing.
> 
> Running "make -d g1 lint" to show the input graph tells me there
> is a rule
> 
>    lint:
>    	@:
> 
> so i'm not surprised it does nothing.
> 
> Next, i tried
> 
>     $ make lint-mandoc
>    make: don't know how to make lint-mandoc
>    Stop in /co/man-pages
> 
> The target simply doesn't exist.

I guess it's due to the use of $(foreach ).  I guess it's a GNU 
extension and make(1POSIX) ignores that creating an empty string.  Since 
the Makefile uses a lot of functions[1], I guess it's not easy to make 
it portable.

The lint part is mostly implemented between lines 432 and 486.

[1]:  addsuffix, wildcard, foreach, filter, patsubst, sort, shell, 
basename, notdir, info.  Not sure how many of those are supported by 
your make(1); maybe none?

> 
> My impression is your Makefile is full of POSIX violations and
> utterly non-portable and gmake(1)-only.  But that's probably OK.
> I mean, the purpose of your project is to document parts of Linux.
> So requiring a tool that almost all Linux users have readily
> available and that users of other systems can install it they
> want to does not seem unreasonable to me.

Yeah, I guess that it's fine, since the extensions are useful, and they 
are virtually everywhere in Linux.

I was wondering if I could make it portable with little changes, but it 
seems not, and not worth even trying.

>  (It's a completely
> different story for a project like GNU troff that is about the
> opposite of Linux only and that basically all UNIX systems want
> to be able to use in production.)

Yes, although they could definitely simplify the build system a little 
bit.  It's one of the most complex builds I've ever seen.

> 
> If i use GNU make instead of our native make, i do get output
> that might be potentially meaningful, even on OpenBSD-current:
> 
>     $ gmake lint-mandoc
>    INSTALL tmp/lint/
>    INSTALL tmp/lint/man0/
>    LINT (mandoc)   tmp/lint/man0/sysexits.h.0.lint.mandoc.touch
>    INSTALL tmp/lint/man1/
>    LINT (mandoc)   tmp/lint/man1/getent.1.lint.mandoc.touch
>    LINT (mandoc)   tmp/lint/man1/iconv.1.lint.mandoc.touch
>    LINT (mandoc)   tmp/lint/man1/intro.1.lint.mandoc.touch
>    LINT (mandoc)   tmp/lint/man1/ldd.1.lint.mandoc.touch
>    LINT (mandoc)   tmp/lint/man1/locale.1.lint.mandoc.touch
>    LINT (mandoc)   tmp/lint/man1/localedef.1.lint.mandoc.touch
>    LINT (mandoc)   tmp/lint/man1/memusage.1.lint.mandoc.touch
>    mandoc: man1/memusage.1:187:2: WARNING: empty block: UR
>    gmake: *** [Makefile:468: tmp/lint/man1/memusage.1.lint.mandoc.touch] Error 2
> 
> That makes sense to me given that i also get
> 
>     $ mandoc -T lint man1/memusage.1
>    mandoc: man1/memusage.1:187:2: WARNING: empty block: UR
> 
> when running the mandoc linter manually.
> 
>> but I'd like to state that in the dependencies
> 
> That sounds reasonable to me.
> 
>> (which I need to write in a new INSTALL file).
> 
> Yes, i think if you want people to test and use that Makefile,
> then documentating in some way what it is supposed to do and
> how it is supposed to be used would make sense.  Sorry if i somehow
> missed the documentation, i looked at README and MAINTAINER_NOTES
> to no avail.

No, you didn't.  I expected autocomplete to help, and targets to be 
obvious in what they mean (as I followed GNU conventions as close as 
possible so that they would be intuitive), but if your make doesn't 
understand my Makefile, autocomplete is obviously not going to help.

Sorry :/

> 
> 
> By the way, this surprised me in the README file:
> 
>    Consider using multiple threads (at least 2) when installing
>    these man pages, as the Makefile is optimized for multiple threads:
>    "make -j install".
> 
> I did not do measurements, but are you sure this is sane advice?
> For an installation procedure, i would have expected execution
> speed to be limited by disk I/O and i would have expected CPU
> consumption to be negligible.  Did you do measurements to confirm
> the relevance of this advice?

Yes, I did:

[
$ sudo make uninstall >/dev/null
$ time sudo make install >/dev/null

real	0m7.722s
user	0m0.004s
sys	0m0.004s
$ sudo make uninstall >/dev/null
$ time sudo make -j2 install >/dev/null

real	0m4.634s
user	0m0.005s
sys	0m0.004s
$ sudo make uninstall >/dev/null
$ time sudo make -j install >/dev/null

real	0m3.170s
user	0m0.000s
sys	0m0.009s
$ touch man2/membarrier.2
$ time sudo make install >/dev/null

real	0m1.766s
user	0m0.005s
sys	0m0.004s
$ touch man2/membarrier.2
$ time sudo make -j2 install >/dev/null

real	0m1.735s
user	0m0.004s
sys	0m0.004s
$ touch man2/membarrier.2
$ time sudo make -j install >/dev/null

real	0m1.737s
user	0m0.001s
sys	0m0.008s
]

You can see that just for the $(shell ...) and all the text functions 
that process the filenames to be installed it is taking almost 1.7 s, 
even if it only needs to update a single page.  That's a lot, but it has 
it's advantages (generating the file list on-the-fly; no ./configure).

Then, the actual installation of the ~2.5k pages (most of them are link 
pages), takes another 1.4 s in multi-process mode, and 6 s in 
single-process mode (so at least 4.6 s that are not I/O).  Maybe it's 
make(1) that has a hard time traversing the tree... I don't know where 
the bottleneck is, but it's clearly there.

> 
> Finally, i don't think it is particularly good style that "make all"
> implies "make install", even if it is clearly documented, which it is.

I completely agree.  That predates me, and I wanted to change it for a 
long time, but I was worried that I might be breaking others' stuff.

But, since you called it, I feel supported to remove that behavior.

> That seems very unusual and not at all what people would expect.
> Usually, "make all" does specifically *not* modify the production
> parts of the system but only *builds* the software such that it can
> later be installed with a separate command.  Arguably, having "make all"
> imply "make uninstall" is even worse.  I don't think anybody would
> expect that running "make all" would attempt to delete stuff from
> their production system!

I guess that the Makefile before I touched is was so basic that you 
could only uninstall everything or install everything, whoever wrote 
that in the Makefile wanted to just type those two with just one command:

<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/Makefile?id=b32a6935592118ce4acfddf95e52fca8fbd3aded>

But that Makefile had one good thing: it was very readable, especially 
compared to mine; but it was no more than a couple of sh(1) scripts 
embedded in it.

> 
>> Maybe, if it's not too hard, we can tweak the Makefile to make it
>> work with make(1POSIX).
> 
> I see lots and lots of syntax in the Makefile where i don't even
> know what it means.  I think if you want to make it POSIX compatible,
> that would likely amount to throwing it all away and starting over
> with a complete rewrite from scratch.  It doesn't look like there is
> much you would be able to keep.

Nah, I think I'll keep it.  Although if I'm shown a simpler POSIX 
Makefile that achieves the same behavior, I'd consider it.

> 
> 
> There is yet another reason why i don't consider it a serious
> issue if the Makefile is for GNU make(1) only.  It is not a rare
> occurence that i use your work - i have repeatedly looked at it
> when working on stuff in OpenBSD to find out how Linux defines
> some features for comparison.

Thanks!  I do check yours too :)

BTW, did you check the changes to queue.3?  I guess you could improve 
yours in a similar manner.

<https://linux-man-pages.blogspot.com/2020/11/man-pages-509-is-released.html>

Also, if you have been following the addition of pages about types, and 
would like to comment, you'll be welcome!

<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=178eaf37e2e971cae88bd4d3f124ede0afbb1015>

>  But i never even noticed there
> is a Makefile in your distribution.

Heh!

>  Commands like
> 
>     $ man -M /co/man-pages open
> 
> work perfectly fine on my system to view the Linux open(2) manual,
> nicely formatted, with no need for installation or a Makefile.
> Even when i put up a copy at
> 
>    https://man.bsd.lv/Linux-5.13/open

How do you generate your HTML pages?  mandoc(1)?  They are nice.

> 
> just copying the eight man* directories and running the makewhatis(8)
> command distributed with mandoc is completely sufficient, no further
> steps are needed for installation, not even on the production web
> server.
> 
> It seems likely to me that other non-Linux users interested in your
> work don't need the Makefile either.

Makes sense.  I'm happy that I didn't break anything by making heavy use 
of GNU extensions.


Cheers,
Alex

-- 
Alejandro Colomar
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Linux man-pages Makefile portability

[Ingo Schwarze]

Hi Alejandro,

Alejandro Colomar wrote on Mon, Jun 20, 2022 at 12:23:45AM +0200:
> On 6/19/22 23:06, Ingo Schwarze wrote:

>> The html target could potentially be interesting, but
>>    https://savannah.nongnu.org/projects/man2html/
>> and the repo linked to from there,
>>    https://cvs.savannah.nongnu.org/viewvc/man2html/
>> gives me the impression this is abandonware that did not receive
>> any maintenance in 19 years.

> I used that because the archaic Makefile present in the tree when I 
> joined was using it (and I didn't know any other tools):
> 
> <https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/Makefile?id=b32a6935592118ce4acfddf95e52fca8fbd3aded>

I see, now it makes sense to me.

I'm relieved i won't have to dig into man2html.  :-)

> But that Makefile was clearly unused since no-one knows.

I'm not saying "noone knows"; some packaging tools on some Linux
distributions might very well rely on the Makefile - i don't know.
All i'm saying is i don't readily see a reason why people *not*
running Linux might need the Makefile...

> If you know a better tool, I could start using it.  Maybe I could use 
> groff(1) directly, with grohtml(1).

I don't think using groff to generate HTML output from manual pages
is a good idea.  It generates low-quality HTML code, and it is
impossible to fix because of the basic concept how groff works.

You see, both mdoc(7) and modern HTML are semantic markup languages.
Groff, however, does the transformation in a two-step process.
In the first step, it throws away *all* the semantic information,
retaining only *physical* formatting cues.  And then in the
second step, it tries to generate HTML from those physical formatting
instructions.  That cannot possibly work.  At best it might generate
early-1990ies-style HTML 1 or HTML 2 code with no support for CSS
whatsoever.
That's an unavoidable consequence of the basic design of grohtml(1).

For man(7), the problem is not quite as bad as for mdoc(7) because
the man(7) language is a 1970ies-style, purely presentational
markup language that conveys no semantic information in the first
place, so there isn't really much that could get lost during the
intermediate step.  Then again, i think nowadays, it makes more
sense to use a converter that can handle *both* languages well
instead of one that is kind of nearly good enough for man(7) but
really inadequate for mdoc(7).

[...]
> I guess it's due to the use of $(foreach ).  I guess it's a GNU 
> extension and make(1POSIX) ignores that creating an empty string.  Since 
> the Makefile uses a lot of functions[1], I guess it's not easy to make 
> it portable.
[...]
> [1]:  addsuffix, wildcard, foreach, filter, patsubst, sort, shell, 
> basename, notdir, info.  Not sure how many of those are supported by 
> your make(1); maybe none?

The OpenBSD implementation of make(1) is much more powerful than
POSIX make, but according to the manual page, you are right that
none of these keywords are supported by OpenBSD make, let alone by
POSIX make.

>>  (It's a completely
>> different story for a project like GNU troff that is about the
>> opposite of Linux only and that basically all UNIX systems want
>> to be able to use in production.)

> Yes, although they could definitely simplify the build system a little 
> bit.  It's one of the most complex builds I've ever seen.

I wouldn't go that far; i guess then you never looked at build systems
of large software like X11 or firefox or boost...  Even the OpenBSD
ports build framework is more complicated than the groff build system.
Then again, the groff build system *does* have a lot of abstraction,
indirection, and complexity *for such a small piece of software*.  :-/
But we're disgressing.

[ ... regarding installation instructions ...]
> No, you didn't.  I expected autocomplete to help,

I almost never use autocomplete except for the names of commands that are
installed system-wide and for the names of files in the local file system.
I didn't even know it is possible to use autocompletion for make targets,
and i dislike the idea.  But don't worry!  Your build system *is*
complicated for a package that actually doesn't need to build anything,
but not so bad that i didn't find what i looked for.  :)

>> I did not do measurements, but are you sure this is sane advice?
>> For an installation procedure, i would have expected execution
>> speed to be limited by disk I/O and i would have expected CPU
>> consumption to be negligible.  Did you do measurements to confirm
>> the relevance of this advice?

> Yes, I did:
> 
> [
> $ sudo make uninstall >/dev/null
> $ time sudo make install >/dev/null
> 
> real	0m7.722s
> user	0m0.004s
> sys	0m0.004s
> $ sudo make uninstall >/dev/null
> $ time sudo make -j2 install >/dev/null
> 
> real	0m4.634s
> user	0m0.005s
> sys	0m0.004s
> $ sudo make uninstall >/dev/null
> $ time sudo make -j install >/dev/null
> 
> real	0m3.170s
> user	0m0.000s
> sys	0m0.009s
> $ touch man2/membarrier.2
> $ time sudo make install >/dev/null
> 
> real	0m1.766s
> user	0m0.005s
> sys	0m0.004s
> $ touch man2/membarrier.2
> $ time sudo make -j2 install >/dev/null
> 
> real	0m1.735s
> user	0m0.004s
> sys	0m0.004s
> $ touch man2/membarrier.2
> $ time sudo make -j install >/dev/null
> 
> real	0m1.737s
> user	0m0.001s
> sys	0m0.008s
> ]
> 
> You can see that just for the $(shell ...) and all the text functions 
> that process the filenames to be installed it is taking almost 1.7 s, 
> even if it only needs to update a single page.

As a side remark, i consider it bad style to use dependencies during
installation: dependencies are for the build stage, not for the
installation stage.  When i say "make install", i just want *all*
the files installed unconditionally for two reasons: On the one hand,
dependency handling is error-prone and it would be bad if some file
does not get installed due to the notorious problem of oversights in
dependency handling (and dependency handling in parallel Makefiles
is even more fragile than in serial ones).  On the other hand, "make
install" also has the purpose of repairing an installation that got
broken in some way or other, and skipping some files because the
build system *thinks* they are probably still installed properly
defeats the purpose IMHO.

> That's a lot, but it has 
> it's advantages (generating the file list on-the-fly; no ./configure).
> 
> Then, the actual installation of the ~2.5k pages (most of them are link 
> pages),

As another aside, i consider using .so bad style.  It is unnecessarily
fragile.  Using hard links on the file system level (see ln(1))
is significantly more robust.  With mandoc(1), you don't need links
at all, but i admit traditional man(1) implementations including
man-db still require them for manual pages having more than one name.

> takes another 1.4 s in multi-process mode, and 6 s in 
> single-process mode (so at least 4.6 s that are not I/O).  Maybe it's 
> make(1) that has a hard time traversing the tree... I don't know where 
> the bottleneck is, but it's clearly there.

I see.  So you need multiple processors purely for dealing with make(1)
overhead...  Gee!  :-/

[...]
> BTW, did you check the changes to queue.3?  I guess you could improve 
> yours in a similar manner.
> 
> <https://linux-man-pages.blogspot.com/2020/11/man-pages-509-is-released.html>

I think in OpenBSD, these changes would get vetoed by large numbers
of developers because they violate the way OpenBSD manual pages are
organized in several ways:

 1. Your queue(7) manual page is placed in the wrong section.
    It is purely about an API provided by a library for the C language.
    Such information unambiguously belongs in section 3 and certainly
    not in section 7.  It is not even an edge case; it is perfectly
    clear what the correct section is.

 2. Your file names and .TH names violate the OpenBSD convention
    that section 2 and 3 manual pages must be named after functions
    or macros.  For example, the page name "slist" is not acceptable
    because no sname() function or macro exists.

 3. Splitting the page up into multiple pages is a bad idea for
    two reasons: it results in significant duplication of information
    and it splits information about interfaces so closely related
    to each other that most of their features are identical across
    multiple pages.

You would have no chance of getting anything like that committed to
OpenBSD.

> Also, if you have been following the addition of pages about types, and 
> would like to comment, you'll be welcome!
> 
> <https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=178eaf37e2e971cae88bd4d3f124ede0afbb1015>

BSD doesn't have manual pages about types, and i don't think there is a
significant benefit from having them.  Most standard types are trivial
and can easily be looked up in the header files with no need for separate
documentation.  In the unusual case that a type has non-trivial syntax
and/or semantics, it can be documented in the manual page of the most
closely related API function; for example, "struct pollfd" is documented
in our poll(2) page.  That makes it easy to find with the usual
"man -k Vt=typename" search command.

Documenting a non-trivial type separately from the functions using it
is counter-productive and makes the documentation hard to read because
programmers *never* need to use a type defined by a library unless they
also want to use related API functions.  Actually, i find it better
to *not* add type names as names of manual pages because that way,
the classical syntax "man functname" can be used to search for
function and macro names and the advanced syntax "man -k Vt=typename"
to search for types, with less potential for confusion.

So in OpenBSD, your pages about types would get vetoed on the grounds
of "pages not named after functions or macros" as well as on the
grounds of "these pages do not document any function or macro;
instead of creating a new page, put the information where it belongs."

That said, other projects are of course free to have such pages if
they want to.  The mandoc(1) program is also able to handle paths like
"man3/id_t.3type".  It will consider that page to be *both* in section
"3" (as specified by the directory name) and in section "3type" (as
specified by the file name and by the .TH macro).  I would consider
it better style to keep section names consistent, i.e. to use either
"man3/id_t.3" .TH id_t 3 or "man3type/id_t.3type" .TH id_t 3type,
but it's not a big deal: since many systems (in particular various
Linux distros) suffer from such inconsistencies, handling such
inconsistencies gracefully is an important feature that certainly
won't get removed.

>>  Commands like
>> 
>>     $ man -M /co/man-pages open
>> 
>> work perfectly fine on my system to view the Linux open(2) manual,
>> nicely formatted, with no need for installation or a Makefile.
>> Even when i put up a copy at
>> 
>>    https://man.bsd.lv/Linux-5.13/open

> How do you generate your HTML pages?  mandoc(1)?  They are nice.

https://man.openbsd.org/mandoc.1#HTML_Output
https://man.openbsd.org/man.cgi.8

I think that's the usual way to generate HTML from manual pages
nowadays.  The following sites also use mandoc for HTML output:

 * https://www.freebsd.org/cgi/man.cgi
 * https://manpages.debian.org/
 * https://man.archlinux.org/
 * https://man.voidlinux.org/

Some of these have their own CGI handling and/or database code,
but they all use the mandoc parser and HTML generator.

Yours,
  Ingo

Re: Linux man-pages Makefile portability

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 17436 bytes --]

[added Branden, as he was involved in discussions regarding man3type;
Branden, you might want to visit this thread from the begining, as I 
only copied the minimum to reply; it's in linux-man@]

Hi, Ingo! and Branden!

On 6/20/22 15:49, Ingo Schwarze wrote:
> 
>> But that Makefile was clearly unused since no-one knows.
> 
> I'm not saying "noone knows"; some packaging tools on some Linux
> distributions might very well rely on the Makefile - i don't know.

Oh, Michael seemed a bit surprised that I started patching the Makefile, 
as if the features I was patching hadn't been used in a very long time.

Anyway, I fixed `make all`.  I never liked it.  Now it builds all that 
it can build, which now is HTML pages, and in the future will probably 
include PDF pages too.  It was that, or make it a no-op, so I thought 
HTML+PDF was more useful.

> All i'm saying is i don't readily see a reason why people *not*
> running Linux might need the Makefile...

Well, the Makefile is basically meant to install (copy) the files to the 
system, so since you copy them to somewhere, `gmake install mandir=...` 
should work, but cp(1) -r tends to be as useful for such a simple package.

> 
>> If you know a better tool, I could start using it.  Maybe I could use
>> groff(1) directly, with grohtml(1).
> 
> I don't think using groff to generate HTML output from manual pages
> is a good idea.  It generates low-quality HTML code, and it is
> impossible to fix because of the basic concept how groff works.

[...]

Hmm, I'll try both and see.  Thanks.

> 
> [...]
>> I guess it's due to the use of $(foreach ).  I guess it's a GNU
>> extension and make(1POSIX) ignores that creating an empty string.  Since
>> the Makefile uses a lot of functions[1], I guess it's not easy to make
>> it portable.
> [...]
>> [1]:  addsuffix, wildcard, foreach, filter, patsubst, sort, shell,
>> basename, notdir, info.  Not sure how many of those are supported by
>> your make(1); maybe none?
> 
> The OpenBSD implementation of make(1) is much more powerful than
> POSIX make, but according to the manual page, you are right that
> none of these keywords are supported by OpenBSD make, let alone by
> POSIX make.

Heh, then to be compatible with BSD make(1) I guess I'd have to hardcode 
the page names, or use suffix rules, none of which convinces me.

Maybe suffix rules could work, but I'd have to stop testing the 
EXAMPLES, which is the most complex part of the Makefile.

I'll consider using them, or at least Substitution References[1] if it 
adds compatibility, at least for the simplest tasks, such as `make install`.

[1]: 
<https://www.gnu.org/software/make/manual/html_node/Substitution-Refs.html#Substitution-Refs>

>> No, you didn't.  I expected autocomplete to help,
> 
> I almost never use autocomplete except for the names of commands that are
> installed system-wide and for the names of files in the local file system.

Oh, I use autocomplete everyday for things like arguments to git 
commands, and I really feel it when I'm in a system where I don't have 
such help.

> I didn't even know it is possible to use autocompletion for make targets,
> and i dislike the idea.  But don't worry!  Your build system *is*
> complicated for a package that actually doesn't need to build anything,
> but not so bad that i didn't find what i looked for.  :)

:)

> As a side remark, i consider it bad style to use dependencies during
> installation: dependencies are for the build stage, not for the
> installation stage.  When i say "make install", i just want *all*
> the files installed unconditionally for two reasons: On the one hand,
> dependency handling is error-prone and it would be bad if some file
> does not get installed due to the notorious problem of oversights in
> dependency handling (and dependency handling in parallel Makefiles
> is even more fragile than in serial ones).  On the other hand, "make
> install" also has the purpose of repairing an installation that got
> broken in some way or other, and skipping some files because the
> build system *thinks* they are probably still installed properly
> defeats the purpose IMHO.

I've had doubts about that, and in the past I tended to do the same as 
you suggest, not because of fear to broken deps, but for making sure I 
don't create temporary files owned by root.  But in this case, where 
there are thousands of files to install, there's an important time 
difference between installing just the diff and installing the whole 
repo, so I asked the following question[2] just to confirm my doubts, 
and added the deps.

Regarding the possibility of broken deps, I believe the solution is to 
fix the Makefile, not to assume that it can't be done right and make it 
dumb; and I try hard to make sure my Makefiles work in multi-process 
mode.  There's always a chance that I got some corner case wrong, but 
this case it's pretty low (and if someone doesn't trust my Makefiles to 
behave well with -j, I don't force to use it, but I recommend it very 
much :)).

Regarding `make install` having a secondary purpose of being kind of a 
reinstall, I disagree.  I tend to write an explicit `make reinstall` 
target for that purpose (implemented as `\t$(MAKE) uninstall\n\t$(MAKE) 
install`); I didn't write it yet for the man-pages, but I'm going to add 
it now.

[2]: 
<https://stackoverflow.com/questions/70901364/should-make-install-depend-on-compilation>

> 
>> That's a lot, but it has
>> it's advantages (generating the file list on-the-fly; no ./configure).
>>
>> Then, the actual installation of the ~2.5k pages (most of them are link
>> pages),
> 
> As another aside, i consider using .so bad style.  It is unnecessarily
> fragile.  Using hard links on the file system level (see ln(1))
> is significantly more robust.  With mandoc(1), you don't need links
> at all, but i admit traditional man(1) implementations including
> man-db still require them for manual pages having more than one name.

I also had that feeling at first.  I just leave it there because of 
"don't fix it if it ain't broke" and it just works.  .so has a good 
side, which is that the Makefile is simpler, as it doesn't need to 
create links.

> 
>> takes another 1.4 s in multi-process mode, and 6 s in
>> single-process mode (so at least 4.6 s that are not I/O).  Maybe it's
>> make(1) that has a hard time traversing the tree... I don't know where
>> the bottleneck is, but it's clearly there.
> 
> I see.  So you need multiple processors purely for dealing with make(1)
> overhead...  Gee!  :-/

Yupee!  :/

I'll see if I can reduce that overhead without losing features.  Maybe I 
improve compatibility in the way.  :)

> 
> [...]
>> BTW, did you check the changes to queue.3?  I guess you could improve
>> yours in a similar manner.
>>
>> <https://linux-man-pages.blogspot.com/2020/11/man-pages-509-is-released.html>
> 
> I think in OpenBSD, these changes would get vetoed by large numbers
> of developers because they violate the way OpenBSD manual pages are
> organized in several ways:
> 
>   1. Your queue(7) manual page is placed in the wrong section.
>      It is purely about an API provided by a library for the C language.
>      Such information unambiguously belongs in section 3 and certainly
>      not in section 7.  It is not even an edge case; it is perfectly
>      clear what the correct section is.

See 2.

> 
>   2. Your file names and .TH names violate the OpenBSD convention
>      that section 2 and 3 manual pages must be named after functions
>      or macros.  For example, the page name "slist" is not acceptable
>      because no sname() function or macro exists.

Heh, I agree!  I would have put them in section 7, but I was new to the 
project, and didn't want to change things too much at the time.  Since 
queue(3) was in man3, I kept the tradition, and the child pages were 
kept in man3.  Probably I should have put them in man7, but blame 
history, not me :)

Buuut, is it me, or I see a contradiction with point 1, which claims 
that queue(3) should be in man3?  We don't have a slist() 
function/macro, but we don't have a queue() one either (maybe 
historically there was one and I don't know it, but I guess not.  My 
systems say:

alx@devuan:/usr/include$ grepc queue
alx@devuan:/usr/include$

alx@debian:/usr/include$ grepc queue
alx@debian:/usr/include$

Should now queue() be in man3 or man7?


> 
>   3. Splitting the page up into multiple pages is a bad idea for
>      two reasons: it results in significant duplication of information
>      and it splits information about interfaces so closely related
>      to each other that most of their features are identical across
>      multiple pages.

Actually, I didn't duplicate information at all, AFAIR.  It was already 
_very_ duplicated in the same queue(3) page, so I just splitted it at 
the right points.  I only had to cut the page into many little ones, 
then translate the pages from mdoc(7) to man(7), and then fix minor 
style issues.

See:

$ wc -l man3/circleq.3 man3/list.3 man3/slist.3 man3/stailq.3 
man3/tailq.3 man7/queue.7
   318 man3/circleq.3
   306 man3/list.3
   317 man3/slist.3
   375 man3/stailq.3
   395 man3/tailq.3
   133 man7/queue.7
  1844 total
$ git checkout man-pages-5.08 >/dev/null 2>&1
$ wc -l man3/queue.3
1231 man3/queue.3

The difference is just source code overhead; the text is almost the same.

Maybe you could still simplify your queue(7) page in a different way, 
without splitting it; it is very repetitive.

> 
> You would have no chance of getting anything like that committed to
> OpenBSD.

Heh, I know.  The only thing that was well received from my side in that 
list was a bug report about some exec(3) function (about alloca(3)).

> 
>> Also, if you have been following the addition of pages about types, and
>> would like to comment, you'll be welcome!
>>
>> <https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=178eaf37e2e971cae88bd4d3f124ede0afbb1015>
> 
> BSD doesn't have manual pages about types, and i don't think there is a
> significant benefit from having them.  Most standard types are trivial
> and can easily be looked up in the header files with no need for separate
> documentation.

Oh, yes, I envy your headers.  They are really readable!
But glibc source code is not as friendly, and it took me a long long 
time to get used to search things in those headers (I still can't find 
some things in their code; function definitions, for example, are very 
cryptic in some cases).

Also, some programmers, especially when starting (but I know of many 
programmers that are "senior" and still have serious issues with types), 
would benefit from documentation specific about types.  That would help 
understand their limitations, and what a type is appropriate for or not.

>  In the unusual case that a type has non-trivial syntax
> and/or semantics, it can be documented in the manual page of the most
> closely related API function; for example, "struct pollfd" is documented
> in our poll(2) page.



> That makes it easy to find with the usual
> "man -k Vt=typename" search command.

Oh, that's where man(7) sucks :)

I wouldn't need these:

$ sed -n /^man_lsfunc/,/^}/p <scripts/bash_aliases
man_lsfunc()
{
	if [ $# -lt 1 ]; then
		>&2 echo "Usage: ${FUNCNAME[0]} <manpage|manNdir>...";
		return $EX_USAGE;
	fi

	for arg in "$@"; do
		man_section "$arg" 'SYNOPSIS';
	done \
	|sed_rm_ccomments \
	|pcregrep -Mn '(?s)^ [\w ]+ \**\w+\([\w\s(,)[\]*]*?(...)?\s*\); *$' \
	|grep '^[0-9]' \
	|sed -E 's/syscall\(SYS_(\w*),?/\1(/' \
	|sed -E 's/^[^(]+ \**(\w+)\(.*/\1/' \
	|uniq;
}
$ sed -n /^man_lsvar/,/^}/p <scripts/bash_aliases
man_lsvar()
{
	if [ $# -lt 1 ]; then
		>&2 echo "Usage: ${FUNCNAME[0]} <manpage|manNdir>...";
		return $EX_USAGE;
	fi

	for arg in "$@"; do
		man_section "$arg" 'SYNOPSIS';
	done \
	|sed_rm_ccomments \
	|pcregrep -Mv '(?s)^ [\w ]+ \**\w+\([\w\s(,)[\]*]+?(...)?\s*\); *$' \
	|pcregrep -Mn \
	  -e '(?s)^ +extern [\w ]+ \**\(\*+[\w ]+\)\([\w\s(,)[\]*]+?\s*\); *$' \
	  -e '^ +extern [\w ]+ \**[\w ]+; *$' \
	|grep '^[0-9]' \
	|grep -v 'typedef' \
	|sed -E 's/^[0-9]+: +extern [^(]+ \**\(\*+(\w* )?(\w+)\)\(.*/\2/' \
	|sed    's/^[0-9]\+: \+extern .* \**\(\w\+\); */\1/' \
	|uniq;
}

And they're not perfect...


> 
> Documenting a non-trivial type separately from the functions using it
> is counter-productive and makes the documentation hard to read because
> programmers *never* need to use a type defined by a library unless they
> also want to use related API functions.

That's not true.  I've needed (or better phrased, wanted) types, even 
when I wasn't using any APIs that used them.  The reason was that I was 
designing an API, and wanted to use the most appropriate types for my 
functions.

Having had documentation about types would have helped *a lot* at the time.

Many programmers don't know all the differences between size_t and 
ssize_t, and for example that ISO C only provides one of them (the other 
is added by POSIX), and I know of one *great* programmer that learnt the 
difference between those types from me a few months ago :).

One could search in the standard documents about the types, but I guess 
we will agree that those documents are not very friendly, especially for 
beginners.

Another case where I've found my type pages very useful was when I 
contributed[3] to iwyu(1)[4].  Having to read the POSIX or ISO C 
documents would have been crazy (I had to do it anyway, to write the 
pages, but I don't want to repeat that process again ;)).

[3]: <https://github.com/include-what-you-use/include-what-you-use/pull/930>
[4]: <https://include-what-you-use.org/>

>  Actually, i find it better
> to *not* add type names as names of manual pages because that way,
> the classical syntax "man functname" can be used to search for
> function and macro names and the advanced syntax "man -k Vt=typename"
> to search for types, with less potential for confusion.

We don't have that feature in man(7), so the closest thing that I do is 
to grep in the glibc and BSDs source code with grepc(1)[3], and also 
`grep -rn ...` inside the man-pages repo.

[5]: <http://www.alejandro-colomar.es/src/alx/alx/grepc.git/>

> 
> So in OpenBSD, your pages about types would get vetoed on the grounds
> of "pages not named after functions or macros" as well as on the
> grounds of "these pages do not document any function or macro;
> instead of creating a new page, put the information where it belongs."

The first argument, I agree, and it's why I didn't use section 3, but 
subsection 3type.

The second, I disagree for the reasons above, but can understand why 
others might disagree with me.  Maybe I can convince you :)

> 
> That said, other projects are of course free to have such pages if
> they want to.  The mandoc(1) program is also able to handle paths like
> "man3/id_t.3type".  It will consider that page to be *both* in section
> "3" (as specified by the directory name) and in section "3type" (as
> specified by the file name and by the .TH macro).  I would consider
> it better style to keep section names consistent, i.e. to use either
> "man3/id_t.3" .TH id_t 3 or "man3type/id_t.3type" .TH id_t 3type,
> but it's not a big deal: since many systems (in particular various
> Linux distros) suffer from such inconsistencies, handling such
> inconsistencies gracefully is an important feature that certainly
> won't get removed.

I considered[6] using man3type, and used man3 in the end just because 
when in doubt I opted for the smallest change.  Knowing that it breaks 
mandoc(1), I'll definitely move to <man3type/>.

[6]: 
<https://lore.kernel.org/linux-man/761bb12f-31e0-369d-8315-d2e1545505c7@gmail.com/T/#u>

> 
>>>   Commands like
>>>
>>>      $ man -M /co/man-pages open
>>>
>>> work perfectly fine on my system to view the Linux open(2) manual,
>>> nicely formatted, with no need for installation or a Makefile.
>>> Even when i put up a copy at
>>>
>>>     https://man.bsd.lv/Linux-5.13/open

Yes, since there's no compilation, `make install` is basically a wrapper 
around `cp -r`.

It has nice features, such as reduced install time by checking 
timestamps, but that's more useful to me as a maintainer (since I 
install several times a minute in some cases), and not so much for end 
users, where a few seconds are not important.

> 
>> How do you generate your HTML pages?  mandoc(1)?  They are nice.
> 
> https://man.openbsd.org/mandoc.1#HTML_Output
> https://man.openbsd.org/man.cgi.8
> 
> I think that's the usual way to generate HTML from manual pages
> nowadays.  The following sites also use mandoc for HTML output:
> 
>   * https://www.freebsd.org/cgi/man.cgi
>   * https://manpages.debian.org/
>   * https://man.archlinux.org/
>   * https://man.voidlinux.org/
> 
> Some of these have their own CGI handling and/or database code,
> but they all use the mandoc parser and HTML generator.

Interesting.  Thanks!

Cheers,

Alex

-- 
Alejandro Colomar
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Linux man-pages Makefile portability

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 1827 bytes --]

Hi Ingo!

On 7/3/22 23:44, Alejandro Colomar wrote:
> [added Branden, as he was involved in discussions regarding man3type;
> Branden, you might want to visit this thread from the begining, as I 
> only copied the minimum to reply; it's in linux-man@]
> 
> On 6/20/22 15:49, Ingo Schwarze wrote:
[...]
>> That said, other projects are of course free to have such pages if
>> they want to.  The mandoc(1) program is also able to handle paths like
>> "man3/id_t.3type".  It will consider that page to be *both* in section
>> "3" (as specified by the directory name) and in section "3type" (as
>> specified by the file name and by the .TH macro).  I would consider
>> it better style to keep section names consistent, i.e. to use either
>> "man3/id_t.3" .TH id_t 3 or "man3type/id_t.3type" .TH id_t 3type,
>> but it's not a big deal: since many systems (in particular various
>> Linux distros) suffer from such inconsistencies, handling such
>> inconsistencies gracefully is an important feature that certainly
>> won't get removed.
> 
> I considered[6] using man3type, and used man3 in the end just because 
> when in doubt I opted for the smallest change.  Knowing that it breaks 
> mandoc(1), I'll definitely move to <man3type/>.
> 
> [6]: 
> <https://lore.kernel.org/linux-man/761bb12f-31e0-369d-8315-d2e1545505c7@gmail.com/T/#u>
> 

I fixed it:
<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=451a27a78d51973b01bfb5d3b1e0ec081d9161e1>

And Debian seems to work fine with man3type/ and man2type/ out of the 
box, so I prefer it this way.  I hope that other projects follow the 
example; and that packagers/distributions also create subsection 
directories, and don't undo my work.

Cheers,

Alex

-- 
Alejandro Colomar
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Linux man-pages Makefile portability

[Ingo Schwarze]

Hi Alejandro,

Alejandro Colomar wrote on Thu, Jul 21, 2022 at 04:17:18PM +0200:
> On 7/3/22 23:44, Alejandro Colomar wrote:

>> [added Branden, as he was involved in discussions regarding man3type;
>> Branden, you might want to visit this thread from the begining, as I 
>> only copied the minimum to reply; it's in linux-man@]

>> On 6/20/22 15:49, Ingo Schwarze wrote:
> [...]
>>> That said, other projects are of course free to have such pages if
>>> they want to.  The mandoc(1) program is also able to handle paths like
>>> "man3/id_t.3type".  It will consider that page to be *both* in section
>>> "3" (as specified by the directory name) and in section "3type" (as
>>> specified by the file name and by the .TH macro).  I would consider
>>> it better style to keep section names consistent, i.e. to use either
>>> "man3/id_t.3" .TH id_t 3 or "man3type/id_t.3type" .TH id_t 3type,
>>> but it's not a big deal: since many systems (in particular various
>>> Linux distros) suffer from such inconsistencies, handling such
>>> inconsistencies gracefully is an important feature that certainly
>>> won't get removed.

>> I considered[6] using man3type, and used man3 in the end just because 
>> when in doubt I opted for the smallest change.  Knowing that it breaks 
>> mandoc(1), I'll definitely move to <man3type/>.

I didn't mean to say man3/id_t.3type "breaks mandoc".  Quite to the
contrary, the above quotation explains that mandoc copes with it.

However, when it comes to robustness with respect to *other* man(1)
implementations apart from mandoc and man-db, i suspect the most
portable and reliable way is using man1 - man9 only with no suffixes,
consistent suffixes like "man3type/id_t.3type" are probably medium
portability and medium reliability, and inconsistent suffixes
like "man3/id_t.3type" and "man3type/id_t.3" are likely the least
portable and the most fragile.

So your change is an improvement.

The system making the heaviest use of section suffixes i'm aware of
is Solaris:

  > uname -a
  SunOS unstable11s 5.11 11.3 sun4u sparc SUNW,SPARC-Enterprise
  > ls /usr/share/man/
  entities      man3dat       man3mvec      man3sysevent    man4b
  fr            man3dax       man3nsl       man3tcl         man5
  fr.ISO8859-1  man3devid     man3nvpair    man3tecla       man5oldap
  fr.UTF-8      man3devinfo   man3oldap     man3tiff        man5openssl
  it            man3dlpi      man3openssl   man3tsol        man7
  it.ISO8859-1  man3dns_sd    man3p         man3uuid        man7d
  it.UTF-8      man3elf       man3pam       man3volmgt      man7fs
  ja_JP.UTF-8   man3exacct    man3pcap      man3x           man7i
  man-index     man3ext       man3perl      man3x11         man7ipp
  man.cf        man3f         man3pi        man3xau         man7m
  man1          man3fcoe      man3picl      man3xaw         man7openssl
  man1b         man3fm        man3picltree  man3xcb         man7p
  man1c         man3fstyp     man3plot      man3xcomposite  man8
  man1m         man3gen       man3pool      man3xcurses     man8oldap
  man1oldap     man3gss       man3proc      man3xcursor     man8s
  man1openssl   man3hbaapi    man3project   man3xevie       man9
  man1s         man3head      man3rad       man3xext        man9e
  man1t         man3iscsit    man3reparse   man3xi          man9f
  man2          man3kstat     man3resolv    man3xinerama    man9p
  man3          man3kvm       man3rpc       man3xmu         man9s
  man3archive   man3layout    man3sasl      man3xnet        pl
  man3c         man3ldap      man3scf       man3xrandr      pl.ISO8859-2
  man3c_db      man3lgrp      man3sec       man3xss         pl.UTF-8
  man3cc4       man3lib       man3sip       man3xt          ru.KOI8-R
  man3cfgadm    man3m         man3slp       man3xtsol       ru.UTF-8
  man3cmi       man3mail      man3snmp      man3xtst        zh_CN.UTF-8
  man3commputil man3malloc    man3socket    man3xv
  man3contract  man3mlib      man3srpt      man3xxf86vm
  man3cpc       man3mp        man3ssh2      man3zonestat
  man3curses    man3mpapi     man3stmf      man4

Inside these directories, they are *mostly* using the convention
"keep both section names consistent", so i do think that is good
to follow.  Even Solaris isn't perfect in that respect, though,
they have, for example,

  /usr/share/man/man3cc4/cartpol.3

but on first sight, i only found about two dozen such examples.

> I fixed it:
> <https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=451a27a78d51973b01bfb5d3b1e0ec081d9161e1>

I did not scrutinize your change in detail, but had a brief look at it
and did not see any obvious problems.

> And Debian seems to work fine with man3type/ and man2type/ out of the 
> box, so I prefer it this way.  I hope that other projects follow the 
> example; and that packagers/distributions also create subsection 
> directories, and don't undo my work.

On Linux, undoing it would make very little sense to me because i expect
that all man(1) programs commonly used on Linux can cope with section
suffixes, and in particular with consistent use of session suffixes.
Besides, undoing it properly is hardly possibly for a packager.
If would require changing all these:

 - directory names
 - file names
 - .TH macros
 - and manual page cross references

So for better or worse, the only sane option for a packages is to follow
your lead.

Yours,
  Ingo

Re: Linux man-pages Makefile portability

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 5984 bytes --]

Hi Ingo!

On 7/22/22 18:59, Ingo Schwarze wrote:
[...]
>>> I considered[6] using man3type, and used man3 in the end just because
>>> when in doubt I opted for the smallest change.  Knowing that it breaks
>>> mandoc(1), I'll definitely move to <man3type/>.
> 
> I didn't mean to say man3/id_t.3type "breaks mandoc".  Quite to the
> contrary, the above quotation explains that mandoc copes with it.

Yeah, I didn't mean break as in "mandoc(1) goes nuts, or crashes", but 
rather as in "it doesn't do what I wanted it to do".

> 
> However, when it comes to robustness with respect to *other* man(1)
> implementations apart from mandoc and man-db, i suspect the most
> portable and reliable way is using man1 - man9 only with no suffixes,
> consistent suffixes like "man3type/id_t.3type" are probably medium
> portability and medium reliability, and inconsistent suffixes
> like "man3/id_t.3type" and "man3type/id_t.3" are likely the least
> portable and the most fragile.
> 
> So your change is an improvement.

:)

> 
> The system making the heaviest use of section suffixes i'm aware of
> is Solaris:
> 
>    > uname -a
>    SunOS unstable11s 5.11 11.3 sun4u sparc SUNW,SPARC-Enterprise
>    > ls /usr/share/man/
>    entities      man3dat       man3mvec      man3sysevent    man4b
>    fr            man3dax       man3nsl       man3tcl         man5
>    fr.ISO8859-1  man3devid     man3nvpair    man3tecla       man5oldap
>    fr.UTF-8      man3devinfo   man3oldap     man3tiff        man5openssl
>    it            man3dlpi      man3openssl   man3tsol        man7
>    it.ISO8859-1  man3dns_sd    man3p         man3uuid        man7d
>    it.UTF-8      man3elf       man3pam       man3volmgt      man7fs
>    ja_JP.UTF-8   man3exacct    man3pcap      man3x           man7i
>    man-index     man3ext       man3perl      man3x11         man7ipp
>    man.cf        man3f         man3pi        man3xau         man7m
>    man1          man3fcoe      man3picl      man3xaw         man7openssl
>    man1b         man3fm        man3picltree  man3xcb         man7p
>    man1c         man3fstyp     man3plot      man3xcomposite  man8
>    man1m         man3gen       man3pool      man3xcurses     man8oldap
>    man1oldap     man3gss       man3proc      man3xcursor     man8s
>    man1openssl   man3hbaapi    man3project   man3xevie       man9
>    man1s         man3head      man3rad       man3xext        man9e
>    man1t         man3iscsit    man3reparse   man3xi          man9f
>    man2          man3kstat     man3resolv    man3xinerama    man9p
>    man3          man3kvm       man3rpc       man3xmu         man9s
>    man3archive   man3layout    man3sasl      man3xnet        pl
>    man3c         man3ldap      man3scf       man3xrandr      pl.ISO8859-2
>    man3c_db      man3lgrp      man3sec       man3xss         pl.UTF-8
>    man3cc4       man3lib       man3sip       man3xt          ru.KOI8-R
>    man3cfgadm    man3m         man3slp       man3xtsol       ru.UTF-8
>    man3cmi       man3mail      man3snmp      man3xtst        zh_CN.UTF-8
>    man3commputil man3malloc    man3socket    man3xv
>    man3contract  man3mlib      man3srpt      man3xxf86vm
>    man3cpc       man3mp        man3ssh2      man3zonestat
>    man3curses    man3mpapi     man3stmf      man4

Wow!
Although it's interesting to know that this list exists:
I can check it when trying to come up with a section name.

I guess Illumos shares this subsectioning scheme.

Do you know from the top of your head if any of those is dedicated to 
constants such as NULL, PATH_MAX, or BUFSIZ?

In my git clone of Illumos, I can't find anything relevant:

alx@asus5775:~/src/illumos$ find  | grep -i null | grep man
./illumos-gate/usr/src/man/man9f/nulldev.9f
./illumos-gate/usr/src/man/man4d/null.4d
./illumos-gate/usr/src/man/man4d/nulldriver.4d


> 
> Inside these directories, they are *mostly* using the convention
> "keep both section names consistent", so i do think that is good
> to follow.  Even Solaris isn't perfect in that respect, though,
> they have, for example,
> 
>    /usr/share/man/man3cc4/cartpol.3
> 
> but on first sight, i only found about two dozen such examples.
> 
>> I fixed it:
>> <https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=451a27a78d51973b01bfb5d3b1e0ec081d9161e1>
> 
> I did not scrutinize your change in detail, but had a brief look at it
> and did not see any obvious problems.
> 
>> And Debian seems to work fine with man3type/ and man2type/ out of the
>> box, so I prefer it this way.  I hope that other projects follow the
>> example; and that packagers/distributions also create subsection
>> directories, and don't undo my work.
> 
> On Linux, undoing it would make very little sense to me because i expect
> that all man(1) programs commonly used on Linux can cope with section
> suffixes, and in particular with consistent use of session suffixes.
> Besides, undoing it properly is hardly possibly for a packager.
> If would require changing all these:
> 
>   - directory names

I actually make it as easy as I can to modify directory names in the 
Makefile through the standard GNU directory variables.

>   - file names

I don't expect them to change filenames, even if they change dirs.  It 
wouldn't make sense at all.

>   - .TH macros

The same as fith file names.  I expect them to want the pages to be in 
subsections, even if they wouldn't like the directory structure.

>   - and manual page cross references

That's a more difficult one.  They would need to run a sed script for 
every release.  A good reason to not modify it.

If they change dirs and don't change this, they'd break .so links.

> 
> So for better or worse, the only sane option for a packages is to follow
> your lead.

:)

Cheers,

Alex

> 
> Yours,
>    Ingo

-- 
Alejandro Colomar
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Linux man-pages Makefile portability

[Ingo Schwarze]

Hi Alejandro,

Alejandro Colomar wrote on Fri, Jul 22, 2022 at 07:37:14PM +0200:
> On 7/22/22 18:59, Ingo Schwarze wrote:
>> Alejandro Colomar wrote:

>>> I considered[6] using man3type, and used man3 in the end just because
>>> when in doubt I opted for the smallest change.  Knowing that it breaks
>>> mandoc(1), I'll definitely move to <man3type/>.

>> I didn't mean to say man3/id_t.3type "breaks mandoc".  Quite to the
>> contrary, the above quotation explains that mandoc copes with it.

> Yeah, I didn't mean break as in "mandoc(1) goes nuts, or crashes", but 
> rather as in "it doesn't do what I wanted it to do".

Even that is an overstatement.  The difference between being in section 3p
only and being in both section 3p and 3 is barely user-visible.

Here is a manual page that is in section 3p and in section 3p only,
on OpenBSD-current:

   $ man -w warnings
  /usr/share/man/man3p/warnings.3p
   $ grep  '^\.TH' $(man -w warnings)
  .TH warnings 3p "2021-03-02" "perl v5.32.1" "Perl Programmers Reference Guide"

But lo and behold:

   $ man -s 3 -w warnings
  /usr/share/man/man3p/warnings.3p

Mandoc still finds the page in section 3 because "3" is a substring of "3p",
and that not only makes sense from the string processing perspective
but also from a logical perspective because section 3p is a particular
corner of section 3 after all.

To cause mandoc to *not* find the page in section 3, the user would have
to type something like

   $ man -k 'sec~3$' -a warnings
  man: nothing appropriate

Even i don't normally use such advanced syntax (regex matching on section
names using the "sec" search keyword and the explicit regex operator
and the -a logical operator for the conjunction of two search criteria).
I doubt whether man-db even supports similar features...

[...]
>> The system making the heaviest use of section suffixes i'm aware of
>> is Solaris:
>> 
>>    > uname -a
>>    SunOS unstable11s 5.11 11.3 sun4u sparc SUNW,SPARC-Enterprise
>>    > ls /usr/share/man/
>>    entities      man3dat       man3mvec      man3sysevent    man4b
>>    fr            man3dax       man3nsl       man3tcl         man5
>>    fr.ISO8859-1  man3devid     man3nvpair    man3tecla       man5oldap
>>    fr.UTF-8      man3devinfo   man3oldap     man3tiff        man5openssl
>>    it            man3dlpi      man3openssl   man3tsol        man7
>>    it.ISO8859-1  man3dns_sd    man3p         man3uuid        man7d
>>    it.UTF-8      man3elf       man3pam       man3volmgt      man7fs
>>    ja_JP.UTF-8   man3exacct    man3pcap      man3x           man7i
>>    man-index     man3ext       man3perl      man3x11         man7ipp
>>    man.cf        man3f         man3pi        man3xau         man7m
>>    man1          man3fcoe      man3picl      man3xaw         man7openssl
>>    man1b         man3fm        man3picltree  man3xcb         man7p
>>    man1c         man3fstyp     man3plot      man3xcomposite  man8
>>    man1m         man3gen       man3pool      man3xcurses     man8oldap
>>    man1oldap     man3gss       man3proc      man3xcursor     man8s
>>    man1openssl   man3hbaapi    man3project   man3xevie       man9
>>    man1s         man3head      man3rad       man3xext        man9e
>>    man1t         man3iscsit    man3reparse   man3xi          man9f
>>    man2          man3kstat     man3resolv    man3xinerama    man9p
>>    man3          man3kvm       man3rpc       man3xmu         man9s
>>    man3archive   man3layout    man3sasl      man3xnet        pl
>>    man3c         man3ldap      man3scf       man3xrandr      pl.ISO8859-2
>>    man3c_db      man3lgrp      man3sec       man3xss         pl.UTF-8
>>    man3cc4       man3lib       man3sip       man3xt          ru.KOI8-R
>>    man3cfgadm    man3m         man3slp       man3xtsol       ru.UTF-8
>>    man3cmi       man3mail      man3snmp      man3xtst        zh_CN.UTF-8
>>    man3commputil man3malloc    man3socket    man3xv
>>    man3contract  man3mlib      man3srpt      man3xxf86vm
>>    man3cpc       man3mp        man3ssh2      man3zonestat
>>    man3curses    man3mpapi     man3stmf      man4

> Wow!
> Although it's interesting to know that this list exists:
> I can check it when trying to come up with a section name.

I would somewhat advise against that.  While i do think that consistency
is good *if* you decide to use a section suffix, i'd still recommend
to use section suffixes sparingly, at least in operating systems
that use them sparingly now.  They provide relatively little value,
make the top directory of the manpath larger and less readable,
and they are in particular *not* well suited to moving stuff out of
the non-suffix directories that users may not wish to see by default.
For example, suppose you created a directory man3py to document
python library functions.  As explained above, these pages will *still*
show up even when people type "man 3" or "man 3p" rather than "man 3py".

> I guess Illumos shares this subsectioning scheme.

More or less, according to
https://src.illumos.org/source/xref/illumos-gate/usr/src/man/ -
it's not identical though.

> Do you know from the top of your head if any of those is dedicated to 
> constants such as NULL, PATH_MAX, or BUFSIZ?

I doubt it, for two reasons.  On the one hand, my impression is that
at least in Solaris, section suffixes are not so much used for
logical subdivision of sections but more according to provenance;
for example, man1b is for BSD compat features, man1s is for commands
specific to SunOS, man3c is for libc, man3ext is for an "extended
library", man3p is for the Sun performance library (available for
both C and FORTRAN 95), and so on.

On the other hand, naming manual pages after symbolic constants
or after type names is so unsual that i doubt any scheme exists
for that.  The most widely used way to look up manual pages
by the names of symbolic constants or type names probably is
using macro keys as implemented in the mandoc version of apropos(1).
That is used by most FreeBSD, OpenBSD, Alpine Linux, and Void Linux.
I admit that doesn't qualify as "widely used", but "most widely used"
is probably true all the same.  ;-)

Yours,
  Ingo

Re: Linux man-pages Makefile portability

[Alejandro Colomar (man-pages)]

[TO += Colin; CC += man-dv-devel@;
there's a bug in man(1)'s autocomplete]

Hi Ingo (and Colin),

On 7/23/22 20:16, Ingo Schwarze wrote:
> [...]
>>> The system making the heaviest use of section suffixes i'm aware of
>>> is Solaris:
>>>
>>>     > uname -a
>>>     SunOS unstable11s 5.11 11.3 sun4u sparc SUNW,SPARC-Enterprise
>>>     > ls /usr/share/man/
>>>     entities      man3dat       man3mvec      man3sysevent    man4b
>>>     fr            man3dax       man3nsl       man3tcl         man5
>>>     fr.ISO8859-1  man3devid     man3nvpair    man3tecla       man5oldap
>>>     fr.UTF-8      man3devinfo   man3oldap     man3tiff        man5openssl
>>>     it            man3dlpi      man3openssl   man3tsol        man7
>>>     it.ISO8859-1  man3dns_sd    man3p         man3uuid        man7d
>>>     it.UTF-8      man3elf       man3pam       man3volmgt      man7fs
>>>     ja_JP.UTF-8   man3exacct    man3pcap      man3x           man7i
>>>     man-index     man3ext       man3perl      man3x11         man7ipp
>>>     man.cf        man3f         man3pi        man3xau         man7m
>>>     man1          man3fcoe      man3picl      man3xaw         man7openssl
>>>     man1b         man3fm        man3picltree  man3xcb         man7p
>>>     man1c         man3fstyp     man3plot      man3xcomposite  man8
>>>     man1m         man3gen       man3pool      man3xcurses     man8oldap
>>>     man1oldap     man3gss       man3proc      man3xcursor     man8s
>>>     man1openssl   man3hbaapi    man3project   man3xevie       man9
>>>     man1s         man3head      man3rad       man3xext        man9e
>>>     man1t         man3iscsit    man3reparse   man3xi          man9f
>>>     man2          man3kstat     man3resolv    man3xinerama    man9p
>>>     man3          man3kvm       man3rpc       man3xmu         man9s
>>>     man3archive   man3layout    man3sasl      man3xnet        pl
>>>     man3c         man3ldap      man3scf       man3xrandr      pl.ISO8859-2
>>>     man3c_db      man3lgrp      man3sec       man3xss         pl.UTF-8
>>>     man3cc4       man3lib       man3sip       man3xt          ru.KOI8-R
>>>     man3cfgadm    man3m         man3slp       man3xtsol       ru.UTF-8
>>>     man3cmi       man3mail      man3snmp      man3xtst        zh_CN.UTF-8
>>>     man3commputil man3malloc    man3socket    man3xv
>>>     man3contract  man3mlib      man3srpt      man3xxf86vm
>>>     man3cpc       man3mp        man3ssh2      man3zonestat
>>>     man3curses    man3mpapi     man3stmf      man4
> 
>> Wow!
>> Although it's interesting to know that this list exists:
>> I can check it when trying to come up with a section name.
> 
> I would somewhat advise against that.  While i do think that consistency
> is good *if* you decide to use a section suffix, i'd still recommend
> to use section suffixes sparingly, at least in operating systems
> that use them sparingly now.  They provide relatively little value,
> make the top directory of the manpath larger and less readable,

I don't think there's any benefit of keeping $(mandir) contain only the 
standard man? subdirs.  Everybody already knows about them, so nobody 
needs to read the output of ls(1) in such a system.  On the other hand, 
reading the output of ls(1) in a system with free use of subsections 
provides useful information.  If one needs to filter a bit, standard 
Unix tools come into play.

> and they are in particular *not* well suited to moving stuff out of
> the non-suffix directories that users may not wish to see by default.
> For example, suppose you created a directory man3py to document
> python library functions.  As explained above, these pages will *still*
> show up even when people type "man 3" or "man 3p" rather than "man 3py".

But the good thing is they can be put the last in the list of pages to 
be shown by man(1), so they won't hide another page.

In my Debian system, man(1) shows:

  1 n l 8 3 0 2 3posix 3pm 3perl 3am 5 4 9 6 7

3type and 3const could go at the end of that, to avoid hiding pages in 
section 7.

While testing some related stuff to add to my arguments, I just 
understood of a bug that I knew existed but never thought too much about 
it, and related to one that Branden reported to me recently:

man 3 foo[TAB]

autocomplete is smart enough to only search pages in section 3 (I don't 
know if it finds pages in subsections from 3; I guess it does).

man [-s]3type foo[TAB]

autocomplete should search in 3type, but it doesn't quite work.
However, I noticed there's a difference in behavior between 3posix and 
3type (3posix only exists as a suffix, but pages live in man3; 3type is 
a correct subsection, using man3type and .3type), so maybe my work can 
help fix that bug in the autocomplete scripts.


I know you don't use autocomplete for those things, but it comes in 
handy to me :)

> 
>> I guess Illumos shares this subsectioning scheme.
> 
> More or less, according to
> https://src.illumos.org/source/xref/illumos-gate/usr/src/man/ -
> it's not identical though.
> 
>> Do you know from the top of your head if any of those is dedicated to
>> constants such as NULL, PATH_MAX, or BUFSIZ?
> 
> I doubt it, for two reasons.  On the one hand, my impression is that
> at least in Solaris, section suffixes are not so much used for
> logical subdivision of sections but more according to provenance;
> for example, man1b is for BSD compat features, man1s is for commands
> specific to SunOS, man3c is for libc, man3ext is for an "extended
> library", man3p is for the Sun performance library (available for
> both C and FORTRAN 95), and so on.
> 
> On the other hand, naming manual pages after symbolic constants
> or after type names is so unsual that i doubt any scheme exists
> for that.

Actually, man3type exists in several systems.  Solaris has it (I guess 
Illumos too), and I've seen it in other systems (something from Oracle, 
IIRC).  libbsd(7) also documents types, although they put them in the 
global namespace, which I think you and I agree that it's not quite 
right because of "documentation about a non-function in man3".

>  The most widely used way to look up manual pages
> by the names of symbolic constants or type names probably is
> using macro keys as implemented in the mandoc version of apropos(1).
> That is used by most FreeBSD, OpenBSD, Alpine Linux, and Void Linux.
> I admit that doesn't qualify as "widely used", but "most widely used"
> is probably true all the same.  ;-)

That leaves out man(7).  And types tend to be not very well documented 
if they are documented as part of a function page.  And they also tend 
to be documented several times (out of sync, of course).


Cheers,

Alex

-- 
Alejandro Colomar
Linux man-pages comaintainer; http://www.kernel.org/doc/man-pages/
http://www.alejandro-colomar.es/

Re: Linux man-pages Makefile portability

[Ingo Schwarze]

Hi Alejandro,

Alejandro Colomar wrote on Sun, Jul 24, 2022 at 01:09:23PM +0200:
> On 7/23/22 20:16, Ingo Schwarze wrote:
>> Alejandro Colomar wrote:

[...]
>>> Although it's interesting to know that this list exists:
>>> I can check it when trying to come up with a section name.

>> I would somewhat advise against that.  While i do think that consistency
>> is good *if* you decide to use a section suffix, i'd still recommend
>> to use section suffixes sparingly, at least in operating systems
>> that use them sparingly now.  They provide relatively little value,
>> make the top directory of the manpath larger and less readable,

> I don't think there's any benefit of keeping $(mandir) contain only the 
> standard man? subdirs.  Everybody already knows about them, so nobody 
> needs to read the output of ls(1) in such a system.

Admitted, it's a weak argument.  Having lots of useless directories
does not do *significant* harm.  Then again, in general, i would say
it is good practice to not add needless files or directories, even
when they are not particularly harmful.  They still make find(1)
and ls -R output and Makefiles longer.  Not doing needless things
is a general aim of design economy.

Besides, when there is not much to gain in the first place, even
weak counter-arguments wield non-zero weight.

> On the other hand, reading the output of ls(1) in a system with free
> use of subsections provides useful information.

I flatly deny that argument.  The Solaris list of suffixes deleted
above does not tell me anything of value.  It's just a list of
random sources of software, often with arbitrary partitions, and
many of the suffixes are non-descriptive.

[...]
>> On the other hand, naming manual pages after symbolic constants
>> or after type names is so unsual that i doubt any scheme exists
>> for that.

> Actually, man3type exists in several systems.  Solaris has it

Not on the Solaris 11 system i have access to:

  > uname -a
  SunOS unstable11s 5.11 11.3 sun4u sparc SUNW,SPARC-Enterprise
  > man -K e | grep 3type
  [ no output]
  > ls -d /usr/share/man/*type*
  /usr/share/man/*type*: No such file or directory
  > ls -d /usr/gnu/share/man/*type*
  /usr/gnu/share/man/*type*: No such file or directory

> (I guess Illumos too),

Not according to https://src.illumos.org/source/search?path=3type
nor according to
https://src.illumos.org/source/xref/illumos-gate/usr/src/man/ .

Again, Solaris suffixes do *not* indicate logical subdivisions,
but only a rather fuzzy, arbitrary, and inconsistent "where we
got these files from", at least if i understand correctly.

> and I've seen it in other systems (something from Oracle, 
> IIRC).  libbsd(7) also documents types, although they put them in the 
> global namespace, which I think you and I agree that it's not quite 
> right because of "documentation about a non-function in man3".

That argument has no merit.  The pre-eminent rule is "all documentation
of function libraries belongs in section 3" and explaining data types
(typically defined by the function libraries in the same header files
as the functions) are clearly part of that.

The rule "all library documentation should be organized by
functions" is secondary to that, and the rule "all section 3
names should be function names" is merely a corollary to that.

So even if you throw the rule "all library dicumentation should
be organized by functions" overboard, that is still no excuse
for moving part of the library documentation out of section 3.
It also makes sense to keep it all together in the same section
because it is needed by exactly the same people (programmers)
in exactly the same way (when reading and writing code and
wondering what words contained in the code mean).

>> The most widely used way to look up manual pages
>> by the names of symbolic constants or type names probably is
>> using macro keys as implemented in the mandoc version of apropos(1).
>> That is used by most FreeBSD, OpenBSD, Alpine Linux, and Void Linux.
>> I admit that doesn't qualify as "widely used", but "most widely used"
>> is probably true all the same.  ;-)

> That leaves out man(7).

Yes.  Searching for preprocessor constants and searching for data
type names are essentially semantic search features.  So it is
your choice to pick a 197x-era markup language that does not provide
semantic markup but only physical markup.  But than it feels
irrational to me to turn around and complain not getting semantic
search.  Unless you are a Prime Minister, you cannot have your cake
and eat it.

Trying to work around the lack of semantic markup by moving
everything into the manual page names feels like very poor design
to me.

> And types tend to be not very well documented if they are
> documented as part of a function page.

We disagree about that, and i won't repeat my full explanation why.
I'll repeat only this one aspect: There are few syntax elements where
context matters as much as for types; types live by how they are
used.  Separating their description from the functions using them
is a disservice to users, forcing them to jump around various pages
instead of having explained together what belongs together.

> And they also tend to be documented several times (out of sync,
> of course).

Any kind of documentation needs discipline and diligence and
maintenance, and none of it can ever be perfect.  But that's no
excuse for artificially tearing apart what users need together.

Yours,
  Ingo

Semantic man(7) markup (was: Linux man-pages Makefile portability)

[G. Branden Robinson]

[-- Attachment #1: Type: text/plain, Size: 5353 bytes --]

At 2022-07-24T17:57:31+0200, Ingo Schwarze wrote:
> Alejandro Colomar wrote on Sun, Jul 24, 2022 at 01:09:23PM +0200:
> > On 7/23/22 20:16, Ingo Schwarze wrote:
> >> The most widely used way to look up manual pages by the names of
> >> symbolic constants or type names probably is using macro keys as
> >> implemented in the mandoc version of apropos(1).  That is used by
> >> most FreeBSD, OpenBSD, Alpine Linux, and Void Linux.  I admit that
> >> doesn't qualify as "widely used", but "most widely used" is
> >> probably true all the same.  ;-)
> 
> > That leaves out man(7).

Perhaps not for long...

> Yes.  Searching for preprocessor constants and searching for data
> type names are essentially semantic search features.  So it is
> your choice to pick a 197x-era markup language that does not provide
> semantic markup but only physical markup.  But than it feels
> irrational to me to turn around and complain not getting semantic
> search.  Unless you are a Prime Minister, you cannot have your cake
> and eat it.
> 
> Trying to work around the lack of semantic markup by moving
> everything into the manual page names feels like very poor design
> to me.

It will not surprise, but might horrify, Ingo to learn that I have an
idea for how to add semantic markup to man(7).

Consider this hypothetical example.

  $ cat man3/man-pages.man
  .DC type B
  .DC field I
  $ cat man3/tm.3type
  .so man3/man-pages.man
[...]
  .SH DESCRIPTION
  .TG type "struct tm"
  describes time, broken down into distinct components.
  .PP
  .TG field tm_isdst
  describes wether daylight saving time is in effect at the time
  described.
[...]

Here, "DC" means "define class", a class of tags.  "TG", if one could
not guess, declares a tag of the type in its first argument with the
remaining arguments being the content thus tagged.

Returning to "DC", we see that it takes a second argument naming a macro
to call which will then apply any desired presentational markup to style
the tagged word.  This second argument need not be present.  In other
words, tagged content need not be visually distinct from its
surroundings.  Even in that event, it can still be useful; see #1 below.

Further, it will be obvious to the experienced *roff user that the macro
called by DC to style the applicable arguments given to TG need not even
be part of the man(7) language.

You could populate "man-pages.man" like this.

  $ cat man3/man-pages.man
  .de CW
  .  ie t \&\f[CR]\\$*\f[]
  .  el   \&\\$*
  ..
  .DC type CW
  .DC field I

This technique breaks the stranglehold of the man(7) font selection
macros.  (You're still limited by the output device's font repertoire,
however.)  If rendering to PostScript or PDF, you could decide to style
certain tags in Zapf Chancery Medium italic, if you wished.  (I cannot
warrant that you won't get yelled at.)

Here are a few perhaps less obvious things this approach would offer.

1.  It enables keyword search by tag.  Whatever does the searching need
    only look for "TG" calls, match the class argument, and return the
    remainder.  A search could be narrowed by limiting both the class
    _and_ the keyword arguments of course, perhaps to answer questions
    like "what pages use 'stat' as data type?".

2.  Degraded operation for other/older man(7) implementations is
    straightforward.  'DC' can be completely ignored.  'TG' can be
    defined as follows.

    .de TG
    \&\\$*
    ..

    or, for truly bloody-minded portability, thus.

    .de TG
    \&\\$1 \\$2 \\$3 \\$4 \\$5 \\$6 \\$7 \\$8 \\$9
    ..

3.  Everyday man(7) page authors need only learn 'TG' and the available
    list of keywords for the suite of man pages to which they are
    contributing.  Hammering out the repertoire of available tag classes
    and the surely monumental bikeshedding of text styling decisions to
    be associated with each tag class is delegated to the project that
    chooses to define them.  The man(7) macro package itself will impose
    no policy and may not even define any tag classes to start with.
    (groff would have some for its own man pages, of course, as I would
    expect Linux man-pages to do.)

4.  Site admins offended at the styling decisions undertaken by various
    projects could reliably override them by editing the files sourced
    by the relevant man pages.  Maybe those should live in /etc rather
    than the man page hierarchy proper.

5.  Misspelling a tag class or using an unavailable one is an error that
    would be easily diagnosed and reported.

To reiterate, groff man(7) would impose no policy regarding the tag
classes or their rendering on anyone.  It similarly would escape the
ongoing problem that mdoc(7) chose for itself by administering
centralized authoritative lists of standards documents, operating system
releases, and other lexica.  Tagful man(7) pages under my proposal would
opt into whatever keyword/class discipline they desire, or not at all.

I am not wedded to the nomenclature for the included files, nor the `DC`
or `TG` macros, except to note that the macro names are available.
(`DT`, putatively for "define tag", is not.  It is already taken.)

I stand ready for the hail of rotten tomatoes.

Regards,
Branden

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Semantic man(7) markup (was: Linux man-pages Makefile portability)

[Ingo Schwarze]

Hi Branden,

> I stand ready for the hail of rotten tomatoes.

I hope you are not surprised that i'll try to answer constructively.

Even though i'm not sure how relevent such an early design draft
is for linux-man@ or man-db-devel@, i'm not trimming the distribution
list just yet, but you might perhaps wish to consider choosing one
appropriate forum rather than cross-posting a design effort that is
likely to be long and difficult to three lists.

G. Branden Robinson wrote on Sun, Jul 24, 2022 at 12:29:47PM -0500:

> It will not surprise, but might horrify, Ingo to learn that I have an
> idea for how to add semantic markup to man(7).

It does not surprise me, you have already moved ahead with adding new
syntax to man(7), though so far in a piece-meal manner.  It would only
horrify me if this were done badly.

Since this is an early draft, it is not a surpise that i will raise
several points of the kind "won't work as you apparently hope" below.
That's not intended as denigration.  In an early design phase, it
is normal that not every idea is viable.  The job is to find the
working ideas in a larger pool of potential ideas and hammer them
into a working whole of minimal complexity that does the job.

> Consider this hypothetical example.
> 
>   $ cat man3/man-pages.man
>   .DC type B
>   .DC field I
>   $ cat man3/tm.3type
>   .so man3/man-pages.man

There is a danger of collisions here between the helper files of
different software projects.  That's maybe manageable; page names
can already collide right now.  Do such helper files really need to
be section-specific?  mdoc(7) teaches that in practice, there are
basically two markup domains: command line utilities and library
functions.  A possible third domain, kernel configuartion directives,
leads a fringe existence at best.  In practice, combining both
domains in one namespace poses no problems.

Besides, you will hardly be surprised that i deem making the
styles configurable detrimental in the first place.  The whole
point of having such styles is enabling a uniform, consistent
user experience.

> [...]
>   .SH DESCRIPTION
>   .TG type "struct tm"

This utterly breaks compatibility.
On formatters not suuporting .TG, the most important
content of the dpocument will simply disappear.

Even if you would somehow manage to get a fallback

  .de TG
  \&\\$2
  ..

in place - and so far, i have no idea how you could do that portably
on an arbitrary and potentially unknown formatter - that would
leave "struct tm" with no markup whatsoever.

One obvious idea to trivially improve the design is to use the
equivalent syntax

  .TG type
  .I "struct tm"

with the syntax constraint ".TG needs to be followed by a font macro"
and the semantic effect "in that case, the custom style overrides
the default style", reminiscent of how CSS can be used to override
the default styles of HTML elements.

This suggestion for improvement is not yet a definitive recommendation
though, it is merely another idea for the pool.

Let me also note that the markup overhead for your proposed scheme
is so far at least three times as large as for mdoc(7).
With modc(7), marking up a work usually requires one two-letter macro
as the markup overhead.  Your scheme requites the tag (which will
usually be at least four letters, sometimes more) in addition.

For spacing and punctuation control, mdoc(7) usually requires
no furthes overhead, except occasionally two letters for Ns.
It's not yet clear how your scheme will fare in that respect
because you proposed no mechanism for spacing and punction control
yet, but that will likely require some overhead, too.

>   describes time, broken down into distinct components.
>   .PP
>   .TG field tm_isdst
>   describes wether daylight saving time is in effect at the time
>   described.
> [...]
> 
> Here, "DC" means "define class", a class of tags.  "TG", if one could
> not guess, declares a tag of the type in its first argument with the
> remaining arguments being the content thus tagged.
> 
> Returning to "DC", we see that it takes a second argument naming a macro
> to call which will then apply any desired presentational markup to style
> the tagged word.  This second argument need not be present.  In other
> words, tagged content need not be visually distinct from its
> surroundings.  Even in that event, it can still be useful; see #1 below.
> 
> Further, it will be obvious to the experienced *roff user that the macro
> called by DC to style the applicable arguments given to TG need not even
> be part of the man(7) language.
> 
> You could populate "man-pages.man" like this.
> 
>   $ cat man3/man-pages.man
>   .de CW
>   .  ie t \&\f[CR]\\$*\f[]
>   .  el   \&\\$*
>   ..
>   .DC type CW
>   .DC field I
> 
> This technique breaks the stranglehold of the man(7) font selection
> macros.  (You're still limited by the output device's font repertoire,
> however.)

I fear you are missing the main aspect that makes using the man(7)
font macros complicated: spacing and punctuation.

Let us start with a moderately simple, everyday example:

  .Ft double Fn atan2 "double y" "double x"

What would that become with your proposed syntax?

  .TG type double
  .TG funcname atan2
  (
  .TG funcarg "double y"
  ,
  .TG funcarg "double x"
  )

What is your plan for that?  Recommending \c everywhere?
Or mdoc(7)-style punctuation handling like

  .TG type double
  .TG funcname atan2 NS
  .TG funcarg ( "double y" ,
  .TG funcarg "double x" )

Practical work shows these trivial cases of spacing around common
punctuation like parentheses and commas are not the only ones that
occur.  Sometimes, spacing needs to be inserted or suppressed in ways
that differ from default rules.  mdoc(7) has several mechanisms for
that, the most important one being .Ns.  Your new scheme might choose
a different approach, but i predict how well that mechanism is going
to be will be one determining factor of how pleasant or unpleasant
the language will become for authors and maintainers.

> If rendering to PostScript or PDF, you could decide to style
> certain tags in Zapf Chancery Medium italic, if you wished.  (I cannot
> warrant that you won't get yelled at.)
> 
> Here are a few perhaps less obvious things this approach would offer.
> 
> 1.  It enables keyword search by tag.  Whatever does the searching need
>     only look for "TG" calls, match the class argument, and return the
>     remainder.  A search could be narrowed by limiting both the class
>     _and_ the keyword arguments of course, perhaps to answer questions
>     like "what pages use 'stat' as data type?".

Sure, just like

  man -k Vt=stat

with mandoc.

> 2.  Degraded operation for other/older man(7) implementations is
>     straightforward.  'DC' can be completely ignored.  'TG' can be
>     defined as follows.
> 
>     .de TG
>     \&\\$*
>     ..

No, that is plain wrong.  It results in

     type struct tm describes time, broken down into distinct components.
     field tm_isdst describes wether daylight saving time is in effect
     at the time described.

You certainly do not want to print out \\$1.

>     or, for truly bloody-minded portability, thus.
> 
>     .de TG
>     \&\\$1 \\$2 \\$3 \\$4 \\$5 \\$6 \\$7 \\$8 \\$9
>     ..

That would exclude any kind of mdoc(7)-style punctuation handling,
making spacing control much harder to implement.  I'm not saying
impossible, but harder.  One way out might be to use \\$2 only
and require quoting of multi-word arguments.

> 3.  Everyday man(7) page authors need only learn 'TG' and the available
>     list of keywords for the suite of man pages to which they are
>     contributing.  Hammering out the repertoire of available tag classes
>     and the surely monumental bikeshedding of text styling decisions to
>     be associated with each tag class is delegated to the project that
>     chooses to define them.

I fear that is a terrible idea.  The DocBook design fiasco teaches
us that even a single committee is easily capable of utterly screwing
up markup consistency.  Leaving the markup lexicon up to individual
projects guarantees hopeless global inconsistency.  Different projects
will mark up different syntax elements, assign different tags for the
same purpose, and assign conflicting styles to the same tags.

Consequently, semantic searching will *not* work at all.  You cannot
even search for a function name if one group tags it as "function"
and another as "funcname" and another as "sub" and another as "procname".
Rendering will be wildly inconsistent because every group of pages,
large or small, comes with its own styles - and larger projects are
unlikely to remain consistent even within themselves.  Due to the
undefined nature of tag names, distributions are unable to fix up
the mess, too.

>     The man(7) macro package itself will impose
>     no policy and may not even define any tag classes to start with.

Branden, seriously, standardizing this is your number two most important
job during this design.  The top one priority, of course, is to
choose a syntax that is short, simple, robust, esay to read, and
easy to write for both usual and unusual use cases, and - if you
want to continue calling the language man(7) - backward compatible.

>     (groff would have some for its own man pages, of course, as I would
>     expect Linux man-pages to do.)
> 
> 4.  Site admins offended at the styling decisions undertaken by various
>     projects could reliably override them by editing the files sourced
>     by the relevant man pages.

That's illusionary until you provide an authoritative and long-time
stable list of valid tags.
Failing to do so is one among the reasons why SGML and XML turned
out as hard to use in practice, and one of the reasons why HTML
ultimately gave up on being XML and instead defined an exhaustive
list of standard elements.

Besides, on one of my machines, i currently have almost 600 packages
installed even though lots of things that are packages on Linux do
*not* count as packages on OpenBSD but are part of the base system
that cannot ever be uninstalled, including thing like compilers and
other development tools and lots of Internet daemons like web server,
mail server, routing daemons, DNS servers and much more.  And yet,
i still have almost 600 packages.  As a rule of thumb, on average one
third of packages have manual pages.  So you want me to edit 200 config
files and resolve conflicts in them whenever updating packages?
I'm not yet convinced that's a very good plan.  :-o

>     Maybe those should live in /etc rather
>     than the man page hierarchy proper.
> 
> 5.  Misspelling a tag class or using an unavailable one is an error that
>     would be easily diagnosed and reported.
> 
> To reiterate, groff man(7) would impose no policy regarding the tag
> classes or their rendering on anyone.  It similarly would escape the
> ongoing problem that mdoc(7) chose for itself by administering
> centralized authoritative lists of standards documents, operating system
> releases, and other lexica.

I freely admit that .St .At .Bx .Nx .Fx .Ox .Dx .Lb are among the least
well designed parts of mdoc(7).  But that design problem is completely
unrelated to the fact that the list of macros (in mdoc(7) parlance)
or tags (in your proposed terminology) needs to be fixed.  That's *the*
central purpuse of a markup language.  Personally, i would go as far
as calling a non-extensible markup language usually better than an
extensible one, but even for an extensible markup language, the
quality of the standard lexicon is the second highest priority,
right after the quality of the basic syntax.

> Tagful man(7) pages under my proposal would
> opt into whatever keyword/class discipline they desire, or not at all.
> 
> I am not wedded to the nomenclature for the included files, nor the `DC`
> or `TG` macros, except to note that the macro names are available.
> (`DT`, putatively for "define tag", is not.  It is already taken.)

Given how utterly your .TG proposal is breaking compatibility,
i wonder whether it would be wiser to give your new macroset a new
name and leave the legacy man(7) language alone.  That would allow
cleaning up some design deficiencies of man(7) at the same time,
for example:

 * properly define display macros, a purpose now served in a
   manner the is partially quirky, partially redundant, and yet
   incomplete by .RS and .EX
 * get rid of the font alternation macros; it is well known that
   even though those squat on a considerable parcel of namespace,
   they only solve part of their task - the infamous three-font
   issue; your proposal needs a solution for that anyway that
   still needs to be developed
 * get rid of the redundancy among .TP .TQ .IP .HP
 * get rid of legacy macros like .AT .DT .LP .P .SM .SB .UC
   that are useless in practice
 * get rid of misdesigned macros like .SY

Please also consider that having a mix of semantic and presentational
markup in a language is usually not the best idea.  As a prominent
example, after a considerable amount of handwringing that lasted for
many years, HTML finally abolished presentational markup completely.
For similar reasons, we somewhat discourage using .Em and .Sy in mdoc(7),
except for purposes similar to <em>, <i>, <strong>, and <b> in HTML 5.

If you choose a new name, compatibility concerns vanish instantly.
But of course, once you publicly announce "this is now ready for
production", stability becomes important again, so the design should
be well thought out from the start.  Finding a major design issue
or design gap after it has been in production for, say, two years
would be very unfortunate.

On the other hand, if you want to continue calling it man(7), then
the logical consequence is that compatibility concerns become
paramount.

Yours,
  Ingo

Re: Linux man-pages Makefile portability

[Colin Watson]

On Sun, Jul 24, 2022 at 01:09:23PM +0200, Alejandro Colomar (man-pages) wrote:
> [TO += Colin; CC += man-dv-devel@;
> there's a bug in man(1)'s autocomplete]

man-db doesn't ship its own autocomplete code at the moment.  This is
probably shipped by some other package in your distribution, although I
don't know exactly what that would be (maybe bash-completion?).

-- 
Colin Watson (he/him)                              [cjwatson@debian.org]

bash-completion doesn't support man subsections (was: Linux man-pages Makefile portability)

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 793 bytes --]

Hi Colin,

On 7/25/22 11:28, Colin Watson wrote:
> On Sun, Jul 24, 2022 at 01:09:23PM +0200, Alejandro Colomar (man-pages) wrote:
>> [TO += Colin; CC += man-dv-devel@;
>> there's a bug in man(1)'s autocomplete]
> 
> man-db doesn't ship its own autocomplete code at the moment.
> This is probably shipped by some other package in your distribution,

Heh, I bet it's the same as yours, as you show up as the maintainer ;)

$ apt-cache show man-db | grep Maintainer
Maintainer: Colin Watson <cjwatson@debian.org>

Yeah, I use Debian (Sid).


> although I don't know exactly what that would be (maybe bash-completion?).
Reported to upstream:
<https://github.com/scop/bash-completion/issues/778>

Cheers,

Alex

-- 
Alejandro Colomar
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]