Stop compressing manual pages (was: Bug#1123959: manpages: Please consider shipping uncompressed man pages)

Stop compressing manual pages (was: Bug#1123959: manpages: Please consider shipping uncompressed man pages)

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 5329 bytes --]

Hi,

On Thu, Dec 25, 2025 at 02:47:33PM +0100, Dr. Tobias Quathamer wrote:
> Am 25.12.25 um 12:20 schrieb Alejandro Colomar:
> > Hello Helge, Tobias,
> > 
> > On Thu, Dec 25, 2025 at 06:07:57AM +0000, Helge Kreutzmann wrote:
> > > Hello Tobias,
> > > if you look at mansect(1), the example given does not work in Debian.
> > > I reported this upstream and got the following reply:
> > > 
> > > > The issue is that Debian compresses manual pages.  Please consider
> > > > changing the policy to not compress manual pages.  The storage savings
> > > > are irrelevant in this age.
> > > 
> > > Could you consider this?
> > 
> > Thanks!
> > 
> > Indeed, compressed manual pages are a pain to work with.  You can't use
> > regular Unix tools to work with them.  With uncompressed manual pages,
> > You can go to /usr/share/man, and run a pipe of programs to do a complex
> > search.  With tools like zgrep(1) and zcat(1), you can do some stuff,
> > but not everything.
> 
> Hi Helge and Alex,
> 
> thanks for your bug report and the provided statistics. I haven't thought
> about this up until now, because it violates Debian Policy. Quoting from
> Section 12.1
> (https://www.debian.org/doc/debian-policy/ch-docs.html#manual-pages):
> 
> "Manual pages should be installed compressed using gzip -9."
> 
> And regarding the terminology using the word "should", this is defined in
> section 1.1 (https://www.debian.org/doc/debian-policy/ch-scope.html#scope):
> 
> "The terms should and should not, and the adjective recommended, denote best
> practices. Non-conformance with these guidelines will generally be
> considered a bug, but will not necessarily render a package unsuitable for
> distribution. These statements correspond to bug severities of important,
> normal, and minor. They are collectively called Policy recommendations."
> 
> So by not compressing the man pages, the Debian package would introduce a
> bug. Moreover, I'd have to explicitely opt out of automatic compression in
> the build stage of the package.
> 
> All of this is doable, of course. But I'm a bit hesitant with just making
> the switch, given that the manpages package is certainly the package with
> the most man pages in the Debian ecosystem -- by a large margin.
> 
> So it might be better to discuss the pros and cons in a broader audience,
> trying to understand why the compression has been chosen initially. Maybe
> only due to disk space limitations back then, but maybe there are other
> reasons as well -- which might still be valid today.

Yup, I'd like that policy to change.  I've added debian-policy@ to this
mail (and also linux-man@).

For those reading only since this email, please have a look at
<https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1123959>, which
itself also references an discussion in the Linux man-pages project from
a couple of years ago:
<https://lore.kernel.org/linux-man/c8cf5be0-04e7-f0a1-179f-eada6182c33e@gmail.com/T/#m272e6ee8939d0836999dd8bb28f2e0e94f48dfc7>.

I'll paste again the numbers:

        $ sudo make install-man prefix=/opt/local/man/gz__1 -j LINK_PAGES=symlink Z=.gz  GZIPFLAGS=-1  | wc -l
        2571
        $ sudo make install-man prefix=/opt/local/man/gz__9 -j LINK_PAGES=symlink Z=.gz  GZIPFLAGS=-9  | wc -l
        2571
        $ sudo make install-man prefix=/opt/local/man/man__ -j LINK_PAGES=symlink Z=                   | wc -l
        2571

        $ du -sh /opt/local/man/*
        5.7M    /opt/local/man/gz__1
        5.5M    /opt/local/man/gz__9
        5.5M    /opt/local/man/gz___
        9.4M    /opt/local/man/man__

        $ export MANPATH=/opt/local/man/gz__1/share/man
        $ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
        17; 0.21
        $ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do gzip -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '"
        17; 1.16

        $ export MANPATH=/opt/local/man/gz__9/share/man
        $ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
        17; 0.20
        $ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do gzip -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '"
        17; 1.17

        $ export MANPATH=/opt/local/man/man__/share/man
        $ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
        17; 0.55
        $ /bin/time -f %e dash -c "find $MANPATH -type f | xargs -P0 grep -l RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
        17; 0.01

Using uncompressed manual pages is both faster and simpler, by orders of
magnitude, when doing complex searches with pipelines.  In the simple
cases where man(1) is enough, the speed is in the same order of
magnitude.

Also, the compression only cuts storage by half, so not even an order of
magnitude.  In this age, where storage is relatively cheap, systems that
have manual pages installed most likely have room enough for the
uncompressed pages.

Please change Debian policy 12.1 ("Manual pages") to recommend
uncompressed pages.


Have a lovely day!
Alex

> 
> Regards,
> Tobias

-- 
<https://www.alejandro-colomar.es>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Stop compressing manual pages

[Russ Allbery]

Alejandro Colomar <alx@kernel.org> writes:
> On Thu, Dec 25, 2025 at 02:47:33PM +0100, Dr. Tobias Quathamer wrote:
>> Am 25.12.25 um 12:20 schrieb Alejandro Colomar:

>>> Indeed, compressed manual pages are a pain to work with.  You can't use
>>> regular Unix tools to work with them.  With uncompressed manual pages,
>>> You can go to /usr/share/man, and run a pipe of programs to do a complex
>>> search.  With tools like zgrep(1) and zcat(1), you can do some stuff,
>>> but not everything.

[...]

>> thanks for your bug report and the provided statistics. I haven't thought
>> about this up until now, because it violates Debian Policy. Quoting from
>> Section 12.1
>> (https://www.debian.org/doc/debian-policy/ch-docs.html#manual-pages):
>> 
>> "Manual pages should be installed compressed using gzip -9."

[...]

> Yup, I'd like that policy to change.  I've added debian-policy@ to this
> mail (and also linux-man@).

Colin, do you have an opinion on this as the man-db maintainer? The
software you maintain is probably the primary consumer by a significant
margin of the installed manual pages.

The rationale in Debian for compressing documentation in general is for
embedded systems and other small installations, and it applies to just
about anything that can be safely compressed (manual pages are only one
example). But this rule also predates such facilities as the nodoc build
profile, and is several decades old and thus predates the growth in
storage size even in small embedded environments that has significantly
outpaced the size of text-adjacent documents. I would definitely want to
get feedback from embedded folks before changing this rule, but at least
at first glance it sounds like a reasonable request worth considering.

-- 
Russ Allbery (rra@debian.org)              <https://www.eyrie.org/~eagle/>

Re: Stop compressing manual pages

[G. Branden Robinson]

[-- Attachment #1: Type: text/plain, Size: 2284 bytes --]

At 2025-12-25T11:06:34-0800, Russ Allbery wrote:
> > Yup, I'd like that policy to change.  I've added debian-policy@ to
> > this mail (and also linux-man@).
>
> The rationale in Debian for compressing documentation in general is
> for embedded systems and other small installations, and it applies to
> just about anything that can be safely compressed (manual pages are
> only one example).  But this rule also predates such facilities as the
> nodoc build profile, and is several decades old and thus predates the
> growth in storage size even in small embedded environments that has
> significantly outpaced the size of text-adjacent documents.  I would
> definitely want to get feedback from embedded folks before changing
> this rule, but at least at first glance it sounds like a reasonable
> request worth considering.

I'd add that, in contrast to the mid-1990s when Debian's man page
compression policy was promulgated--my recollection is that it was an
early, early decision, already in place when I started using Debian in
January 1996--transparent compression is now an oft-implemented feature
of file systems, including some that are popular in embedded systems,
such as JFFS2, where it's been the case for at least 19 years.[1]

Further, the selection of compression algorithm and container format has
become a popular site for partisan battles over the same.[2][3][4][5][6]
Since Debian already generates sufficient partisan battles over issues
specific to our practices, it might be advantageous to abandon this one.
(Speaking for myself, I find deflate/gzip satisfactory, and I intend to
release groff 1.24.0 as a gzipped tape archive.)

Adopting this change would enable man-db man(1) to discard the
zsoelim(1) tool, simplifying the code base and logic depending on this
tool--but I defer to Colin's judgment of how advantageous that'd be.

Regards,
Branden

[1] https://lwn.net/Articles/219827/
[2] https://linuxreviews.org/Comparison_of_Compression_Algorithms
[3] https://www.nongnu.org/lzip/xz_inadequate.html
[4] https://engineering.fb.com/2016/08/31/core-infra/smaller-and-faster-data-compression-with-zstandard/
[5] https://www.reddit.com/r/archlinux/comments/eiia99/zst_packages_consistently_larger_than_xz/
[6] https://sysdfree.wordpress.com/2020/01/04/293/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Stop compressing manual pages

[Marc Haber]

On Thu, Dec 25, 2025 at 11:06:34AM -0800, Russ Allbery wrote:
>The rationale in Debian for compressing documentation in general is for
>embedded systems and other small installations, and it applies to just
>about anything that can be safely compressed (manual pages are only one
>example)

In embedded and other small installations, I'd prefer everyting under 
/usr/share/man just excluded from installations on the package manager 
level, but I never have been able to find out how to rub dpkg 
--path-exclude the right way to do that. While we're at it, exclude 
everything under /usr/share/doc with the exception of 
/usr/share/doc/*/copyright*

Having html docs or others compressed in /usr/share/doc is another pet 
peeve that makes it harder to use the on-disk docs.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: Stop compressing manual pages

[Simon McVittie]

On Fri, 26 Dec 2025 at 11:51:40 +0100, Marc Haber wrote:
>In embedded and other small installations, I'd prefer everyting under 
>/usr/share/man just excluded from installations on the package manager 
>level, but I never have been able to find out how to rub dpkg 
>--path-exclude the right way to do that. While we're at it, exclude 
>everything under /usr/share/doc with the exception of 
>/usr/share/doc/*/copyright*

Here's how the semi-official debian:sid-slim, etc. Docker images do it:

https://github.com/debuerreotype/debuerreotype/blob/master/scripts/debuerreotype-slimify
https://github.com/debuerreotype/debuerreotype/blob/master/scripts/.slimify-excludes
(each non-comment line $x becomes "path-exclude $x" in /etc/dpkg.cfg.d/docker)
https://github.com/debuerreotype/debuerreotype/blob/master/scripts/.slimify-includes
(each non-comment line $x becomes "path-include $x" in /etc/dpkg.cfg.d/docker)

     smcv

Re: Stop compressing manual pages

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 2095 bytes --]

Hi Colin,

On 2025-12-25T11:06:34-0800, Russ Allbery wrote:
> Alejandro Colomar <alx@kernel.org> writes:
> > On Thu, Dec 25, 2025 at 02:47:33PM +0100, Dr. Tobias Quathamer wrote:
> >> Am 25.12.25 um 12:20 schrieb Alejandro Colomar:
> 
> >>> Indeed, compressed manual pages are a pain to work with.  You can't use
> >>> regular Unix tools to work with them.  With uncompressed manual pages,
> >>> You can go to /usr/share/man, and run a pipe of programs to do a complex
> >>> search.  With tools like zgrep(1) and zcat(1), you can do some stuff,
> >>> but not everything.
> 
> [...]
> 
> >> thanks for your bug report and the provided statistics. I haven't thought
> >> about this up until now, because it violates Debian Policy. Quoting from
> >> Section 12.1
> >> (https://www.debian.org/doc/debian-policy/ch-docs.html#manual-pages):
> >> 
> >> "Manual pages should be installed compressed using gzip -9."
> 
> [...]
> 
> > Yup, I'd like that policy to change.  I've added debian-policy@ to this
> > mail (and also linux-man@).
> 
> Colin, do you have an opinion on this as the man-db maintainer? The
> software you maintain is probably the primary consumer by a significant
> margin of the installed manual pages.

Ping.


Have a lovely day!
Alex

> 
> The rationale in Debian for compressing documentation in general is for
> embedded systems and other small installations, and it applies to just
> about anything that can be safely compressed (manual pages are only one
> example). But this rule also predates such facilities as the nodoc build
> profile, and is several decades old and thus predates the growth in
> storage size even in small embedded environments that has significantly
> outpaced the size of text-adjacent documents. I would definitely want to
> get feedback from embedded folks before changing this rule, but at least
> at first glance it sounds like a reasonable request worth considering.
> 
> -- 
> Russ Allbery (rra@debian.org)              <https://www.eyrie.org/~eagle/>

-- 
<https://www.alejandro-colomar.es>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Stop compressing manual pages

[Colin Watson]

On Thu, Dec 25, 2025 at 11:06:34AM -0800, Russ Allbery wrote:
> Alejandro Colomar <alx@kernel.org> writes:
> > On Thu, Dec 25, 2025 at 02:47:33PM +0100, Dr. Tobias Quathamer wrote:
> >> thanks for your bug report and the provided statistics. I haven't thought
> >> about this up until now, because it violates Debian Policy. Quoting from
> >> Section 12.1
> >> (https://www.debian.org/doc/debian-policy/ch-docs.html#manual-pages):
> >> 
> >> "Manual pages should be installed compressed using gzip -9."
> 
> [...]
> 
> > Yup, I'd like that policy to change.  I've added debian-policy@ to this
> > mail (and also linux-man@).
> 
> Colin, do you have an opinion on this as the man-db maintainer? The
> software you maintain is probably the primary consumer by a significant
> margin of the installed manual pages.
> 
> The rationale in Debian for compressing documentation in general is for
> embedded systems and other small installations, and it applies to just
> about anything that can be safely compressed (manual pages are only one
> example). But this rule also predates such facilities as the nodoc build
> profile, and is several decades old and thus predates the growth in
> storage size even in small embedded environments that has significantly
> outpaced the size of text-adjacent documents. I would definitely want to
> get feedback from embedded folks before changing this rule, but at least
> at first glance it sounds like a reasonable request worth considering.

Sorry for being slow to reply to this.

It's not something I feel as strongly about as Alejandro: in man-db 
we've already paid most of the costs (in terms of software complexity) 
of making compressed manual pages work well, and we'll have to keep that 
code around for the foreseeable future no matter what.  But it's true 
that there are continuing annoyances with ad-hoc shell pipelines.

Let's put it this way: as man-db maintainer, I have no objection to 
recommending uncompressed manual pages.  Sort of +0 or +0.5.  I 
definitely agree that embedded folks ought to be consulted, if they 
aren't already simply excluding manual pages entirely.

-- 
Colin Watson (he/him)                              [cjwatson@debian.org]

Re: Stop compressing manual pages

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 2612 bytes --]

Hi Colin,

On 2026-04-20T11:14:52+0100, Colin Watson wrote:
> On Thu, Dec 25, 2025 at 11:06:34AM -0800, Russ Allbery wrote:
> > Alejandro Colomar <alx@kernel.org> writes:
> > > On Thu, Dec 25, 2025 at 02:47:33PM +0100, Dr. Tobias Quathamer wrote:
> > >> thanks for your bug report and the provided statistics. I haven't thought
> > >> about this up until now, because it violates Debian Policy. Quoting from
> > >> Section 12.1
> > >> (https://www.debian.org/doc/debian-policy/ch-docs.html#manual-pages):
> > >> 
> > >> "Manual pages should be installed compressed using gzip -9."
> > 
> > [...]
> > 
> > > Yup, I'd like that policy to change.  I've added debian-policy@ to this
> > > mail (and also linux-man@).
> > 
> > Colin, do you have an opinion on this as the man-db maintainer? The
> > software you maintain is probably the primary consumer by a significant
> > margin of the installed manual pages.
> > 
> > The rationale in Debian for compressing documentation in general is for
> > embedded systems and other small installations, and it applies to just
> > about anything that can be safely compressed (manual pages are only one
> > example). But this rule also predates such facilities as the nodoc build
> > profile, and is several decades old and thus predates the growth in
> > storage size even in small embedded environments that has significantly
> > outpaced the size of text-adjacent documents. I would definitely want to
> > get feedback from embedded folks before changing this rule, but at least
> > at first glance it sounds like a reasonable request worth considering.
> 
> Sorry for being slow to reply to this.
> 
> It's not something I feel as strongly about as Alejandro: in man-db 
> we've already paid most of the costs (in terms of software complexity) 
> of making compressed manual pages work well, and we'll have to keep that 
> code around for the foreseeable future no matter what.  But it's true 
> that there are continuing annoyances with ad-hoc shell pipelines.
> 
> Let's put it this way: as man-db maintainer, I have no objection to 
> recommending uncompressed manual pages.  Sort of +0 or +0.5.

Thanks!

>  I 
> definitely agree that embedded folks ought to be consulted, if they 
> aren't already simply excluding manual pages entirely.

Does anyone know how to contact them?  Could you please add them to CC
(some key people and/or a mailing list)?


Have a lovely night!
Alex

> 
> -- 
> Colin Watson (he/him)                              [cjwatson@debian.org]

-- 
<https://www.alejandro-colomar.es>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]