* Stop compressing manual pages (was: Bug#1123959: manpages: Please consider shipping uncompressed man pages) [not found] ` <fec615b5-af5b-46cd-ae09-d9343db6da77@debian.org> @ 2025-12-25 14:15 ` Alejandro Colomar 2025-12-25 19:06 ` Stop compressing manual pages Russ Allbery 0 siblings, 1 reply; 5+ messages in thread From: Alejandro Colomar @ 2025-12-25 14:15 UTC (permalink / raw) To: debian-policy, Dr. Tobias Quathamer Cc: linux-man, Helge Kreutzmann, G. Branden Robinson, Colin Watson [-- Attachment #1: Type: text/plain, Size: 5329 bytes --] Hi, On Thu, Dec 25, 2025 at 02:47:33PM +0100, Dr. Tobias Quathamer wrote: > Am 25.12.25 um 12:20 schrieb Alejandro Colomar: > > Hello Helge, Tobias, > > > > On Thu, Dec 25, 2025 at 06:07:57AM +0000, Helge Kreutzmann wrote: > > > Hello Tobias, > > > if you look at mansect(1), the example given does not work in Debian. > > > I reported this upstream and got the following reply: > > > > > > > The issue is that Debian compresses manual pages. Please consider > > > > changing the policy to not compress manual pages. The storage savings > > > > are irrelevant in this age. > > > > > > Could you consider this? > > > > Thanks! > > > > Indeed, compressed manual pages are a pain to work with. You can't use > > regular Unix tools to work with them. With uncompressed manual pages, > > You can go to /usr/share/man, and run a pipe of programs to do a complex > > search. With tools like zgrep(1) and zcat(1), you can do some stuff, > > but not everything. > > Hi Helge and Alex, > > thanks for your bug report and the provided statistics. I haven't thought > about this up until now, because it violates Debian Policy. Quoting from > Section 12.1 > (https://www.debian.org/doc/debian-policy/ch-docs.html#manual-pages): > > "Manual pages should be installed compressed using gzip -9." > > And regarding the terminology using the word "should", this is defined in > section 1.1 (https://www.debian.org/doc/debian-policy/ch-scope.html#scope): > > "The terms should and should not, and the adjective recommended, denote best > practices. Non-conformance with these guidelines will generally be > considered a bug, but will not necessarily render a package unsuitable for > distribution. These statements correspond to bug severities of important, > normal, and minor. They are collectively called Policy recommendations." > > So by not compressing the man pages, the Debian package would introduce a > bug. Moreover, I'd have to explicitely opt out of automatic compression in > the build stage of the package. > > All of this is doable, of course. But I'm a bit hesitant with just making > the switch, given that the manpages package is certainly the package with > the most man pages in the Debian ecosystem -- by a large margin. > > So it might be better to discuss the pros and cons in a broader audience, > trying to understand why the compression has been chosen initially. Maybe > only due to disk space limitations back then, but maybe there are other > reasons as well -- which might still be valid today. Yup, I'd like that policy to change. I've added debian-policy@ to this mail (and also linux-man@). For those reading only since this email, please have a look at <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1123959>, which itself also references an discussion in the Linux man-pages project from a couple of years ago: <https://lore.kernel.org/linux-man/c8cf5be0-04e7-f0a1-179f-eada6182c33e@gmail.com/T/#m272e6ee8939d0836999dd8bb28f2e0e94f48dfc7>. I'll paste again the numbers: $ sudo make install-man prefix=/opt/local/man/gz__1 -j LINK_PAGES=symlink Z=.gz GZIPFLAGS=-1 | wc -l 2571 $ sudo make install-man prefix=/opt/local/man/gz__9 -j LINK_PAGES=symlink Z=.gz GZIPFLAGS=-9 | wc -l 2571 $ sudo make install-man prefix=/opt/local/man/man__ -j LINK_PAGES=symlink Z= | wc -l 2571 $ du -sh /opt/local/man/* 5.7M /opt/local/man/gz__1 5.5M /opt/local/man/gz__9 5.5M /opt/local/man/gz___ 9.4M /opt/local/man/man__ $ export MANPATH=/opt/local/man/gz__1/share/man $ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '" 17; 0.21 $ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do gzip -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '" 17; 1.16 $ export MANPATH=/opt/local/man/gz__9/share/man $ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '" 17; 0.20 $ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do gzip -d - <\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs printf '%s; '" 17; 1.17 $ export MANPATH=/opt/local/man/man__/share/man $ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '" 17; 0.55 $ /bin/time -f %e dash -c "find $MANPATH -type f | xargs -P0 grep -l RLIMIT_NOFILE | wc -l | xargs printf '%s; '" 17; 0.01 Using uncompressed manual pages is both faster and simpler, by orders of magnitude, when doing complex searches with pipelines. In the simple cases where man(1) is enough, the speed is in the same order of magnitude. Also, the compression only cuts storage by half, so not even an order of magnitude. In this age, where storage is relatively cheap, systems that have manual pages installed most likely have room enough for the uncompressed pages. Please change Debian policy 12.1 ("Manual pages") to recommend uncompressed pages. Have a lovely day! Alex > > Regards, > Tobias -- <https://www.alejandro-colomar.es> [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Stop compressing manual pages 2025-12-25 14:15 ` Stop compressing manual pages (was: Bug#1123959: manpages: Please consider shipping uncompressed man pages) Alejandro Colomar @ 2025-12-25 19:06 ` Russ Allbery 2025-12-26 3:08 ` G. Branden Robinson 2025-12-26 10:51 ` Marc Haber 0 siblings, 2 replies; 5+ messages in thread From: Russ Allbery @ 2025-12-25 19:06 UTC (permalink / raw) To: Alejandro Colomar Cc: debian-policy, Dr. Tobias Quathamer, linux-man, Helge Kreutzmann, G. Branden Robinson, Colin Watson Alejandro Colomar <alx@kernel.org> writes: > On Thu, Dec 25, 2025 at 02:47:33PM +0100, Dr. Tobias Quathamer wrote: >> Am 25.12.25 um 12:20 schrieb Alejandro Colomar: >>> Indeed, compressed manual pages are a pain to work with. You can't use >>> regular Unix tools to work with them. With uncompressed manual pages, >>> You can go to /usr/share/man, and run a pipe of programs to do a complex >>> search. With tools like zgrep(1) and zcat(1), you can do some stuff, >>> but not everything. [...] >> thanks for your bug report and the provided statistics. I haven't thought >> about this up until now, because it violates Debian Policy. Quoting from >> Section 12.1 >> (https://www.debian.org/doc/debian-policy/ch-docs.html#manual-pages): >> >> "Manual pages should be installed compressed using gzip -9." [...] > Yup, I'd like that policy to change. I've added debian-policy@ to this > mail (and also linux-man@). Colin, do you have an opinion on this as the man-db maintainer? The software you maintain is probably the primary consumer by a significant margin of the installed manual pages. The rationale in Debian for compressing documentation in general is for embedded systems and other small installations, and it applies to just about anything that can be safely compressed (manual pages are only one example). But this rule also predates such facilities as the nodoc build profile, and is several decades old and thus predates the growth in storage size even in small embedded environments that has significantly outpaced the size of text-adjacent documents. I would definitely want to get feedback from embedded folks before changing this rule, but at least at first glance it sounds like a reasonable request worth considering. -- Russ Allbery (rra@debian.org) <https://www.eyrie.org/~eagle/> ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Stop compressing manual pages 2025-12-25 19:06 ` Stop compressing manual pages Russ Allbery @ 2025-12-26 3:08 ` G. Branden Robinson 2025-12-26 10:51 ` Marc Haber 1 sibling, 0 replies; 5+ messages in thread From: G. Branden Robinson @ 2025-12-26 3:08 UTC (permalink / raw) To: Russ Allbery Cc: Alejandro Colomar, debian-policy, Dr. Tobias Quathamer, linux-man, Helge Kreutzmann, Colin Watson [-- Attachment #1: Type: text/plain, Size: 2284 bytes --] At 2025-12-25T11:06:34-0800, Russ Allbery wrote: > > Yup, I'd like that policy to change. I've added debian-policy@ to > > this mail (and also linux-man@). > > The rationale in Debian for compressing documentation in general is > for embedded systems and other small installations, and it applies to > just about anything that can be safely compressed (manual pages are > only one example). But this rule also predates such facilities as the > nodoc build profile, and is several decades old and thus predates the > growth in storage size even in small embedded environments that has > significantly outpaced the size of text-adjacent documents. I would > definitely want to get feedback from embedded folks before changing > this rule, but at least at first glance it sounds like a reasonable > request worth considering. I'd add that, in contrast to the mid-1990s when Debian's man page compression policy was promulgated--my recollection is that it was an early, early decision, already in place when I started using Debian in January 1996--transparent compression is now an oft-implemented feature of file systems, including some that are popular in embedded systems, such as JFFS2, where it's been the case for at least 19 years.[1] Further, the selection of compression algorithm and container format has become a popular site for partisan battles over the same.[2][3][4][5][6] Since Debian already generates sufficient partisan battles over issues specific to our practices, it might be advantageous to abandon this one. (Speaking for myself, I find deflate/gzip satisfactory, and I intend to release groff 1.24.0 as a gzipped tape archive.) Adopting this change would enable man-db man(1) to discard the zsoelim(1) tool, simplifying the code base and logic depending on this tool--but I defer to Colin's judgment of how advantageous that'd be. Regards, Branden [1] https://lwn.net/Articles/219827/ [2] https://linuxreviews.org/Comparison_of_Compression_Algorithms [3] https://www.nongnu.org/lzip/xz_inadequate.html [4] https://engineering.fb.com/2016/08/31/core-infra/smaller-and-faster-data-compression-with-zstandard/ [5] https://www.reddit.com/r/archlinux/comments/eiia99/zst_packages_consistently_larger_than_xz/ [6] https://sysdfree.wordpress.com/2020/01/04/293/ [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Stop compressing manual pages 2025-12-25 19:06 ` Stop compressing manual pages Russ Allbery 2025-12-26 3:08 ` G. Branden Robinson @ 2025-12-26 10:51 ` Marc Haber 2025-12-26 14:52 ` Simon McVittie 1 sibling, 1 reply; 5+ messages in thread From: Marc Haber @ 2025-12-26 10:51 UTC (permalink / raw) To: Russ Allbery Cc: Alejandro Colomar, debian-policy, Dr. Tobias Quathamer, linux-man, Helge Kreutzmann, G. Branden Robinson, Colin Watson On Thu, Dec 25, 2025 at 11:06:34AM -0800, Russ Allbery wrote: >The rationale in Debian for compressing documentation in general is for >embedded systems and other small installations, and it applies to just >about anything that can be safely compressed (manual pages are only one >example) In embedded and other small installations, I'd prefer everyting under /usr/share/man just excluded from installations on the package manager level, but I never have been able to find out how to rub dpkg --path-exclude the right way to do that. While we're at it, exclude everything under /usr/share/doc with the exception of /usr/share/doc/*/copyright* Having html docs or others compressed in /usr/share/doc is another pet peeve that makes it harder to use the on-disk docs. Greetings Marc -- ----------------------------------------------------------------------------- Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421 ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Stop compressing manual pages 2025-12-26 10:51 ` Marc Haber @ 2025-12-26 14:52 ` Simon McVittie 0 siblings, 0 replies; 5+ messages in thread From: Simon McVittie @ 2025-12-26 14:52 UTC (permalink / raw) To: Marc Haber Cc: Russ Allbery, Alejandro Colomar, debian-policy, Dr. Tobias Quathamer, linux-man, Helge Kreutzmann, G. Branden Robinson, Colin Watson On Fri, 26 Dec 2025 at 11:51:40 +0100, Marc Haber wrote: >In embedded and other small installations, I'd prefer everyting under >/usr/share/man just excluded from installations on the package manager >level, but I never have been able to find out how to rub dpkg >--path-exclude the right way to do that. While we're at it, exclude >everything under /usr/share/doc with the exception of >/usr/share/doc/*/copyright* Here's how the semi-official debian:sid-slim, etc. Docker images do it: https://github.com/debuerreotype/debuerreotype/blob/master/scripts/debuerreotype-slimify https://github.com/debuerreotype/debuerreotype/blob/master/scripts/.slimify-excludes (each non-comment line $x becomes "path-exclude $x" in /etc/dpkg.cfg.d/docker) https://github.com/debuerreotype/debuerreotype/blob/master/scripts/.slimify-includes (each non-comment line $x becomes "path-include $x" in /etc/dpkg.cfg.d/docker) smcv ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2025-12-26 14:52 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <aUzUvdZEJpDHb3QX@meinfjell.helgefjelltest.de>
[not found] ` <aU0WjfHED1esOwPy@devuan>
[not found] ` <fec615b5-af5b-46cd-ae09-d9343db6da77@debian.org>
2025-12-25 14:15 ` Stop compressing manual pages (was: Bug#1123959: manpages: Please consider shipping uncompressed man pages) Alejandro Colomar
2025-12-25 19:06 ` Stop compressing manual pages Russ Allbery
2025-12-26 3:08 ` G. Branden Robinson
2025-12-26 10:51 ` Marc Haber
2025-12-26 14:52 ` Simon McVittie
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).