[PATCH 0/4] Address some issues with sphinx detection

[PATCH 0/4] Address some issues with sphinx detection

[Mauro Carvalho Chehab]

Cc: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>,
 linux-kernel@vger.kernel.org, Jonathan Corbet <corbet@lwn.net>,
 ksummit-discuss@lists.linuxfoundation.org
Cc: "Jonathan Corbet" <corbet@lwn.net>
Cc: "Mauro Carvalho Chehab" <mchehab+huawei@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Mauro Carvalho Chehab <mchehab@kernel.org>
Cc: ksummit-discuss@lists.linuxfoundation.org
Cc: linux-doc@vger.kernel.org
Cc: linux-kernel@vger.kernel.org

Checking if Sphinx is available and read to run is tricky, and may involve
installing several packages for the document build to happen.

There are two options to install Sphinx:

	- via distro-provided packages;
	- via pip, using virtualenv/venv.

Some recent discussions showed a couple of misleading instructions.

This series improves the Sphinx detection by:

- Fixing the logich with checks if a past venv is working and recommend
  just enabling it instead of installing a new venv;
- Detect if sphinx-build stopped working on a venv. This may happen during
  distribution updates;
- Move the PDF minimal version to be later, in order for it to be printed only
  after finishing the Sphinx version check;

Additionally, as now the Sphinx provided on almost all modern distros are
above the minimal required version, place instructions about how to install
Sphinx from the distro-provided packages after placing the instructions for
installing it via venv.

This will hopefully help to have more developers checking documentation
builds with

	make htmldocs

Mauro Carvalho Chehab (4):
  scripts: sphinx-pre-install: fix venv version check logic
  scripts: sphinx-pre-install: report broken venv
  scripts: sphinx-pre-install: check for PDF min version later on
  scripts: sphinx-pre-install: provide both venv and package installs

 scripts/sphinx-pre-install | 74 ++++++++++++++++++++++++++++----------
 1 file changed, 55 insertions(+), 19 deletions(-)

-- 
2.36.1

[PATCH 1/4] scripts: sphinx-pre-install: fix venv version check logic

[Mauro Carvalho Chehab]

Cc: "Jonathan Corbet" <corbet@lwn.net>
Cc: "Mauro Carvalho Chehab" <mchehab+huawei@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Mauro Carvalho Chehab <mchehab@kernel.org>
Cc: ksummit-discuss@lists.linuxfoundation.org
Cc: linux-doc@vger.kernel.org
Cc: linux-kernel@vger.kernel.org

The logic which checks if the venv version is good enough
but was not activated is broken: it is checking against
the wrong val, making it to recommend to re-create a venv
every time. Fix it.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---

To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover.
See [PATCH 0/4] at: https://lore.kernel.org/all/cover.1656664906.git.mchehab@kernel.org/

 scripts/sphinx-pre-install | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/scripts/sphinx-pre-install b/scripts/sphinx-pre-install
index f126ecbb0494..ae8c49734899 100755
--- a/scripts/sphinx-pre-install
+++ b/scripts/sphinx-pre-install
@@ -741,7 +741,7 @@ sub recommend_sphinx_upgrade()
 
 	# Get the highest version from sphinx_*/bin/sphinx-build and the
 	# corresponding command to activate the venv/virtenv
-	$activate_cmd = get_virtenv();
+	($activate_cmd, $venv_ver) = get_virtenv();
 
 	# Store the highest version from Sphinx existing virtualenvs
 	if (($activate_cmd ne "") && ($venv_ver gt $cur_version)) {
@@ -759,10 +759,14 @@ sub recommend_sphinx_upgrade()
 	# Either there are already a virtual env or a new one should be created
 	$need_pip = 1;
 
+	return if (!$latest_avail_ver);
+
 	# Return if the reason is due to an upgrade or not
 	if ($latest_avail_ver lt $rec_version) {
 		$rec_sphinx_upgrade = 1;
 	}
+
+	return $latest_avail_ver;
 }
 
 #
@@ -820,7 +824,7 @@ sub recommend_sphinx_version($)
 	}
 
 	# Suggest newer versions if current ones are too old
-	if ($latest_avail_ver && $cur_version ge $min_version) {
+	if ($latest_avail_ver && $latest_avail_ver ge $min_version) {
 		# If there's a good enough version, ask the user to enable it
 		if ($latest_avail_ver ge $rec_version) {
 			printf "\nNeed to activate Sphinx (version $latest_avail_ver) on virtualenv with:\n";
@@ -897,7 +901,7 @@ sub check_needs()
 		}
 	}
 
-	recommend_sphinx_upgrade();
+	my $venv_ver = recommend_sphinx_upgrade();
 
 	my $virtualenv_cmd;
 
-- 
2.36.1

[PATCH 2/4] scripts: sphinx-pre-install: report broken venv

[Mauro Carvalho Chehab]

Cc: "Jonathan Corbet" <corbet@lwn.net>
Cc: "Mauro Carvalho Chehab" <mchehab+huawei@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Mauro Carvalho Chehab <mchehab@kernel.org>
Cc: ksummit-discuss@lists.linuxfoundation.org
Cc: linux-doc@vger.kernel.org
Cc: linux-kernel@vger.kernel.org

After distro upgrades, the directory names for python may change.
On such case, the previously-created venv will be broken, and
sphinx-build won't run.

Add a logic to report it.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---

To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover.
See [PATCH 0/4] at: https://lore.kernel.org/all/cover.1656664906.git.mchehab@kernel.org/

 scripts/sphinx-pre-install | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/sphinx-pre-install b/scripts/sphinx-pre-install
index ae8c49734899..18537e5af692 100755
--- a/scripts/sphinx-pre-install
+++ b/scripts/sphinx-pre-install
@@ -720,6 +720,12 @@ sub get_virtenv()
 		next if (! -f $sphinx_cmd);
 
 		my $ver = get_sphinx_version($sphinx_cmd);
+
+		if (!$ver) {
+			$f =~ s#/bin/activate##;
+			print("Warning: virtual envionment $f is not working.\nPython version upgrade? Remove it with:\n\n\trm -rf $f\n\n");
+		}
+
 		if ($need_sphinx && ($ver ge $min_version)) {
 			return ($f, $ver);
 		} elsif ($ver gt $cur_version) {
-- 
2.36.1

[PATCH 3/4] scripts: sphinx-pre-install: check for PDF min version later on

[Mauro Carvalho Chehab]

Cc: "Jonathan Corbet" <corbet@lwn.net>
Cc: "Mauro Carvalho Chehab" <mchehab+huawei@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Mauro Carvalho Chehab <mchehab@kernel.org>
Cc: ksummit-discuss@lists.linuxfoundation.org
Cc: linux-doc@vger.kernel.org
Cc: linux-kernel@vger.kernel.org

Better to add the PDF note late for venv recommendation.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---

To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover.
See [PATCH 0/4] at: https://lore.kernel.org/all/cover.1656664906.git.mchehab@kernel.org/

 scripts/sphinx-pre-install | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/scripts/sphinx-pre-install b/scripts/sphinx-pre-install
index 18537e5af692..930a6d058c12 100755
--- a/scripts/sphinx-pre-install
+++ b/scripts/sphinx-pre-install
@@ -785,12 +785,13 @@ sub recommend_sphinx_version($)
 {
 	my $virtualenv_cmd = shift;
 
-	if ($latest_avail_ver lt $min_pdf_version) {
-		print "note: If you want pdf, you need at least Sphinx $min_pdf_version.\n";
-	}
-
 	# Version is OK. Nothing to do.
-	return if ($cur_version && ($cur_version ge $rec_version));
+	if ($cur_version && ($cur_version ge $rec_version)) {
+		if ($cur_version lt $min_pdf_version) {
+			print "note: If you want pdf, you need at least Sphinx $min_pdf_version.\n";
+		}
+		return;
+	};
 
 	if (!$need_sphinx) {
 		# sphinx-build is present and its version is >= $min_version
@@ -837,6 +838,10 @@ sub recommend_sphinx_version($)
 			printf "\t. $activate_cmd\n";
 			deactivate_help();
 
+			if ($latest_avail_ver lt $min_pdf_version) {
+				print "note: If you want pdf, you need at least Sphinx $min_pdf_version.\n";
+			}
+
 			return;
 		}
 
-- 
2.36.1

[PATCH 4/4] scripts: sphinx-pre-install: provide both venv and package installs

[Mauro Carvalho Chehab]

Cc: "Jonathan Corbet" <corbet@lwn.net>
Cc: "Mauro Carvalho Chehab" <mchehab+huawei@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Mauro Carvalho Chehab <mchehab@kernel.org>
Cc: ksummit-discuss@lists.linuxfoundation.org
Cc: linux-doc@vger.kernel.org
Cc: linux-kernel@vger.kernel.org

As it is not a consensus about installing sphinx using venv, and
modern distributions are now shipping with Sphinx versions above
the minimal requirements to build the docs, provide both venv
and package install commands by default.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---

To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover.
See [PATCH 0/4] at: https://lore.kernel.org/all/cover.1656664906.git.mchehab@kernel.org/

 scripts/sphinx-pre-install | 43 ++++++++++++++++++++++++++++----------
 1 file changed, 32 insertions(+), 11 deletions(-)

diff --git a/scripts/sphinx-pre-install b/scripts/sphinx-pre-install
index 930a6d058c12..27bac4fbe35b 100755
--- a/scripts/sphinx-pre-install
+++ b/scripts/sphinx-pre-install
@@ -25,6 +25,7 @@ my $need_sphinx = 0;
 my $need_pip = 0;
 my $need_virtualenv = 0;
 my $rec_sphinx_upgrade = 0;
+my $verbose_warn_install = 1;
 my $install = "";
 my $virtenv_dir = "";
 my $python_cmd = "";
@@ -103,10 +104,12 @@ sub check_missing(%)
 			next;
 		}
 
-		if ($is_optional) {
-			print "Warning: better to also install \"$prog\".\n";
-		} else {
-			print "ERROR: please install \"$prog\", otherwise, build won't work.\n";
+		if ($verbose_warn_install) {
+			if ($is_optional) {
+				print "Warning: better to also install \"$prog\".\n";
+			} else {
+				print "ERROR: please install \"$prog\", otherwise, build won't work.\n";
+			}
 		}
 		if (defined($map{$prog})) {
 			$install .= " " . $map{$prog};
@@ -386,7 +389,8 @@ sub give_debian_hints()
 	check_missing(\%map);
 
 	return if (!$need && !$optional);
-	printf("You should run:\n\n\tsudo apt-get install $install\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n\tsudo apt-get install $install\n");
 }
 
 sub give_redhat_hints()
@@ -458,10 +462,12 @@ sub give_redhat_hints()
 
 	if (!$old) {
 		# dnf, for Fedora 18+
-		printf("You should run:\n\n\tsudo dnf install -y $install\n");
+		printf("You should run:\n") if ($verbose_warn_install);
+		printf("\n\tsudo dnf install -y $install\n");
 	} else {
 		# yum, for RHEL (and clones) or Fedora version < 18
-		printf("You should run:\n\n\tsudo yum install -y $install\n");
+		printf("You should run:\n") if ($verbose_warn_install);
+		printf("\n\tsudo yum install -y $install\n");
 	}
 }
 
@@ -509,7 +515,8 @@ sub give_opensuse_hints()
 	check_missing(\%map);
 
 	return if (!$need && !$optional);
-	printf("You should run:\n\n\tsudo zypper install --no-recommends $install\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n\tsudo zypper install --no-recommends $install\n");
 }
 
 sub give_mageia_hints()
@@ -553,7 +560,8 @@ sub give_mageia_hints()
 	check_missing(\%map);
 
 	return if (!$need && !$optional);
-	printf("You should run:\n\n\tsudo $packager_cmd $install\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n\tsudo $packager_cmd $install\n");
 }
 
 sub give_arch_linux_hints()
@@ -583,7 +591,8 @@ sub give_arch_linux_hints()
 	check_missing(\%map);
 
 	return if (!$need && !$optional);
-	printf("You should run:\n\n\tsudo pacman -S $install\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n\tsudo pacman -S $install\n");
 }
 
 sub give_gentoo_hints()
@@ -610,7 +619,8 @@ sub give_gentoo_hints()
 
 	return if (!$need && !$optional);
 
-	printf("You should run:\n\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n");
 
 	my $imagemagick = "media-gfx/imagemagick svg png";
 	my $cairo = "media-gfx/graphviz cairo pdf";
@@ -873,6 +883,16 @@ sub recommend_sphinx_version($)
 	printf "\t. $virtenv_dir/bin/activate\n";
 	printf "\tpip install -r $requirement_file\n";
 	deactivate_help();
+
+	printf "\nAnother option would be to install Sphinx package with:\n";
+
+	%missing = ();
+	$pdf = 0;
+	$optional = 0;
+	add_package("python-sphinx", 0);
+	check_python_module("sphinx_rtd_theme", 1);
+
+	check_distros();
 }
 
 sub check_needs()
@@ -945,6 +965,7 @@ sub check_needs()
 	check_program("latexmk", 2) if ($pdf);
 
 	# Do distro-specific checks and output distro-install commands
+	$verbose_warn_install = 0;
 	check_distros();
 
 	if (!$python_cmd) {
-- 
2.36.1

[PATCH v2 0/5] Address some issues with sphinx detection

[Mauro Carvalho Chehab]

Checking if Sphinx is available and read to run is tricky, and may involve
installing several packages for the document build to happen.

There are two options to install Sphinx:

	- via distro-provided packages;
	- via pip, using virtualenv/venv.

Some recent discussions showed a couple of misleading instructions.

This series improves the Sphinx detection by:

- Fixing the logich with checks if a past venv is working and recommend
  just enabling it instead of installing a new venv;
- Detect if sphinx-build stopped working on a venv. This may happen during
  distribution updates;
- Move the PDF minimal version to be later, in order for it to be printed only
  after finishing the Sphinx version check;

Additionally, as now the Sphinx provided on almost all modern distros are
above the minimal required version, place instructions about how to install
Sphinx from the distro-provided packages after placing the instructions for
installing it via venv.

This will hopefully help to have more developers checking documentation
builds with

	make htmldocs

---

v2:
- Fix some issues at the logic that was recommending installing via
  distro-package manager;
- Add a notice about Sphinx 3.0+ false-positive warnings due to a
  Sphinx bug. People using a version different than the recommended
  one should know about that.

Mauro Carvalho Chehab (5):
  scripts: sphinx-pre-install: fix venv version check logic
  scripts: sphinx-pre-install: report broken venv
  scripts: sphinx-pre-install: check for PDF min version later on
  scripts: sphinx-pre-install: provide both venv and package installs
  scripts: sphinx-pre-install: place a warning for Sphinx >= 3.0

 scripts/sphinx-pre-install | 90 +++++++++++++++++++++++++++++---------
 1 file changed, 69 insertions(+), 21 deletions(-)

-- 
2.36.1

[PATCH v2 1/5] scripts: sphinx-pre-install: fix venv version check logic

[Mauro Carvalho Chehab]

The logic which checks if the venv version is good enough
but was not activated is broken: it is checking against
the wrong val, making it to recommend to re-create a venv
every time. Fix it.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---

To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover.
See [PATCH v2 0/5] at: https://lore.kernel.org/all/cover.1656756450.git.mchehab@kernel.org/

 scripts/sphinx-pre-install | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/scripts/sphinx-pre-install b/scripts/sphinx-pre-install
index f126ecbb0494..ae8c49734899 100755
--- a/scripts/sphinx-pre-install
+++ b/scripts/sphinx-pre-install
@@ -741,7 +741,7 @@ sub recommend_sphinx_upgrade()
 
 	# Get the highest version from sphinx_*/bin/sphinx-build and the
 	# corresponding command to activate the venv/virtenv
-	$activate_cmd = get_virtenv();
+	($activate_cmd, $venv_ver) = get_virtenv();
 
 	# Store the highest version from Sphinx existing virtualenvs
 	if (($activate_cmd ne "") && ($venv_ver gt $cur_version)) {
@@ -759,10 +759,14 @@ sub recommend_sphinx_upgrade()
 	# Either there are already a virtual env or a new one should be created
 	$need_pip = 1;
 
+	return if (!$latest_avail_ver);
+
 	# Return if the reason is due to an upgrade or not
 	if ($latest_avail_ver lt $rec_version) {
 		$rec_sphinx_upgrade = 1;
 	}
+
+	return $latest_avail_ver;
 }
 
 #
@@ -820,7 +824,7 @@ sub recommend_sphinx_version($)
 	}
 
 	# Suggest newer versions if current ones are too old
-	if ($latest_avail_ver && $cur_version ge $min_version) {
+	if ($latest_avail_ver && $latest_avail_ver ge $min_version) {
 		# If there's a good enough version, ask the user to enable it
 		if ($latest_avail_ver ge $rec_version) {
 			printf "\nNeed to activate Sphinx (version $latest_avail_ver) on virtualenv with:\n";
@@ -897,7 +901,7 @@ sub check_needs()
 		}
 	}
 
-	recommend_sphinx_upgrade();
+	my $venv_ver = recommend_sphinx_upgrade();
 
 	my $virtualenv_cmd;
 
-- 
2.36.1

[PATCH v2 2/5] scripts: sphinx-pre-install: report broken venv

[Mauro Carvalho Chehab]

After distro upgrades, the directory names for python may change.
On such case, the previously-created venv will be broken, and
sphinx-build won't run.

Add a logic to report it.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---

To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover.
See [PATCH v2 0/5] at: https://lore.kernel.org/all/cover.1656756450.git.mchehab@kernel.org/

 scripts/sphinx-pre-install | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/sphinx-pre-install b/scripts/sphinx-pre-install
index ae8c49734899..18537e5af692 100755
--- a/scripts/sphinx-pre-install
+++ b/scripts/sphinx-pre-install
@@ -720,6 +720,12 @@ sub get_virtenv()
 		next if (! -f $sphinx_cmd);
 
 		my $ver = get_sphinx_version($sphinx_cmd);
+
+		if (!$ver) {
+			$f =~ s#/bin/activate##;
+			print("Warning: virtual envionment $f is not working.\nPython version upgrade? Remove it with:\n\n\trm -rf $f\n\n");
+		}
+
 		if ($need_sphinx && ($ver ge $min_version)) {
 			return ($f, $ver);
 		} elsif ($ver gt $cur_version) {
-- 
2.36.1

[PATCH v2 3/5] scripts: sphinx-pre-install: check for PDF min version later on

[Mauro Carvalho Chehab]

Better to add the PDF note late for venv recommendation.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---

To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover.
See [PATCH v2 0/5] at: https://lore.kernel.org/all/cover.1656756450.git.mchehab@kernel.org/

 scripts/sphinx-pre-install | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/scripts/sphinx-pre-install b/scripts/sphinx-pre-install
index 18537e5af692..930a6d058c12 100755
--- a/scripts/sphinx-pre-install
+++ b/scripts/sphinx-pre-install
@@ -785,12 +785,13 @@ sub recommend_sphinx_version($)
 {
 	my $virtualenv_cmd = shift;
 
-	if ($latest_avail_ver lt $min_pdf_version) {
-		print "note: If you want pdf, you need at least Sphinx $min_pdf_version.\n";
-	}
-
 	# Version is OK. Nothing to do.
-	return if ($cur_version && ($cur_version ge $rec_version));
+	if ($cur_version && ($cur_version ge $rec_version)) {
+		if ($cur_version lt $min_pdf_version) {
+			print "note: If you want pdf, you need at least Sphinx $min_pdf_version.\n";
+		}
+		return;
+	};
 
 	if (!$need_sphinx) {
 		# sphinx-build is present and its version is >= $min_version
@@ -837,6 +838,10 @@ sub recommend_sphinx_version($)
 			printf "\t. $activate_cmd\n";
 			deactivate_help();
 
+			if ($latest_avail_ver lt $min_pdf_version) {
+				print "note: If you want pdf, you need at least Sphinx $min_pdf_version.\n";
+			}
+
 			return;
 		}
 
-- 
2.36.1

[PATCH v2 4/5] scripts: sphinx-pre-install: provide both venv and package installs

[Mauro Carvalho Chehab]

As it is not a consensus about installing sphinx using venv, and
modern distributions are now shipping with Sphinx versions above
the minimal requirements to build the docs, provide both venv
and package install commands by default.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---

To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover.
See [PATCH v2 0/5] at: https://lore.kernel.org/all/cover.1656756450.git.mchehab@kernel.org/

 scripts/sphinx-pre-install | 55 +++++++++++++++++++++++++++++---------
 1 file changed, 42 insertions(+), 13 deletions(-)

diff --git a/scripts/sphinx-pre-install b/scripts/sphinx-pre-install
index 930a6d058c12..106d75425d3f 100755
--- a/scripts/sphinx-pre-install
+++ b/scripts/sphinx-pre-install
@@ -25,6 +25,7 @@ my $need_sphinx = 0;
 my $need_pip = 0;
 my $need_virtualenv = 0;
 my $rec_sphinx_upgrade = 0;
+my $verbose_warn_install = 1;
 my $install = "";
 my $virtenv_dir = "";
 my $python_cmd = "";
@@ -103,10 +104,12 @@ sub check_missing(%)
 			next;
 		}
 
-		if ($is_optional) {
-			print "Warning: better to also install \"$prog\".\n";
-		} else {
-			print "ERROR: please install \"$prog\", otherwise, build won't work.\n";
+		if ($verbose_warn_install) {
+			if ($is_optional) {
+				print "Warning: better to also install \"$prog\".\n";
+			} else {
+				print "ERROR: please install \"$prog\", otherwise, build won't work.\n";
+			}
 		}
 		if (defined($map{$prog})) {
 			$install .= " " . $map{$prog};
@@ -386,7 +389,8 @@ sub give_debian_hints()
 	check_missing(\%map);
 
 	return if (!$need && !$optional);
-	printf("You should run:\n\n\tsudo apt-get install $install\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n\tsudo apt-get install $install\n");
 }
 
 sub give_redhat_hints()
@@ -458,10 +462,12 @@ sub give_redhat_hints()
 
 	if (!$old) {
 		# dnf, for Fedora 18+
-		printf("You should run:\n\n\tsudo dnf install -y $install\n");
+		printf("You should run:\n") if ($verbose_warn_install);
+		printf("\n\tsudo dnf install -y $install\n");
 	} else {
 		# yum, for RHEL (and clones) or Fedora version < 18
-		printf("You should run:\n\n\tsudo yum install -y $install\n");
+		printf("You should run:\n") if ($verbose_warn_install);
+		printf("\n\tsudo yum install -y $install\n");
 	}
 }
 
@@ -509,7 +515,8 @@ sub give_opensuse_hints()
 	check_missing(\%map);
 
 	return if (!$need && !$optional);
-	printf("You should run:\n\n\tsudo zypper install --no-recommends $install\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n\tsudo zypper install --no-recommends $install\n");
 }
 
 sub give_mageia_hints()
@@ -553,7 +560,8 @@ sub give_mageia_hints()
 	check_missing(\%map);
 
 	return if (!$need && !$optional);
-	printf("You should run:\n\n\tsudo $packager_cmd $install\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n\tsudo $packager_cmd $install\n");
 }
 
 sub give_arch_linux_hints()
@@ -583,7 +591,8 @@ sub give_arch_linux_hints()
 	check_missing(\%map);
 
 	return if (!$need && !$optional);
-	printf("You should run:\n\n\tsudo pacman -S $install\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n\tsudo pacman -S $install\n");
 }
 
 sub give_gentoo_hints()
@@ -610,7 +619,8 @@ sub give_gentoo_hints()
 
 	return if (!$need && !$optional);
 
-	printf("You should run:\n\n");
+	printf("You should run:\n") if ($verbose_warn_install);
+	printf("\n");
 
 	my $imagemagick = "media-gfx/imagemagick svg png";
 	my $cairo = "media-gfx/graphviz cairo pdf";
@@ -700,7 +710,7 @@ sub check_distros()
 
 sub deactivate_help()
 {
-	printf "\nIf you want to exit the virtualenv, you can use:\n";
+	printf "\n    If you want to exit the virtualenv, you can use:\n";
 	printf "\tdeactivate\n";
 }
 
@@ -863,7 +873,7 @@ sub recommend_sphinx_version($)
 			print "To upgrade Sphinx, use:\n\n";
 		}
 	} else {
-		print "Sphinx needs to be installed either as a package or via pip/pypi with:\n";
+		print "\nSphinx needs to be installed either:\n1) via pip/pypi with:\n\n";
 	}
 
 	$python_cmd = find_python_no_venv();
@@ -873,6 +883,25 @@ sub recommend_sphinx_version($)
 	printf "\t. $virtenv_dir/bin/activate\n";
 	printf "\tpip install -r $requirement_file\n";
 	deactivate_help();
+
+	printf "\n2) As a package with:\n";
+
+	my $old_need = $need;
+	my $old_optional = $optional;
+	%missing = ();
+	$pdf = 0;
+	$optional = 0;
+	$install = "";
+	$verbose_warn_install = 0;
+
+	add_package("python-sphinx", 0);
+	check_python_module("sphinx_rtd_theme", 1);
+
+	check_distros();
+
+	$need = $old_need;
+	$optional = $old_optional;
+
 }
 
 sub check_needs()
-- 
2.36.1

[PATCH v2 5/5] scripts: sphinx-pre-install: place a warning for Sphinx >= 3.0

[Mauro Carvalho Chehab]

Newer versions of Sphinx have a known bug:
	https://github.com/sphinx-doc/sphinx/pull/8313

Those currently produces 11 false-positive warnings On Sphinx
version 3.1+.

While Sphinx fix is not applied, let's warn adventurers that would
be using Sphinx installed via distro packages.

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---

To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover.
See [PATCH v2 0/5] at: https://lore.kernel.org/all/cover.1656756450.git.mchehab@kernel.org/

 scripts/sphinx-pre-install | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/scripts/sphinx-pre-install b/scripts/sphinx-pre-install
index 106d75425d3f..271c4eb1d702 100755
--- a/scripts/sphinx-pre-install
+++ b/scripts/sphinx-pre-install
@@ -902,6 +902,10 @@ sub recommend_sphinx_version($)
 	$need = $old_need;
 	$optional = $old_optional;
 
+	printf "\n    Please note that Sphinx >= 3.0 will currently produce false-positive\n";
+	printf "   warning when the same name is used for more than one type (functions,\n";
+	printf "   structs, enums,...). This is known Sphinx bug. For more details, see:\n";
+	printf "\thttps://github.com/sphinx-doc/sphinx/pull/8313\n";
 }
 
 sub check_needs()
-- 
2.36.1

Re: [PATCH v2 0/5] Address some issues with sphinx detection

[Akira Yokosawa]

Hi Mauro,

On Sat,  2 Jul 2022 11:11:24 +0100, Mauro Carvalho Chehab  wrote:
> Checking if Sphinx is available and read to run is tricky, and may involve
> installing several packages for the document build to happen.
> 
> There are two options to install Sphinx:
> 
> 	- via distro-provided packages;
> 	- via pip, using virtualenv/venv.
> 
> Some recent discussions showed a couple of misleading instructions.
> 
> This series improves the Sphinx detection by:
> 
> - Fixing the logich with checks if a past venv is working and recommend
>   just enabling it instead of installing a new venv;
> - Detect if sphinx-build stopped working on a venv. This may happen during
>   distribution updates;
> - Move the PDF minimal version to be later, in order for it to be printed only
>   after finishing the Sphinx version check;
> 
> Additionally, as now the Sphinx provided on almost all modern distros are
> above the minimal required version, place instructions about how to install
> Sphinx from the distro-provided packages after placing the instructions for
> installing it via venv.
> 
> This will hopefully help to have more developers checking documentation
> builds with
> 
> 	make htmldocs
So this is a meta-level feedback considering the most likely uses
of sphinx-pre-install.

I think first-time users of sphinx-pre-install are more likely
interested in getting ready for running "make html".  They won't
bother with "make pdfdocs".  They won't likely be interested in
virtualenv, either.

So I think it would be reasonable to change the default behavior
of sphinx-pre-install.

I mean the reasonable behavior without any option would be that of
when both --no-pdf and --no-virtualenv are given to the current
version.

There are a few issues on --no-pdf.

It says imagemagick and gcc are necessary, but they are redundant
in "make html", as far as I see.

Furthermore, it doesn't check dvipng and latex, which are used
for generating math equation images in HTML.

Fedora, RHEL/CentOS, and openSUSE Leap provide helpful packages
for installing math expression support.

    Fedora 36               python3-sphinx-latex (python3-sphinx depends on this)
    RHEL 9/CentOS stream 9  ditto
    openSUSE Leap 15.4      python3-Sphinx_4_2_0-latex
                                (python3-Sphinx_4_2_0 depends on this) or
                            python3-Sphinx-latex
                                (python3-Sphinx depends on this, version: 2.3.1)

Other distros, whose texlive packages are coarse grained, don't provide
such helper packages.

Also, as mentioned previously, RHEL 9/CentOS stream9's texlive-xecjk 
doesn't work at the moment due to the lack of its dependency (texlive-ctex).
I opened a bug ticket at RedHat bugzilla:
   https://bugzilla.redhat.com/show_bug.cgi?id=2086254

Unfortunately, I've heard no response yet.
If you know some means to boost its priority, please provide a comment
or two to the ticket.

Until the issue can be resolved, "make pdfdocs" is limited to non-CJK
build on CentOS stream 9, RHEL 9 and its clones if you must stick to
distro packages.  For non-CJK build to work, you must not have
google-noto-sans-cjk-ttc-fonts.

openSUSE Leap does not support full CJK build of "make pdfdocs", either.
Its Noto font packaging is peculiar and a similar named package of
noto-sans-cjk-fonts installs a set of language-specific fonts, each
of which doesn't qualify as a CJK font.

Seeing these problems of LTS distros, I'd suggest sphinx-pre-install
--pdf would check packages for non-CJK builds.

Another option of --cjk would show you additional packages for full CJK build.

TL;DR, my suggestion of options and defaults to sphinx-pre-install:

    --no-pdf (default): for htmldocs only
    --no-virtualenv (default): distro Sphinx package
         (mention --virtualenv if distro Sphinx package is too young)
    --virtualenv: Sphinx by venv/virtualenv
    --pdf: for pdfdocs
        --no-cjk (default): don't bother with CJK pdfdocs
        --cjk: for CJK pdfdocs
               (print warning if user's distro doesn't support CJK)

Thoughts?

        Thanks, Akira

>
> ---
> 
> v2:
> - Fix some issues at the logic that was recommending installing via
>   distro-package manager;
> - Add a notice about Sphinx 3.0+ false-positive warnings due to a
>   Sphinx bug. People using a version different than the recommended
>   one should know about that.
> 
> Mauro Carvalho Chehab (5):
>   scripts: sphinx-pre-install: fix venv version check logic
>   scripts: sphinx-pre-install: report broken venv
>   scripts: sphinx-pre-install: check for PDF min version later on
>   scripts: sphinx-pre-install: provide both venv and package installs
>   scripts: sphinx-pre-install: place a warning for Sphinx >= 3.0
> 
>  scripts/sphinx-pre-install | 90 +++++++++++++++++++++++++++++---------
>  1 file changed, 69 insertions(+), 21 deletions(-)
> 
> -- 
> 2.36.1

Re: [PATCH v2 0/5] Address some issues with sphinx detection

[Akira Yokosawa]

[CC: update address of ksummit]

On Tue, 5 Jul 2022 13:15:57 +0900, Akira Yokosawa wrote:
> Hi Mauro,
[...]
> 
> Fedora, RHEL/CentOS, and openSUSE Leap provide helpful packages
> for installing math expression support.
> 
>     Fedora 36               python3-sphinx-latex (python3-sphinx depends on this)
>     RHEL 9/CentOS stream 9  ditto
>     openSUSE Leap 15.4      python3-Sphinx_4_2_0-latex
>                                 (python3-Sphinx_4_2_0 depends on this) or
>                             python3-Sphinx-latex
>                                 (python3-Sphinx depends on this, version: 2.3.1)

These packages are supposed to cover LaTeX packages necessary
for building LaTeX sources the version of Sphinx generates.

HOWEVER, in my test of openSUSE Leap 15.4, pythno3-Sphinx-4_2_0-latex
does not cover texlive-tex-gyre, which is required since Sphinx 4.0.0.

Changelog of Sphinx 4.0.0 [1] says:

> Dependencies
>
> 4.0.0b1
>
> [...]
>   * LaTeX: add tex-gyre font dependency

[1]: https://www.sphinx-doc.org/en/master/changes.html#release-4-0-0-released-may-09-2021

I'm thinking of opening a ticket at openSUSE's bugzilla.

Fedora 36's python3-sphinx-latex (for Sphinx 4.4.0) has
texlive-collection-fontsrecommended and covers texlive-tex-gyre naturally.

        Thanks, Akira

> 

Re: [PATCH v2 0/5] Address some issues with sphinx detection

[Mauro Carvalho Chehab]

Em Wed, 6 Jul 2022 23:31:09 +0900
Akira Yokosawa <akiyks@gmail.com> escreveu:

> [CC: update address of ksummit]
> 
> On Tue, 5 Jul 2022 13:15:57 +0900, Akira Yokosawa wrote:
> > Hi Mauro,  
> [...]
> > 
> > Fedora, RHEL/CentOS, and openSUSE Leap provide helpful packages
> > for installing math expression support.
> > 
> >     Fedora 36               python3-sphinx-latex (python3-sphinx depends on this)
> >     RHEL 9/CentOS stream 9  ditto
> >     openSUSE Leap 15.4      python3-Sphinx_4_2_0-latex
> >                                 (python3-Sphinx_4_2_0 depends on this) or
> >                             python3-Sphinx-latex
> >                                 (python3-Sphinx depends on this, version: 2.3.1)  
> 
> These packages are supposed to cover LaTeX packages necessary
> for building LaTeX sources the version of Sphinx generates.
> 
> HOWEVER, in my test of openSUSE Leap 15.4, pythno3-Sphinx-4_2_0-latex
> does not cover texlive-tex-gyre, which is required since Sphinx 4.0.0.
> 
> Changelog of Sphinx 4.0.0 [1] says:
> 
> > Dependencies
> >
> > 4.0.0b1
> >
> > [...]
> >   * LaTeX: add tex-gyre font dependency  
> 
> [1]: https://www.sphinx-doc.org/en/master/changes.html#release-4-0-0-released-may-09-2021
> 
> I'm thinking of opening a ticket at openSUSE's bugzilla.
> 
> Fedora 36's python3-sphinx-latex (for Sphinx 4.4.0) has
> texlive-collection-fontsrecommended and covers texlive-tex-gyre naturally.
> 

Yeah, I remember the script started using some meta-packages for LaTeX.
This had to change a couple of times because distros that have dozens of
packages for it were not installing everything on their meta packages.
So, the packages install but they don't run as they depend on other
non-installed packages.

That's basically why I added a list of *.sty inside the script and started
using kpsewhich to double-check if all dependencies are there.

Regards,
Mauro

Re: [PATCH v2 0/5] Address some issues with sphinx detection

[Jonathan Corbet]

Akira Yokosawa <akiyks@gmail.com> writes:

> TL;DR, my suggestion of options and defaults to sphinx-pre-install:
>
>     --no-pdf (default): for htmldocs only
>     --no-virtualenv (default): distro Sphinx package
>          (mention --virtualenv if distro Sphinx package is too young)
>     --virtualenv: Sphinx by venv/virtualenv
>     --pdf: for pdfdocs
>         --no-cjk (default): don't bother with CJK pdfdocs
>         --cjk: for CJK pdfdocs
>                (print warning if user's distro doesn't support CJK)
>
> Thoughts?

I think this makes sense.  As far as I can tell, PDF builds are a
relative rarity these days, and most people would rather not have to
deal with virtualenv if they can avoid it.  We should definitely
emphasize native installs whenever that can work.

I'm planning to go ahead and apply Mauro's sphinx-pre-install patches
since they make things better, but then we should look at these tweaks.

Thanks,

jon

Re: [PATCH v2 0/5] Address some issues with sphinx detection

[Mauro Carvalho Chehab]

Em Thu, 07 Jul 2022 12:45:14 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Akira Yokosawa <akiyks@gmail.com> writes:
> 
> > TL;DR, my suggestion of options and defaults to sphinx-pre-install:
> >
> >     --no-pdf (default): for htmldocs only
> >     --no-virtualenv (default): distro Sphinx package
> >          (mention --virtualenv if distro Sphinx package is too young)
> >     --virtualenv: Sphinx by venv/virtualenv
> >     --pdf: for pdfdocs
> >         --no-cjk (default): don't bother with CJK pdfdocs
> >         --cjk: for CJK pdfdocs
> >                (print warning if user's distro doesn't support CJK)
> > Thoughts?  
> 
> I think this makes sense.  As far as I can tell, PDF builds are a
> relative rarity these days, and most people would rather not have to
> deal with virtualenv if they can avoid it.  We should definitely
> emphasize native installs whenever that can work.
> 
> I'm planning to go ahead and apply Mauro's sphinx-pre-install patches
> since they make things better, but then we should look at these tweaks.

IMO it makes sense to apply them, as they address some existing issues
on it. 

We can improve the script later on with Akira's comments and after having 
some discussions about what would be the default behavior that would fit
better.

Regards,
Mauro

Re: [PATCH v2 0/5] Address some issues with sphinx detection

[Mauro Carvalho Chehab]

Em Tue, 5 Jul 2022 13:15:57 +0900
Akira Yokosawa <akiyks@gmail.com> escreveu:

> Hi Mauro,
> 
> On Sat,  2 Jul 2022 11:11:24 +0100, Mauro Carvalho Chehab  wrote:
> > Checking if Sphinx is available and read to run is tricky, and may involve
> > installing several packages for the document build to happen.
> > 
> > There are two options to install Sphinx:
> > 
> > 	- via distro-provided packages;
> > 	- via pip, using virtualenv/venv.
> > 
> > Some recent discussions showed a couple of misleading instructions.
> > 
> > This series improves the Sphinx detection by:
> > 
> > - Fixing the logich with checks if a past venv is working and recommend
> >   just enabling it instead of installing a new venv;
> > - Detect if sphinx-build stopped working on a venv. This may happen during
> >   distribution updates;
> > - Move the PDF minimal version to be later, in order for it to be printed only
> >   after finishing the Sphinx version check;
> > 
> > Additionally, as now the Sphinx provided on almost all modern distros are
> > above the minimal required version, place instructions about how to install
> > Sphinx from the distro-provided packages after placing the instructions for
> > installing it via venv.
> > 
> > This will hopefully help to have more developers checking documentation
> > builds with
> > 
> > 	make htmldocs  
> So this is a meta-level feedback considering the most likely uses
> of sphinx-pre-install.
> 
> I think first-time users of sphinx-pre-install are more likely
> interested in getting ready for running "make html".  They won't
> bother with "make pdfdocs". 

True, but, as you're pointing below, math expressions require LaTeX.

The idea of using --no-pdf is to setup an environment without LaTeX,
meaning that math tags would only be partially parsed: basically, the
output would be html with LaTeX-like math expressions (at least last
time I tried).

> They won't likely be interested in virtualenv, either.

Yes and no. The big issue with using distro packages is that it will
produce 11 false-positive warnings due to duplicated C symbols. This
will only be (hopefully) fixed on a later Sphinx 5.x (or 6.0). So,
it would take more than 6 months to get rid of those.

Using 2.4.4, after the fixes I sent, plus the 3 fixes from IIO tree
(yet to be merged on -next), there will be just 4 warnings.

So, IMO, for me, is still preferred to use 2.4.4 via venv.

> So I think it would be reasonable to change the default behavior
> of sphinx-pre-install.

With this series, both venv and non-venv settings will be shown by
default, allowing the developer decide what he prefers.

Still, I'm not 100% sure if this is the best thing to do. 

One alternative would be to run the script on an even more silent mode
when called via makefile, in a way that it would, instead it will keep:

- not display anything if sphinx-build works;
- display enable/disable commands if venv is detected;

But, instead of showing install options, it would instead, print a message
like:

	Can't build the documentation. Please run:

	./scripts/sphinx-pre-install <options)

	Where options are:
	  --no-virtualenv	- Recommend installing Sphinx instead of using a virtualenv
	  --no-pdf		- don't check for dependencies required to build PDF docs
	  ...

Another alternative would be to use --no-pdf for make htmldocs/epubdocs
target, and a "--pdf" for make pdfdocs.

> I mean the reasonable behavior without any option would be that of
> when both --no-pdf and --no-virtualenv are given to the current
> version.
> 
> There are a few issues on --no-pdf.
> 
> It says imagemagick and gcc are necessary, but they are redundant
> in "make html", as far as I see.

Well, gcc is not really necessary, but anyone using the Kernel tree
very likely needs it. So, it doesn't hurt checking for it.

With regards to imagemagick, I'm not sure if, after your patches changing
the way to build image, it is still needed. We need to review it.

Changing the dependency chain is a lot of work, as we need to retest
everything on all supported platforms, using minimal install. Here, I'm
using lxc download, as it usually gets something close to the minimal
distro install.

> Furthermore, it doesn't check dvipng and latex, which are used
> for generating math equation images in HTML.

Not checking for LaTeX is per design. It makes the install a lot havier
than what's actually needed. dvipng is a new dependency after the
changes on svg and dot conversions. Yeah, we need to test it on all
distros and properly add it where needed.

> Fedora, RHEL/CentOS, and openSUSE Leap provide helpful packages
> for installing math expression support.
> 
>     Fedora 36               python3-sphinx-latex (python3-sphinx depends on this)
>     RHEL 9/CentOS stream 9  ditto
>     openSUSE Leap 15.4      python3-Sphinx_4_2_0-latex
>                                 (python3-Sphinx_4_2_0 depends on this) or
>                             python3-Sphinx-latex
>                                 (python3-Sphinx depends on this, version: 2.3.1)

yes, but this will install LaTeX. We don't want this for the minimal htmldocs
build type, as math is used only on a handful set of documents, and most
developers can live without that.

> Other distros, whose texlive packages are coarse grained, don't provide
> such helper packages.
> 
> Also, as mentioned previously, RHEL 9/CentOS stream9's texlive-xecjk 
> doesn't work at the moment due to the lack of its dependency (texlive-ctex).

LTS distros like RHEL and SUSE are usually a lot more conservative and may
not have everything. The building system needs to cope with that.

> I opened a bug ticket at RedHat bugzilla:
>    https://bugzilla.redhat.com/show_bug.cgi?id=2086254
> 
> Unfortunately, I've heard no response yet.
> If you know some means to boost its priority, please provide a comment
> or two to the ticket.
>
> Until the issue can be resolved, "make pdfdocs" is limited to non-CJK
> build on CentOS stream 9, RHEL 9 and its clones if you must stick to
> distro packages. 

Even if they add now, RHEL 9.0 won't have it. So, the script would need
to check if the distro supports it.

> For non-CJK build to work, you must not have
> google-noto-sans-cjk-ttc-fonts.

That doesn't sound right. We should probably fix conf.py to do the right
thing. I mean, if this package is installed but texlive-xecjk, it should
disable CJK fonts.

> openSUSE Leap does not support full CJK build of "make pdfdocs", either.
> Its Noto font packaging is peculiar and a similar named package of
> noto-sans-cjk-fonts installs a set of language-specific fonts, each
> of which doesn't qualify as a CJK font.
> 
> Seeing these problems of LTS distros, I'd suggest sphinx-pre-install
> --pdf would check packages for non-CJK builds.

Makes sense.

> Another option of --cjk would show you additional packages for full CJK build.

Also makes sense. I don't think they're mutually exclusive.

> TL;DR, my suggestion of options and defaults to sphinx-pre-install:
> 
>     --no-pdf (default): for htmldocs only
>     --no-virtualenv (default): distro Sphinx package
>          (mention --virtualenv if distro Sphinx package is too young)
>     --virtualenv: Sphinx by venv/virtualenv
>     --pdf: for pdfdocs
>         --no-cjk (default): don't bother with CJK pdfdocs
>         --cjk: for CJK pdfdocs
>                (print warning if user's distro doesn't support CJK)

The options make sense. Still I would discuss more about what would be
the best default:

	- Suggest both venv and no-venv;
	- Ask the user to run the script;
	- have different behaviors when called with make pdfdocs
	  or make htmldocs / epubdocs.

Regards,
Mauro

Expectation to --no-pdf option (was Re: [PATCH v2 0/5] Address some issues with sphinx detection)

[Akira Yokosawa]

On Thu, 7 Jul 2022 21:15:58 +0100, Mauro Carvalho Chehab wrote:
> Em Tue, 5 Jul 2022 13:15:57 +0900
> Akira Yokosawa <akiyks@gmail.com> escreveu:
> 
>> Hi Mauro,
>>
[...]

>> So this is a meta-level feedback considering the most likely uses
>> of sphinx-pre-install.
>>
>> I think first-time users of sphinx-pre-install are more likely
>> interested in getting ready for running "make html".  They won't
>> bother with "make pdfdocs". 
> 
> True, but, as you're pointing below, math expressions require LaTeX.
> 
> The idea of using --no-pdf is to setup an environment without LaTeX,
> meaning that math tags would only be partially parsed: basically, the
> output would be html with LaTeX-like math expressions (at least last
> time I tried).

Oh, that is your intention.  :-/

When I saw this help text in "./scripts/sphinx-pre-install --help":

    --no-pdf	- don't check for dependencies required to build PDF docs

for the first time, my expectation was like this:

    If I make "./scripts/sphinx-pre-install --no-pdf" happy, "make htmldocs"
    will complete without any warning on package requirements.

But I got this warning from "make htmldocs" among others:

    WARNING: dvipng command 'dvipng' cannot be run (needed for math display), check the imgmath_dvipng setting

And my reaction to it was something like:

    "What!!!  Why sphinx-pre-install didn't complain?"
    "OK. I'll ignore sphinx-pre-install.  Let's see what I need to install."

The reason why this mismatch has never surfaced as a problem is, I suppose,
because most people don't use --no-pdf at all.

However, in thinking of making --no-pdf the default, I thought --no-pdf
should be more trustworthy.

If your goal is to provide LaTeX-free packages for "make htmldocs",
the option should have been named --no-latex or similar (probably in
addition to --no-pdf), no?

Well, arguing does not help. I stop here.

More constructive approach would be to make "make htmldocs" be
truly latex-free.  Then --no-pdf and --no-latex will be equivalent.

Fortunately, it is possible by using the mathjax extension instead of
imgmath.  With mathjax, rendering of math expressions is delegated to
web browsers.

The resulting HTML will depend on mathjax code and math fonts from
somewhere in the cloud (by default, https://cdnjs.cloudflare.com/ajax/libs/mathjax/...
or https://cdn.jsdelivr.net/npm/mathjax@3/... depending on the Sphinx
version).

All you need is this one-liner:

--------
diff --git a/Documentation/conf.py b/Documentation/conf.py

index 934727e23e0e..fe1084510329 100644

--- a/Documentation/conf.py

+++ b/Documentation/conf.py

@@ -106,7 +106,7 @@ else:

 autosectionlabel_prefix_document = True

 autosectionlabel_maxdepth = 2

 

-extensions.append("sphinx.ext.imgmath")

+extensions.append("sphinx.ext.mathjax")

 

 # Add any paths that contain templates here, relative to this directory.

 templates_path = ['_templates']
--------

In my tests, the mathjax extension works with all the versions of Sphinx
I tested (1.7.9, 2.4.4, 3.4.3 (debian bullseye), 4.2.0 (openSUSE LEAP 15.4),
and 5.0.2).
Note that math expressions should look much sharper (vector fonts)
than those from imgmath (pixel images).
The time for a browser to complete the rendering might be longer than
with imgmath, especially for pages with a lot of math expressions,
though.  (Yes, I see some of media documents have a lot of them.)

When you are detached from network connections, browsers will give
up and show those expressions in mathjax source code.

Mauro, wouldn't this approach work for you?

        Thanks, Akira

PS: Replies to the other topics will be sent separately.

Re: Expectation to --no-pdf option (was Re: [PATCH v2 0/5] Address some issues with sphinx detection)

[Jonathan Corbet]

Akira Yokosawa <akiyks@gmail.com> writes:

> In my tests, the mathjax extension works with all the versions of Sphinx
> I tested (1.7.9, 2.4.4, 3.4.3 (debian bullseye), 4.2.0 (openSUSE LEAP 15.4),
> and 5.0.2).
> Note that math expressions should look much sharper (vector fonts)
> than those from imgmath (pixel images).
> The time for a browser to complete the rendering might be longer than
> with imgmath, especially for pages with a lot of math expressions,
> though.  (Yes, I see some of media documents have a lot of them.)
>
> When you are detached from network connections, browsers will give
> up and show those expressions in mathjax source code.

Pulling in a bunch of JavaScript from the net while browsing the kernel
docs is not an entirely pleasing solution either.  But perhaps it's
preferable to loading the system with Latex.  I kind of wish we could
just do without the fancy math processing entirely, but I lost that
battle some time ago...:)

Mauro, have you looked at that option?

Thanks,

jon

Re: Expectation to --no-pdf option (was Re: [PATCH v2 0/5] Address some issues with sphinx detection)

[Mauro Carvalho Chehab]

Em Fri, 08 Jul 2022 08:02:52 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Akira Yokosawa <akiyks@gmail.com> writes:
> 
> > In my tests, the mathjax extension works with all the versions of Sphinx
> > I tested (1.7.9, 2.4.4, 3.4.3 (debian bullseye), 4.2.0 (openSUSE LEAP 15.4),
> > and 5.0.2).
> > Note that math expressions should look much sharper (vector fonts)
> > than those from imgmath (pixel images).
> > The time for a browser to complete the rendering might be longer than
> > with imgmath, especially for pages with a lot of math expressions,
> > though.  (Yes, I see some of media documents have a lot of them.)
> >
> > When you are detached from network connections, browsers will give
> > up and show those expressions in mathjax source code. 

> -extensions.append("sphinx.ext.imgmath")
> +extensions.append("sphinx.ext.mathjax")

There are two problems with this:

1. mathjax doesn't work for PDF output - nor would work if we add support
   for man pages some day;
2. Some Kernel developers disable javascript.

As imgmath works everywere, we opted to use it instead. We were
actually hoping that the lack of proper math support on Sphinx were
something that later Sphinx versions after 1.3.1 would have fixed. 
> 
> Pulling in a bunch of JavaScript from the net while browsing the kernel
> docs is not an entirely pleasing solution either.  But perhaps it's
> preferable to loading the system with Latex. 

If we could change conf.py to either load either one depending if it is
building PDF or not would be the best. The technical problem is that the 
javascript solution is not compatible with PDF output.

> I kind of wish we could
> just do without the fancy math processing entirely, but I lost that
> battle some time ago...:)

The media uAPI book uses math, and also vm/memory-model.rst. The last
one could easily use just a C code.

yet, it sounds funny to not have it used inside DRM. GPU processing
does lots of math. i would expect to have math used there for things
similar to the usages on media.

Trying to describe the colorspace conversion expressions there without
math would be really painful:

	https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/colorspaces-details.html?highlight=detailed+colorspace

Anyway, if you have some other solution, I'd gladly use something else.

> 
> Mauro, have you looked at that option?

Yes, but it got discarded due to pdf output. Not sure if this would work
on epub either.

> 
> Thanks,
> 
> jon

Re: Expectation to --no-pdf option (was Re: [PATCH v2 0/5] Address some issues with sphinx detection)

[Akira Yokosawa]

On Fri, 8 Jul 2022 15:59:10 +0100,
Mauro Carvalho Chehab wrote:
> Em Fri, 08 Jul 2022 08:02:52 -0600
> Jonathan Corbet <corbet@lwn.net> escreveu:
> 
>> Akira Yokosawa <akiyks@gmail.com> writes:
>>
>>> In my tests, the mathjax extension works with all the versions of Sphinx
>>> I tested (1.7.9, 2.4.4, 3.4.3 (debian bullseye), 4.2.0 (openSUSE LEAP 15.4),
>>> and 5.0.2).
>>> Note that math expressions should look much sharper (vector fonts)
>>> than those from imgmath (pixel images).
>>> The time for a browser to complete the rendering might be longer than
>>> with imgmath, especially for pages with a lot of math expressions,
>>> though.  (Yes, I see some of media documents have a lot of them.)
>>>
>>> When you are detached from network connections, browsers will give
>>> up and show those expressions in mathjax source code. 
> 
>> -extensions.append("sphinx.ext.imgmath")
>> +extensions.append("sphinx.ext.mathjax")
> 
> There are two problems with this:
> 
> 1. mathjax doesn't work for PDF output - nor would work if we add support
>    for man pages some day;

Hmm, if I understand what is written in the following page:
    https://www.sphinx-doc.org/en/master/usage/extensions/math.html

, both imgmath and mathjax extensions are relevant only for HTML output.

It says:

    Changed in version 1.8: Math support for non-HTML builders is integrated
    to sphinx-core. So mathbase extension is no longer needed.

When did you see the issue of "mathjax doesn't work for PDF output" ?

> 2. Some Kernel developers disable javascript.
OK, mathjax has no chance, then...

        Thanks, Akira

Re: Expectation to --no-pdf option (was Re: [PATCH v2 0/5] Address some issues with sphinx detection)

[Akira Yokosawa]

[-CC: ksummit-discuss]
On Sat, 9 Jul 2022 00:27:25 +0900, Akira Yokosawa wrote:
> On Fri, 8 Jul 2022 15:59:10 +0100,
> Mauro Carvalho Chehab wrote:
>> Em Fri, 08 Jul 2022 08:02:52 -0600
>> Jonathan Corbet <corbet@lwn.net> escreveu:
>>
>>> Akira Yokosawa <akiyks@gmail.com> writes:
>>>
>>>> In my tests, the mathjax extension works with all the versions of Sphinx
>>>> I tested (1.7.9, 2.4.4, 3.4.3 (debian bullseye), 4.2.0 (openSUSE LEAP 15.4),
>>>> and 5.0.2).
>>>> Note that math expressions should look much sharper (vector fonts)
>>>> than those from imgmath (pixel images).
>>>> The time for a browser to complete the rendering might be longer than
>>>> with imgmath, especially for pages with a lot of math expressions,
>>>> though.  (Yes, I see some of media documents have a lot of them.)
>>>>
>>>> When you are detached from network connections, browsers will give
>>>> up and show those expressions in mathjax source code. 
>>
>>> -extensions.append("sphinx.ext.imgmath")
>>> +extensions.append("sphinx.ext.mathjax")
>>
>> There are two problems with this:
>>
>> 1. mathjax doesn't work for PDF output - nor would work if we add support
>>    for man pages some day;
> 
> Hmm, if I understand what is written in the following page:
>     https://www.sphinx-doc.org/en/master/usage/extensions/math.html
> 
> , both imgmath and mathjax extensions are relevant only for HTML output.
> 
> It says:
> 
>     Changed in version 1.8: Math support for non-HTML builders is integrated
>     to sphinx-core. So mathbase extension is no longer needed.
> 
> When did you see the issue of "mathjax doesn't work for PDF output" ?

For the record,

I tested mathjax and PDF output with Sphinx 1.7.9, whose latex mode
can't handle nested tables.
I had no problem in building userspace-api.pdf and math expressions
in it look perfect.

So I believe mathjax does not affect PDF output.

Mauro wrote:
> As imgmath works everywere, we opted to use it instead. We were
> actually hoping that the lack of proper math support on Sphinx were
> something that later Sphinx versions after 1.3.1 would have fixed. 

I'm not going to test earlier versions of Sphinx and I have no idea
of what issue Mauro saw at the time, but it sounds to me the issue
has been fixed since.

> 
>> 2. Some Kernel developers disable javascript.
> OK, mathjax has no chance, then...

On the second thought, I think mathjax (latex-free "make htmldocs")
is good enough for test build purposes.  When javascript is disabled,
math expressions are rendered in mathjax source.

As conf.py is programmable, it is possible to choose sphinx.ext.imgmath
when dvipng is found on the build system.

As for sphinx-pre-install, what about adding an option

    --no-js   For those who disable javascript in their browser.

which provide the list of required packages for dvipng?
 
Thoughts?

Some comforting news for Jon:

Jon wrote:
> Pulling in a bunch of JavaScript from the net while browsing the kernel
> docs is not an entirely pleasing solution either.  But perhaps it's
> preferable to loading the system with Latex.

With Sphinx >=4.0, mathjax is loaded only when a page with math expressions
is opened.  Sphinx is improving in this regard.
    
        Thanks, Akira

> 
>         Thanks, Akira
> 

Re: Expectation to --no-pdf option (was Re: [PATCH v2 0/5] Address some issues with sphinx detection)

[Mauro Carvalho Chehab]

Em Sat, 9 Jul 2022 08:01:02 +0900
Akira Yokosawa <akiyks@gmail.com> escreveu:

> [-CC: ksummit-discuss]
> On Sat, 9 Jul 2022 00:27:25 +0900, Akira Yokosawa wrote:
> > On Fri, 8 Jul 2022 15:59:10 +0100,
> > Mauro Carvalho Chehab wrote:  
> >> Em Fri, 08 Jul 2022 08:02:52 -0600
> >> Jonathan Corbet <corbet@lwn.net> escreveu:
> >>  
> >>> Akira Yokosawa <akiyks@gmail.com> writes:
> >>>  
> >>>> In my tests, the mathjax extension works with all the versions of Sphinx
> >>>> I tested (1.7.9, 2.4.4, 3.4.3 (debian bullseye), 4.2.0 (openSUSE LEAP 15.4),
> >>>> and 5.0.2).
> >>>> Note that math expressions should look much sharper (vector fonts)
> >>>> than those from imgmath (pixel images).
> >>>> The time for a browser to complete the rendering might be longer than
> >>>> with imgmath, especially for pages with a lot of math expressions,
> >>>> though.  (Yes, I see some of media documents have a lot of them.)
> >>>>
> >>>> When you are detached from network connections, browsers will give
> >>>> up and show those expressions in mathjax source code.   
> >>  
> >>> -extensions.append("sphinx.ext.imgmath")
> >>> +extensions.append("sphinx.ext.mathjax")  
> >>
> >> There are two problems with this:
> >>
> >> 1. mathjax doesn't work for PDF output - nor would work if we add support
> >>    for man pages some day;  
> > 
> > Hmm, if I understand what is written in the following page:
> >     https://www.sphinx-doc.org/en/master/usage/extensions/math.html
> > 
> > , both imgmath and mathjax extensions are relevant only for HTML output.
> > 
> > It says:
> > 
> >     Changed in version 1.8: Math support for non-HTML builders is integrated
> >     to sphinx-core. So mathbase extension is no longer needed.
> > 
> > When did you see the issue of "mathjax doesn't work for PDF output" ?  
> 
> For the record,
> 
> I tested mathjax and PDF output with Sphinx 1.7.9, whose latex mode
> can't handle nested tables.
> I had no problem in building userspace-api.pdf and math expressions
> in it look perfect.
> 
> So I believe mathjax does not affect PDF output.

Did you also test epubdocs?

It was an issue when we decided to use imgmath. If this got fixed for
both supported non-html outputs, we can start using mathjax and get
rid of installing latex and dvipng.

> Mauro wrote:
> > As imgmath works everywere, we opted to use it instead. We were
> > actually hoping that the lack of proper math support on Sphinx were
> > something that later Sphinx versions after 1.3.1 would have fixed.   
> 
> I'm not going to test earlier versions of Sphinx and I have no idea
> of what issue Mauro saw at the time, but it sounds to me the issue
> has been fixed since.

Good.
 
> >   
> >> 2. Some Kernel developers disable javascript.  
> > OK, mathjax has no chance, then...  
> 
> On the second thought, I think mathjax (latex-free "make htmldocs")
> is good enough for test build purposes.  When javascript is disabled,
> math expressions are rendered in mathjax source.

Hmm... are there a way to use it with javascript disabled? If so, maybe
we can force it to always render math expressions during the build, instead
or relying on javascript at exec time.

> As conf.py is programmable, it is possible to choose sphinx.ext.imgmath
> when dvipng is found on the build system.

Not sure I like the idea. This would actually mean in practice that
all developers that are currently doing doc builds will keep using
imgmath, because they all have it already installed.

> As for sphinx-pre-install, what about adding an option
> 
>     --no-js   For those who disable javascript in their browser.
> 
> which provide the list of required packages for dvipng?

It is not that simple.

Sphinx has a configurable theme engine. On our builds, we're using
since the beginning the RTD (readthedocs) theme as default, but
recent versions default to classic if sphinx_rtd_theme package is
not installed.

All themes I know that provide a search button use JS to implement
such feature.

So, a "--no-js" won't provide a javascript-free build environment.

-

On a side discussion, should we keep recommending the install of 
sphinx_rtd_theme? It is not mandatory anymore to have it installed,
and the theme is more a matter of personal preferences. 

Also, when testing or modifying the docs, the theme doesn't really
matter.

So, IMHO, we could stop recommending it.

Regards,
Mauro   

Re: Expectation to --no-pdf option (was Re: [PATCH v2 0/5] Address some issues with sphinx detection)

[Akira Yokosawa]

On Sat, 9 Jul 2022 08:59:46 +0100,
Mauro Carvalho Chehab wrote:
> Em Sat, 9 Jul 2022 08:01:02 +0900
> Akira Yokosawa <akiyks@gmail.com> escreveu:
> 
>> [-CC: ksummit-discuss]
>> On Sat, 9 Jul 2022 00:27:25 +0900, Akira Yokosawa wrote:
>>> On Fri, 8 Jul 2022 15:59:10 +0100,
>>> Mauro Carvalho Chehab wrote:  
>>>> Em Fri, 08 Jul 2022 08:02:52 -0600
>>>> Jonathan Corbet <corbet@lwn.net> escreveu:
>>>>  
>>>>> Akira Yokosawa <akiyks@gmail.com> writes:
>>>>>  
>>>>>> In my tests, the mathjax extension works with all the versions of Sphinx
>>>>>> I tested (1.7.9, 2.4.4, 3.4.3 (debian bullseye), 4.2.0 (openSUSE LEAP 15.4),
>>>>>> and 5.0.2).
>>>>>> Note that math expressions should look much sharper (vector fonts)
>>>>>> than those from imgmath (pixel images).
>>>>>> The time for a browser to complete the rendering might be longer than
>>>>>> with imgmath, especially for pages with a lot of math expressions,
>>>>>> though.  (Yes, I see some of media documents have a lot of them.)
>>>>>>
>>>>>> When you are detached from network connections, browsers will give
>>>>>> up and show those expressions in mathjax source code.   
>>>>  
>>>>> -extensions.append("sphinx.ext.imgmath")
>>>>> +extensions.append("sphinx.ext.mathjax")  
>>>>
>>>> There are two problems with this:
>>>>
>>>> 1. mathjax doesn't work for PDF output - nor would work if we add support
>>>>    for man pages some day;  
>>>
>>> Hmm, if I understand what is written in the following page:
>>>     https://www.sphinx-doc.org/en/master/usage/extensions/math.html
>>>
>>> , both imgmath and mathjax extensions are relevant only for HTML output.
>>>
>>> It says:
>>>
>>>     Changed in version 1.8: Math support for non-HTML builders is integrated
>>>     to sphinx-core. So mathbase extension is no longer needed.
>>>
>>> When did you see the issue of "mathjax doesn't work for PDF output" ?  
>>
>> For the record,
>>
>> I tested mathjax and PDF output with Sphinx 1.7.9, whose latex mode
>> can't handle nested tables.
>> I had no problem in building userspace-api.pdf and math expressions
>> in it look perfect.
>>
>> So I believe mathjax does not affect PDF output.
> 
> Did you also test epubdocs?
No, I didn't.

Now I see that mathjax doesn't work with epubdocs.

I think I found a means to use mathjax only for htmldocs.
It is mentioned in an issue at Sphinx:

    https://github.com/sphinx-doc/sphinx/issues/3740#issuecomment-385503305

Actually, it turns out that the mathjax extension is loaded by default
since Sphinx 1.8.  A simple assignment in conf.py can enable it:

    html_math_renderer = 'mathjax'

As a matter of fact, it can be enabled via the SPHINXOPTS make variable.

Just try this against current docs-next:

    make SPHINXDIRS=userspace-api SPHINXOPTS="-D html_math_renderer='mathjax'" htmldocs

> 
> It was an issue when we decided to use imgmath. If this got fixed for
> both supported non-html outputs, we can start using mathjax and get
> rid of installing latex and dvipng.

As far as I know, ebook readers don't understand javascript and mathjax
doesn't work. imgmath is mandatory for epub.

> 
>> Mauro wrote:
>>> As imgmath works everywere, we opted to use it instead. We were
>>> actually hoping that the lack of proper math support on Sphinx were
>>> something that later Sphinx versions after 1.3.1 would have fixed.   
>>
>> I'm not going to test earlier versions of Sphinx and I have no idea
>> of what issue Mauro saw at the time, but it sounds to me the issue
>> has been fixed since.
> 
> Good.

I'm suspecting now that pdfdocs had no issue with mathjax, but I'm not sure.

>  
>>>   
>>>> 2. Some Kernel developers disable javascript.  
>>> OK, mathjax has no chance, then...  
>>
>> On the second thought, I think mathjax (latex-free "make htmldocs")
>> is good enough for test build purposes.  When javascript is disabled,
>> math expressions are rendered in mathjax source.
> 
> Hmm... are there a way to use it with javascript disabled? 

No chance. Javascript is a must for mathjax.

>                                                            If so, maybe
> we can force it to always render math expressions during the build, instead
> or relying on javascript at exec time.
> 
>> As conf.py is programmable, it is possible to choose sphinx.ext.imgmath
>> when dvipng is found on the build system.
> 
> Not sure I like the idea. This would actually mean in practice that
> all developers that are currently doing doc builds will keep using
> imgmath, because they all have it already installed.

The purpose here is to get more developers who actually run "make htmldocs"
on their (not limited to) documentation changes.  Difference of math rendering
doesn't matter much.

One possible improvement for imgmath is to use its SVG format option, which
uses dvisvgm instead.  It will provide scalable math expressions similar to
those provided by mathjax.  Of course you need properly working dvisvgm.

> 
>> As for sphinx-pre-install, what about adding an option
>>
>>     --no-js   For those who disable javascript in their browser.
>>
>> which provide the list of required packages for dvipng?
> 
> It is not that simple.
> 
> Sphinx has a configurable theme engine. On our builds, we're using
> since the beginning the RTD (readthedocs) theme as default, but
> recent versions default to classic if sphinx_rtd_theme package is
> not installed.
> 
> All themes I know that provide a search button use JS to implement
> such feature.
> 
> So, a "--no-js" won't provide a javascript-free build environment.
I see.

What about "--imgmath" ? 

> 
> -
> 
> On a side discussion, should we keep recommending the install of 
> sphinx_rtd_theme? It is not mandatory anymore to have it installed,
> and the theme is more a matter of personal preferences. 
> 
> Also, when testing or modifying the docs, the theme doesn't really
> matter.
> 
> So, IMHO, we could stop recommending it.
I agree.

        Thanks, Akira

> 
> Regards,
> Mauro   

Re: [PATCH v2 0/5] Address some issues with sphinx detection

[Tomasz Warniełło]

On Sat,  2 Jul 2022 11:11:24 +0100
Mauro Carvalho Chehab <mchehab@kernel.org> wrote:

> There are two options to install Sphinx:
> 
> 	- via distro-provided packages;
> 	- via pip, using virtualenv/venv.

There are countless ways to install any software. So much for my entrée.

*

I don't know this script. Just what I've read in this thread and what I could
extract from it in no more than ten minutes. I understand what it does - or
rather I do just approximately. And alright! Why not help the Gentoos and
Fedoras and Etc's. Noble.

But - wouldn't it be million times easier for all parties to base the support
on a Dockerfile? Or a script producing the right Dockerfile possibly,
considering matters like Java Script opt- ins and outs. If this is not a
cheap pedantry in this context. I don't know.

These here bases look quite official:

https://hub.docker.com/u/sphinxdoc

(Not that I've used them or know them.)

The admins of whichever distribution will find their ways in the mesh, raw
requirements suffice.

The non-admins may have problems with installing software and also running
Docker, as that provides su powers in the straight scenario. `pip` comes as
an alternative, if I'm not wrong. So maybe this pair is the golden arrow?

--thanks
T