* [PATCH RFC v2 0/2] add script for containers building the documentation
@ 2024-12-17 16:08 Quentin Schulz
2024-12-17 16:08 ` [PATCH RFC v2 1/2] docs: use literalinclude for system requirements Quentin Schulz
2024-12-17 16:08 ` [PATCH RFC v2 2/2] tools: add script for building documentation inside containers Quentin Schulz
0 siblings, 2 replies; 6+ messages in thread
From: Quentin Schulz @ 2024-12-17 16:08 UTC (permalink / raw)
To: Quentin Schulz, docs
Based on initial work done by Antonin Godard[1].
This does two things:
- move all the installation steps to fulfil the system requirements for
each distro into separate files,
- add a script for using those separate files for building a container
which allows to build the documentation within,
Note that only instructions for Debian Bookworm (12), Ubuntu 22.04 and
24.04, and Fedora 38, 39 and 40, currently work.
openSUSE seems to be running the sphinx build from a 3.6 Python
interpreter which has an incompatible API with the Python calls used in
Sphinx for our documentation. This is fixed/worked-around with this
patch from Antonin:
https://lore.kernel.org/yocto-docs/20241217-capture-output-compat-v1-1-d8f147a12eea@bootlin.com/
15.4, 15.5 and 15.6 tested with that patch applied.
Almalinux instructions are missing a bunch of packages and I reached a
point where I couldn't find a package providing everysel.sty though
texlive-ms seems to be providing it according to pkgs.org.
How far I went:
"""
diff --git a/documentation/tools/almalinux_host_packages_docs_pdf.sh b/documentation/tools/almalinux_host_packages_docs_pdf.sh
index 8341eb8c2..19f45312b 100644
--- a/documentation/tools/almalinux_host_packages_docs_pdf.sh
+++ b/documentation/tools/almalinux_host_packages_docs_pdf.sh
@@ -1 +1,5 @@
-sudo dnf install latexmk texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-tex-gyre texlive-xetex
+sudo dnf install -y epel-release
+sudo yum install dnf-plugins-core
+sudo dnf config-manager --set-enabled crb
+sudo dnf makecache
+sudo dnf install -y latexmk texlive-capt-of texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexrecommended texlive-collection-xetex texlive-ctex texlive-fncychap texlive-framed texlive-gnu-freefont texlive-ms texlive-needspace texlive-tabulary texlive-tex-gyre texlive-upquote texlive-wrapfig texlive-xetex
"""
Note the dnf install -y for the packages is required for installing the
GPG key of one of the repos noninteractively.
Older Debian and Ubuntu both lack python3-saneyaml and similar Python
packages in their packagefeed.
Ubuntu 23.04 package feed isn't accessible anymore.
Note that this script cannot be run in parallel right now as all
containers would be sharing the same build directory. This could be
mitigated by allowing BUILDDIR to be overridden from the environment
before being used by make and have it overridden by the script to a
different location. There are probably other corner cases not handled
(likely concurrent set_versions.py for example?).
[1] https://lore.kernel.org/yocto-docs/20241205-docs-build-dockerfiles-v2-1-047cb3245adf@bootlin.com/
[2] https://lore.kernel.org/yocto-docs/20241202110128.27711-1-guenael.muller@smile.fr/
Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
---
Changes in v2:
- rebase on master,
- fix issues for recent Ubuntu by setting the timezone,
- add support for building pdf docs from within container,
- fix support for passing multiple make targets to the script
- have all pip3-packages-requiring-distros use one file to make it
easier to maintain,
- put scripts into a dedicated repo,
- rename variables and filenames to remove host_packages/packages from
them to make things a bit more digest,
- fix incorrect comment in script about reading poky.yaml.in,
- Link to v1: https://lore.kernel.org/r/20241210-instructions-shell-container-v1-0-6a7cdc404ff4@cherry.de
---
Quentin Schulz (2):
docs: use literalinclude for system requirements
tools: add script for building documentation inside containers
documentation/poky.yaml.in | 234 ---------------------
documentation/ref-manual/system-requirements.rst | 82 +++++---
documentation/tools/Containerfile.almalinux | 1 +
documentation/tools/Containerfile.apt | 26 +++
documentation/tools/Containerfile.debian | 1 +
documentation/tools/Containerfile.dnf | 25 +++
documentation/tools/Containerfile.fedora | 1 +
documentation/tools/Containerfile.ubuntu | 1 +
documentation/tools/Containerfile.zypper | 24 +++
documentation/tools/build-docs-container | 113 ++++++++++
.../tools/host_packages_scripts/almalinux_docs.sh | 1 +
.../host_packages_scripts/almalinux_docs_pdf.sh | 1 +
.../host_packages_scripts/almalinux_essential.sh | 5 +
.../tools/host_packages_scripts/fedora_docs.sh | 1 +
.../tools/host_packages_scripts/fedora_docs_pdf.sh | 1 +
.../host_packages_scripts/fedora_essential.sh | 1 +
.../tools/host_packages_scripts/opensuse_docs.sh | 1 +
.../host_packages_scripts/opensuse_docs_pdf.sh | 1 +
.../host_packages_scripts/opensuse_essential.sh | 2 +
.../tools/host_packages_scripts/pip3_docs.sh | 1 +
.../tools/host_packages_scripts/ubuntu_docs.sh | 1 +
.../tools/host_packages_scripts/ubuntu_docs_pdf.sh | 1 +
.../host_packages_scripts/ubuntu_essential.sh | 1 +
23 files changed, 257 insertions(+), 269 deletions(-)
---
base-commit: 02c090718ac602f7d4760dd28dadbf0631668d49
change-id: 20241210-instructions-shell-container-c5a84c043a35
Best regards,
--
Quentin Schulz <quentin.schulz@cherry.de>
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH RFC v2 1/2] docs: use literalinclude for system requirements
2024-12-17 16:08 [PATCH RFC v2 0/2] add script for containers building the documentation Quentin Schulz
@ 2024-12-17 16:08 ` Quentin Schulz
2024-12-17 16:08 ` [PATCH RFC v2 2/2] tools: add script for building documentation inside containers Quentin Schulz
1 sibling, 0 replies; 6+ messages in thread
From: Quentin Schulz @ 2024-12-17 16:08 UTC (permalink / raw)
To: Quentin Schulz, docs
From: Quentin Schulz <quentin.schulz@cherry.de>
The YAML variables for the host dependencies are updated by hand and
actually only used inside code blocks.
Let's migrate all instructions into separate shell scripts that are then
literalinclude'd into the Sphinx documentation.
This allows a few things:
- ability to run shellcheck on the scripts if we ever want to
- manually calling the appropriate script from a supported distro to
build stuff (distro or bitbake/yocto stuff)
- use this script to create containers to do CI of documentation on
different distros, to make sure our instructions are all up to date,
Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
---
documentation/poky.yaml.in | 234 ---------------------
documentation/ref-manual/system-requirements.rst | 82 +++++---
.../tools/host_packages_scripts/almalinux_docs.sh | 1 +
.../host_packages_scripts/almalinux_docs_pdf.sh | 1 +
.../host_packages_scripts/almalinux_essential.sh | 5 +
.../tools/host_packages_scripts/fedora_docs.sh | 1 +
.../tools/host_packages_scripts/fedora_docs_pdf.sh | 1 +
.../host_packages_scripts/fedora_essential.sh | 1 +
.../tools/host_packages_scripts/opensuse_docs.sh | 1 +
.../host_packages_scripts/opensuse_docs_pdf.sh | 1 +
.../host_packages_scripts/opensuse_essential.sh | 2 +
.../tools/host_packages_scripts/pip3_docs.sh | 1 +
.../tools/host_packages_scripts/ubuntu_docs.sh | 1 +
.../tools/host_packages_scripts/ubuntu_docs_pdf.sh | 1 +
.../host_packages_scripts/ubuntu_essential.sh | 1 +
15 files changed, 65 insertions(+), 269 deletions(-)
diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in
index 0933cbbd351985899ecaea94ae42748bd7ba1753..d637952bbed13b22afac95bd77aef9f2e35fa630 100644
--- a/documentation/poky.yaml.in
+++ b/documentation/poky.yaml.in
@@ -25,237 +25,3 @@ MIN_DISK_SPACE : "90"
MIN_DISK_SPACE_RM_WORK : "40"
# RAM (Gbytes) needed to generate qemux86-64 core-image-sato on Ubuntu 22.04 (x86-64) on a 4 core system
MIN_RAM : "8"
-
-#
-# Dependencies
-#
-
-# Shared between distros
-PIP3_HOST_PACKAGES_DOC: sphinx sphinx_rtd_theme pyyaml
-
-UBUNTU_DEBIAN_HOST_PACKAGES_ESSENTIAL: >-
- build-essential
- chrpath
- cpio
- debianutils
- diffstat
- file
- gawk
- gcc
- git
- iputils-ping
- libacl1
- liblz4-tool
- locales
- python3
- python3-git
- python3-jinja2
- python3-pexpect
- python3-pip
- python3-subunit
- socat
- texinfo
- unzip
- wget
- xz-utils
- zstd
-
-UBUNTU_DEBIAN_HOST_PACKAGES_DOC: >-
- git
- librsvg2-bin
- locales
- make
- python3-saneyaml
- python3-sphinx-rtd-theme
- sphinx
-
-UBUNTU_DEBIAN_HOST_PACKAGES_DOC_PDF: >-
- fonts-freefont-otf
- latexmk
- tex-gyre
- texlive-fonts-extra
- texlive-fonts-recommended
- texlive-lang-all
- texlive-latex-extra
- texlive-latex-recommended
- texlive-xetex
-
-FEDORA_HOST_PACKAGES_ESSENTIAL: >-
- bzip2
- ccache
- chrpath
- cpio
- cpp
- diffstat
- diffutils
- file
- findutils
- gawk
- gcc
- gcc-c++
- git
- glibc-devel
- glibc-langpack-en
- gzip
- hostname
- libacl
- lz4
- make
- patch
- perl
- perl-Data-Dumper
- perl-File-Compare
- perl-File-Copy
- perl-FindBin
- perl-Text-ParseWords
- perl-Thread-Queue
- perl-bignum
- perl-locale
- python
- python3
- python3-GitPython
- python3-jinja2
- python3-pexpect
- python3-pip
- rpcgen
- socat
- tar
- texinfo
- unzip
- wget
- which
- xz
- zstd
-
-FEDORA_HOST_PACKAGES_DOC: >-
- git
- glibc-locale-source
- librsvg2-tools
- make
- python3-pip
- which
-
-FEDORA_HOST_PACKAGES_DOC_PDF: >-
- 'texlive-collection-lang*'
- latexmk
- texlive-collection-fontsextra
- texlive-collection-fontsrecommended
- texlive-collection-latex
- texlive-collection-latexextra
- texlive-collection-latexrecommended
- texlive-collection-xetex
- texlive-fncychap
- texlive-gnu-freefont
- texlive-tex-gyre
- texlive-xetex
-
-OPENSUSE_HOST_PACKAGES_ESSENTIAL: >-
- bzip2
- chrpath
- diffstat
- gcc
- gcc-c++
- git
- gzip
- hostname
- libacl1
- lz4
- make
- makeinfo
- patch
- python
- python-curses
- python-xml
- python3
- python3-Jinja2
- python3-curses
- python3-pexpect
- python3-pip
- rpcgen
- socat
- tar
- wget
- which
- xz
- zstd
-
-OPENSUSE_PIP3_HOST_PACKAGES_ESSENTIAL: GitPython
-
-OPENSUSE_HOST_PACKAGES_DOC: >-
- git
- glibc-i18ndata
- make
- python3-pip
- rsvg-convert
- which
-
-OPENSUSE_HOST_PACKAGES_DOC_PDF: >-
- 'texlive-collection-lang*'
- texlive-collection-fontsextra
- texlive-collection-fontsrecommended
- texlive-collection-latex
- texlive-collection-latexextra
- texlive-collection-latexrecommended
- texlive-collection-xetex
- texlive-fncychap
- texlive-gnu-freefont
- texlive-latexmk
- texlive-tex-gyre
- texlive-xetex
-
-ALMALINUX_HOST_PACKAGES_ESSENTIAL: >-
- bzip2
- ccache
- chrpath
- cpio
- cpp
- diffstat
- diffutils
- gawk
- gcc
- gcc-c++
- git
- glibc-devel
- glibc-langpack-en
- gzip
- libacl
- lz4
- make
- patch
- perl
- perl-Data-Dumper
- perl-Text-ParseWords
- perl-Thread-Queue
- python3
- python3-GitPython
- python3-jinja2
- python3-pexpect
- python3-pip
- rpcgen
- socat
- tar
- texinfo
- unzip
- wget
- which
- xz
- zstd
-
-ALMALINUX_HOST_PACKAGES_DOC: >-
- git
- glibc-locale-source
- librsvg2-tools
- make
- python3-pip
- which
-
-ALMALINUX_HOST_PACKAGES_DOC_PDF: >-
- latexmk
- texlive-collection-fontsrecommended
- texlive-collection-latex
- texlive-collection-latexrecommended
- texlive-collection-xetex
- texlive-fncychap
- texlive-gnu-freefont
- texlive-tex-gyre
- texlive-xetex
diff --git a/documentation/ref-manual/system-requirements.rst b/documentation/ref-manual/system-requirements.rst
index b087d374d23a71104dc9c050e96e4a2c20f5abb0..93b95a7fb251c4a0192c2d29a1dbf365eb69871e 100644
--- a/documentation/ref-manual/system-requirements.rst
+++ b/documentation/ref-manual/system-requirements.rst
@@ -158,9 +158,10 @@ Ubuntu and Debian
-----------------
Here are the packages needed to build an image on a headless system
-with a supported Ubuntu or Debian Linux distribution::
+with a supported Ubuntu or Debian Linux distribution:
- $ sudo apt install &UBUNTU_DEBIAN_HOST_PACKAGES_ESSENTIAL;
+.. literalinclude:: ../tools/host_packages_scripts/ubuntu_essential.sh
+ :language: shell
You also need to ensure you have the ``en_US.UTF-8`` locale enabled::
@@ -189,64 +190,71 @@ If this is not the case, you can reconfigure the ``locales`` package to add it
$ sudo apt build-dep qemu
$ sudo apt remove oss4-dev
-Here are the packages needed to build Project documentation manuals::
+Here are the packages needed to build Project documentation manuals:
- $ sudo apt install &UBUNTU_DEBIAN_HOST_PACKAGES_DOC;
+.. literalinclude:: ../tools/host_packages_scripts/ubuntu_docs.sh
+ :language: shell
In addition to the previous packages, here are the packages needed to build the
-documentation in PDF format::
+documentation in PDF format:
- $ sudo apt install &UBUNTU_DEBIAN_HOST_PACKAGES_DOC_PDF;
+.. literalinclude:: ../tools/host_packages_scripts/ubuntu_docs_pdf.sh
+ :language: shell
Fedora Packages
---------------
Here are the packages needed to build an image on a headless system
-with a supported Fedora Linux distribution::
+with a supported Fedora Linux distribution:
- $ sudo dnf install &FEDORA_HOST_PACKAGES_ESSENTIAL;
+.. literalinclude:: ../tools/host_packages_scripts/fedora_essential.sh
+ :language: shell
-Here are the packages needed to build Project documentation manuals::
+Here are the packages needed to build Project documentation manuals:
- $ sudo dnf install &FEDORA_HOST_PACKAGES_DOC;
- $ sudo pip3 install &PIP3_HOST_PACKAGES_DOC;
+.. literalinclude:: ../tools/host_packages_scripts/fedora_docs.sh
+ :language: shell
+
+.. literalinclude:: ../tools/host_packages_scripts/pip3_docs.sh
+ :language: shell
In addition to the previous packages, here are the packages needed to build the
-documentation in PDF format::
+documentation in PDF format:
- $ sudo dnf install &FEDORA_HOST_PACKAGES_DOC_PDF;
+.. literalinclude:: ../tools/host_packages_scripts/fedora_docs_pdf.sh
+ :language: shell
openSUSE Packages
-----------------
Here are the packages needed to build an image on a headless system
-with a supported openSUSE distribution::
+with a supported openSUSE distribution:
- $ sudo zypper install &OPENSUSE_HOST_PACKAGES_ESSENTIAL;
- $ sudo pip3 install &OPENSUSE_PIP3_HOST_PACKAGES_ESSENTIAL;
+.. literalinclude:: ../tools/host_packages_scripts/opensuse_essential.sh
+ :language: shell
-Here are the packages needed to build Project documentation manuals::
+Here are the packages needed to build Project documentation manuals:
- $ sudo zypper install &OPENSUSE_HOST_PACKAGES_DOC;
- $ sudo pip3 install &PIP3_HOST_PACKAGES_DOC;
+.. literalinclude:: ../tools/host_packages_scripts/opensuse_docs.sh
+ :language: shell
+
+.. literalinclude:: ../tools/host_packages_scripts/pip3_docs.sh
+ :language: shell
In addition to the previous packages, here are the packages needed to build the
-documentation in PDF format::
-
- $ sudo zypper install &OPENSUSE_HOST_PACKAGES_DOC_PDF;
+documentation in PDF format:
+.. literalinclude:: ../tools/host_packages_scripts/opensuse_docs_pdf.sh
+ :language: shell
AlmaLinux Packages
------------------
Here are the packages needed to build an image on a headless system
-with a supported AlmaLinux distribution::
+with a supported AlmaLinux distribution:
- $ sudo dnf install -y epel-release
- $ sudo yum install dnf-plugins-core
- $ sudo dnf config-manager --set-enabled crb
- $ sudo dnf makecache
- $ sudo dnf install &ALMALINUX_HOST_PACKAGES_ESSENTIAL;
+.. literalinclude:: ../tools/host_packages_scripts/almalinux_essential.sh
+ :language: shell
.. note::
@@ -261,15 +269,13 @@ with a supported AlmaLinux distribution::
- The ``makecache`` command consumes additional Metadata from
``epel-release``.
-Here are the packages needed to build Project documentation manuals::
+Here are the packages needed to build Project documentation manuals:
- $ sudo dnf install &ALMALINUX_HOST_PACKAGES_DOC;
- $ sudo pip3 install &PIP3_HOST_PACKAGES_DOC;
+.. literalinclude:: ../tools/host_packages_scripts/almalinux_docs.sh
+ :language: shell
-In addition to the previous packages, here are the packages needed to build the
-documentation in PDF format::
-
- $ sudo dnf install &ALMALINUX_HOST_PACKAGES_DOC_PDF;
+.. literalinclude:: ../tools/host_packages_scripts/pip3_docs.sh
+ :language: shell
.. warning::
@@ -278,6 +284,12 @@ documentation in PDF format::
``texlive-collection-latexextra``, so you may run into issues. These may be
installed using `tlmgr <https://tug.org/texlive/tlmgr.html>`_.
+In addition to the previous packages, here are the packages needed to build the
+documentation in PDF format:
+
+.. literalinclude:: ../tools/host_packages_scripts/almalinux_docs_pdf.sh
+ :language: shell
+
.. _system-requirements-buildtools:
Required Git, tar, Python, make and gcc Versions
diff --git a/documentation/tools/host_packages_scripts/almalinux_docs.sh b/documentation/tools/host_packages_scripts/almalinux_docs.sh
new file mode 100644
index 0000000000000000000000000000000000000000..8188d529a1e9b8034e5b14ed4e9e633e7ae0600b
--- /dev/null
+++ b/documentation/tools/host_packages_scripts/almalinux_docs.sh
@@ -0,0 +1 @@
+sudo dnf install git glibc-locale-source librsvg2-tools make python3-pip which
diff --git a/documentation/tools/host_packages_scripts/almalinux_docs_pdf.sh b/documentation/tools/host_packages_scripts/almalinux_docs_pdf.sh
new file mode 100644
index 0000000000000000000000000000000000000000..8341eb8c252d50b8a5cd3ff09ff013b1d49249bf
--- /dev/null
+++ b/documentation/tools/host_packages_scripts/almalinux_docs_pdf.sh
@@ -0,0 +1 @@
+sudo dnf install latexmk texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-tex-gyre texlive-xetex
diff --git a/documentation/tools/host_packages_scripts/almalinux_essential.sh b/documentation/tools/host_packages_scripts/almalinux_essential.sh
new file mode 100644
index 0000000000000000000000000000000000000000..16f25fa1f29c84d512df97988f527d1ea834bea7
--- /dev/null
+++ b/documentation/tools/host_packages_scripts/almalinux_essential.sh
@@ -0,0 +1,5 @@
+sudo dnf install -y epel-release
+sudo yum install dnf-plugins-core
+sudo dnf config-manager --set-enabled crb
+sudo dnf makecache
+sudo dnf install bzip2 ccache chrpath cpio cpp diffstat diffutils gawk gcc gcc-c++ git glibc-devel glibc-langpack-en gzip libacl lz4 make patch perl perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue python3 python3-GitPython python3-jinja2 python3-pexpect python3-pip rpcgen socat tar texinfo unzip wget which xz zstd
diff --git a/documentation/tools/host_packages_scripts/fedora_docs.sh b/documentation/tools/host_packages_scripts/fedora_docs.sh
new file mode 100644
index 0000000000000000000000000000000000000000..8188d529a1e9b8034e5b14ed4e9e633e7ae0600b
--- /dev/null
+++ b/documentation/tools/host_packages_scripts/fedora_docs.sh
@@ -0,0 +1 @@
+sudo dnf install git glibc-locale-source librsvg2-tools make python3-pip which
diff --git a/documentation/tools/host_packages_scripts/fedora_docs_pdf.sh b/documentation/tools/host_packages_scripts/fedora_docs_pdf.sh
new file mode 100644
index 0000000000000000000000000000000000000000..816f1c36ec35e046782d341bb70dff1834bfd8a1
--- /dev/null
+++ b/documentation/tools/host_packages_scripts/fedora_docs_pdf.sh
@@ -0,0 +1 @@
+sudo dnf install 'texlive-collection-lang*' latexmk texlive-collection-fontsextra texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexextra texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-tex-gyre texlive-xetex
diff --git a/documentation/tools/host_packages_scripts/fedora_essential.sh b/documentation/tools/host_packages_scripts/fedora_essential.sh
new file mode 100644
index 0000000000000000000000000000000000000000..4841853e1e224683b563c8d6112446f0633a081b
--- /dev/null
+++ b/documentation/tools/host_packages_scripts/fedora_essential.sh
@@ -0,0 +1 @@
+sudo dnf install bzip2 ccache chrpath cpio cpp diffstat diffutils file findutils gawk gcc gcc-c++ git glibc-devel glibc-langpack-en gzip hostname libacl lz4 make patch perl perl-Data-Dumper perl-File-Compare perl-File-Copy perl-FindBin perl-Text-ParseWords perl-Thread-Queue perl-bignum perl-locale python python3 python3-GitPython python3-jinja2 python3-pexpect python3-pip rpcgen socat tar texinfo unzip wget which xz zstd
diff --git a/documentation/tools/host_packages_scripts/opensuse_docs.sh b/documentation/tools/host_packages_scripts/opensuse_docs.sh
new file mode 100644
index 0000000000000000000000000000000000000000..7d36eb0f99668168dc4fed44d71181338bdc59a3
--- /dev/null
+++ b/documentation/tools/host_packages_scripts/opensuse_docs.sh
@@ -0,0 +1 @@
+sudo zypper install git glibc-i18ndata make python3-pip rsvg-convert which
diff --git a/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh b/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh
new file mode 100644
index 0000000000000000000000000000000000000000..ee9f07886ce1c0eac8e1d1118cda8ddc496f0139
--- /dev/null
+++ b/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh
@@ -0,0 +1 @@
+sudo zypper install 'texlive-collection-lang*' texlive-collection-fontsextra texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexextra texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-latexmk texlive-tex-gyre texlive-xetex
diff --git a/documentation/tools/host_packages_scripts/opensuse_essential.sh b/documentation/tools/host_packages_scripts/opensuse_essential.sh
new file mode 100644
index 0000000000000000000000000000000000000000..a784f4a5dc05f3734177f08ed97663a04d20a978
--- /dev/null
+++ b/documentation/tools/host_packages_scripts/opensuse_essential.sh
@@ -0,0 +1,2 @@
+sudo zypper install bzip2 chrpath diffstat gcc gcc-c++ git gzip hostname libacl1 lz4 make makeinfo patch python python-curses python-xml python3 python3-Jinja2 python3-curses python3-pexpect python3-pip rpcgen socat tar wget which xz zstd
+sudo pip3 install GitPython
diff --git a/documentation/tools/host_packages_scripts/pip3_docs.sh b/documentation/tools/host_packages_scripts/pip3_docs.sh
new file mode 100644
index 0000000000000000000000000000000000000000..fd6ad98053fb6aa1ff55669191dc99578c830455
--- /dev/null
+++ b/documentation/tools/host_packages_scripts/pip3_docs.sh
@@ -0,0 +1 @@
+sudo pip3 install sphinx sphinx_rtd_theme pyyaml
diff --git a/documentation/tools/host_packages_scripts/ubuntu_docs.sh b/documentation/tools/host_packages_scripts/ubuntu_docs.sh
new file mode 100644
index 0000000000000000000000000000000000000000..d2a1832e279057519321a1fb7820efd60d886564
--- /dev/null
+++ b/documentation/tools/host_packages_scripts/ubuntu_docs.sh
@@ -0,0 +1 @@
+sudo apt install git librsvg2-bin locales make python3-saneyaml python3-saneyaml python3-sphinx-rtd-theme sphinx
diff --git a/documentation/tools/host_packages_scripts/ubuntu_docs_pdf.sh b/documentation/tools/host_packages_scripts/ubuntu_docs_pdf.sh
new file mode 100644
index 0000000000000000000000000000000000000000..d6310bd21deae81df32d0cb843aab4a5622ac4fa
--- /dev/null
+++ b/documentation/tools/host_packages_scripts/ubuntu_docs_pdf.sh
@@ -0,0 +1 @@
+sudo apt install fonts-freefont-otf latexmk tex-gyre texlive-fonts-extra texlive-fonts-recommended texlive-lang-all texlive-latex-extra texlive-latex-recommended texlive-xetex
diff --git a/documentation/tools/host_packages_scripts/ubuntu_essential.sh b/documentation/tools/host_packages_scripts/ubuntu_essential.sh
new file mode 100644
index 0000000000000000000000000000000000000000..f3388c88c93d885e3b035651e434c669654ee241
--- /dev/null
+++ b/documentation/tools/host_packages_scripts/ubuntu_essential.sh
@@ -0,0 +1 @@
+sudo apt-get install build-essential chrpath cpio debianutils diffstat file gawk gcc git iputils-ping libacl1 liblz4-tool locales python3 python3-git python3-jinja2 python3-pexpect python3-pip python3-subunit socat texinfo unzip wget xz-utils zstd
--
2.47.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH RFC v2 2/2] tools: add script for building documentation inside containers
2024-12-17 16:08 [PATCH RFC v2 0/2] add script for containers building the documentation Quentin Schulz
2024-12-17 16:08 ` [PATCH RFC v2 1/2] docs: use literalinclude for system requirements Quentin Schulz
@ 2024-12-17 16:08 ` Quentin Schulz
2024-12-26 10:41 ` [docs] " Antonin Godard
1 sibling, 1 reply; 6+ messages in thread
From: Quentin Schulz @ 2024-12-17 16:08 UTC (permalink / raw)
To: Quentin Schulz, docs
From: Quentin Schulz <quentin.schulz@cherry.de>
This adds a script for building a container and building the
documentation within that new container image.
The openSUSE instructions now require a --non-interactive flag otherwise
they fail to run. Sadly there doesn't seem to be a way to have this in
an environment variable à-la DEBIAN_FRONTEND=noninteractive.
Somehow tzdata package in Ubuntu doesn't respect
DEBIAN_FRONTEND=noninteractive hence why the timezone needs to be set by
hand.
Note that only instructions for Debian Bookworm (12), Ubuntu 22.04 and
24.04, and Fedora 38, 39 and 40, currently work.
Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
---
documentation/tools/Containerfile.almalinux | 1 +
documentation/tools/Containerfile.apt | 26 +++++
documentation/tools/Containerfile.debian | 1 +
documentation/tools/Containerfile.dnf | 25 +++++
documentation/tools/Containerfile.fedora | 1 +
documentation/tools/Containerfile.ubuntu | 1 +
documentation/tools/Containerfile.zypper | 24 +++++
documentation/tools/build-docs-container | 113 +++++++++++++++++++++
.../tools/host_packages_scripts/opensuse_docs.sh | 2 +-
.../host_packages_scripts/opensuse_docs_pdf.sh | 2 +-
.../host_packages_scripts/opensuse_essential.sh | 2 +-
11 files changed, 195 insertions(+), 3 deletions(-)
diff --git a/documentation/tools/Containerfile.almalinux b/documentation/tools/Containerfile.almalinux
new file mode 120000
index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74
--- /dev/null
+++ b/documentation/tools/Containerfile.almalinux
@@ -0,0 +1 @@
+Containerfile.dnf
\ No newline at end of file
diff --git a/documentation/tools/Containerfile.apt b/documentation/tools/Containerfile.apt
new file mode 100644
index 0000000000000000000000000000000000000000..5e30b65eb8f81a7764c00eb1078dabaf59b64517
--- /dev/null
+++ b/documentation/tools/Containerfile.apt
@@ -0,0 +1,26 @@
+ARG ARG_FROM=debian:12 # default value to avoid warning
+FROM $ARG_FROM
+
+ARG DOCS=ubuntu_docs.sh
+ARG DOCS_PDF=ubuntu_docs_pdf.sh
+
+ENV DEBIAN_FRONTEND=noninteractive
+ARG TZ=Europe/Vienna
+
+# relative to the location of the dockerfile
+COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh
+COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
+
+RUN ln -fs "/usr/share/zoneinfo/$TZ" /etc/localtime \
+ && apt-get update \
+ && apt-get install -y sudo \
+ && yes | /temp/host_packages_docs.sh \
+ && yes | /temp/host_packages_docs_pdf.sh \
+ && apt-get --yes autoremove \
+ && apt-get clean \
+ && rm -rf /temp
+
+RUN git config --global --add safe.directory /docs
+
+ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
+CMD ["publish"]
diff --git a/documentation/tools/Containerfile.debian b/documentation/tools/Containerfile.debian
new file mode 120000
index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a
--- /dev/null
+++ b/documentation/tools/Containerfile.debian
@@ -0,0 +1 @@
+Containerfile.apt
\ No newline at end of file
diff --git a/documentation/tools/Containerfile.dnf b/documentation/tools/Containerfile.dnf
new file mode 100644
index 0000000000000000000000000000000000000000..3dae74445585961a6f2d29b8acde09f2456dd886
--- /dev/null
+++ b/documentation/tools/Containerfile.dnf
@@ -0,0 +1,25 @@
+ARG ARG_FROM=fedora:40 # default value to avoid warning
+FROM $ARG_FROM
+
+ARG DOCS=fedora_docs.sh
+ARG DOCS_PDF=fedora_docs_pdf.sh
+ARG PIP3=pip3_docs.sh
+
+# relative to the location of the dockerfile
+COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh
+COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
+COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh
+
+RUN dnf update -y \
+ && dnf install -y sudo \
+ && yes | /temp/host_packages_docs.sh \
+ && yes | /temp/host_packages_docs_pdf.sh \
+ && yes | /temp/pip3_docs.sh \
+ && dnf autoremove -y \
+ && dnf clean all -y \
+ && rm -rf /temp
+
+RUN git config --global --add safe.directory /docs
+
+ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
+CMD ["publish"]
diff --git a/documentation/tools/Containerfile.fedora b/documentation/tools/Containerfile.fedora
new file mode 120000
index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74
--- /dev/null
+++ b/documentation/tools/Containerfile.fedora
@@ -0,0 +1 @@
+Containerfile.dnf
\ No newline at end of file
diff --git a/documentation/tools/Containerfile.ubuntu b/documentation/tools/Containerfile.ubuntu
new file mode 120000
index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a
--- /dev/null
+++ b/documentation/tools/Containerfile.ubuntu
@@ -0,0 +1 @@
+Containerfile.apt
\ No newline at end of file
diff --git a/documentation/tools/Containerfile.zypper b/documentation/tools/Containerfile.zypper
new file mode 100644
index 0000000000000000000000000000000000000000..bcda5261ae98e8df16a3bcef17a7bf204033da99
--- /dev/null
+++ b/documentation/tools/Containerfile.zypper
@@ -0,0 +1,24 @@
+ARG ARG_FROM=opensuse/leap:15.4 # default value to avoid warning
+FROM $ARG_FROM
+
+ARG DOCS=opensuse_docs.sh
+ARG DOCS_PDF=opensuse_docs_pdf.sh
+ARG PIP3=pip3_docs.sh
+
+# relative to the location of the dockerfile
+COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh
+COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
+COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh
+
+RUN zypper update -y \
+ && zypper install -y sudo \
+ && yes | /temp/host_packages_docs.sh \
+ && yes | /temp/host_packages_docs_pdf.sh \
+ && yes | /temp/pip3_docs.sh \
+ && zypper clean --all \
+ && rm -rf /temp
+
+RUN git config --global --add safe.directory /docs
+
+ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
+CMD ["publish"]
diff --git a/documentation/tools/build-docs-container b/documentation/tools/build-docs-container
new file mode 100755
index 0000000000000000000000000000000000000000..f382af75a7e89cfcc8a6080fdbc65fb68ff9ea8d
--- /dev/null
+++ b/documentation/tools/build-docs-container
@@ -0,0 +1,113 @@
+#!/usr/bin/env bash
+#
+# Build a container ready to build the documentation be reading the dependencies
+# listed in shell scripts in documentation/tools/host_packages_scripts, and
+# start a documentation build in this container.
+#
+# Usage:
+#
+# ./documentation/tools/build-docs-container <image> [<make target>]
+#
+# e.g.:
+#
+# ./documentation/tools/build-docs-container ubuntu:24.04 html
+#
+# Will build the docs in an Ubuntu 24.04 container in html.
+#
+# The container engine can be selected by exporting CONTAINERCMD in the
+# environment. The default is docker, but podman can also be used.
+
+set -eu -o pipefail
+
+SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
+CONTAINERCMD=${CONTAINERCMD:-docker}
+DOCS_DIR="$SCRIPT_DIR/../.."
+SH_DIR="$SCRIPT_DIR/host_packages_scripts"
+
+main ()
+{
+ local image="$1"
+ shift
+
+ OCI=$(which "$CONTAINERCMD")
+
+ # docker build doesn't accept 2 colons, so "sanitize" the name
+ local sanitized_dockername
+ sanitized_dockername=$(echo "$image" | tr ':.' '-')
+
+ local version
+ version=$(echo "$image" | awk -F: '{print $NF}')
+
+ case $image in
+ almalinux*)
+ containerfile=Containerfile.almalinux
+ docs=almalinux_docs.sh
+ docs_pdf=almalinux_docs_pdf.sh
+ pip3=pip3_docs.sh
+ ;;
+ debian*)
+ containerfile=Containerfile.debian
+ docs=ubuntu_docs.sh
+ docs_pdf=ubuntu_docs_pdf.sh
+ ;;
+ fedora*)
+ containerfile=Containerfile.fedora
+ docs=fedora_docs.sh
+ docs_pdf=fedora_docs_pdf.sh
+ pip3=pip3_docs.sh
+ ;;
+ opensuse* | leap*)
+ image=opensuse/leap:$version
+ containerfile=Containerfile.zypper
+ docs=opensuse_docs.sh
+ docs_pdf=opensuse_docs_pdf.sh
+ pip3=pip3_docs.sh
+ ;;
+ ubuntu*)
+ containerfile=Containerfile.ubuntu
+ docs=ubuntu_docs.sh
+ docs_pdf=ubuntu_docs_pdf.sh
+ ;;
+ *)
+ echo "$image not supported"
+ exit 1
+ ;;
+ esac
+
+ $OCI build \
+ --tag "yocto-docs-$sanitized_dockername:latest" \
+ --build-arg ARG_FROM="docker.io/$image" \
+ --build-arg DOCS="$docs" \
+ --build-arg DOCS_PDF="$docs_pdf" \
+ --build-arg PIP3="${pip3:-}" \
+ --file "$SCRIPT_DIR/$containerfile" \
+ "$SH_DIR/"
+
+ local -a args_run=(
+ --rm
+ --interactive
+ --tty
+ --volume="$DOCS_DIR:/docs:rw"
+ --workdir=/docs
+ --security-opt label=disable
+ )
+
+ if [ "$OCI" = "docker" ]; then
+ args_run+=(
+ --user="$(id -u)":"$(id -g)"
+ )
+ elif [ "$OCI" = "podman" ]; then
+ # we need net access to fetch bitbake terms
+ args_run+=(
+ --cap-add=NET_RAW
+ --userns=keep-id
+ )
+ fi
+
+ $OCI run \
+ "${args_run[@]}" \
+ "yocto-docs-$sanitized_dockername" \
+ "$@"
+}
+
+main "$@"
diff --git a/documentation/tools/host_packages_scripts/opensuse_docs.sh b/documentation/tools/host_packages_scripts/opensuse_docs.sh
index 7d36eb0f99668168dc4fed44d71181338bdc59a3..896ad52b5042a9679555a30e582fa32076c3a3e7 100644
--- a/documentation/tools/host_packages_scripts/opensuse_docs.sh
+++ b/documentation/tools/host_packages_scripts/opensuse_docs.sh
@@ -1 +1 @@
-sudo zypper install git glibc-i18ndata make python3-pip rsvg-convert which
+sudo zypper --non-interactive install git glibc-i18ndata make python3-pip rsvg-convert which
diff --git a/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh b/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh
index ee9f07886ce1c0eac8e1d1118cda8ddc496f0139..f8a3b800d9790c0434f0c3f577ea1916a7fe1b28 100644
--- a/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh
+++ b/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh
@@ -1 +1 @@
-sudo zypper install 'texlive-collection-lang*' texlive-collection-fontsextra texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexextra texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-latexmk texlive-tex-gyre texlive-xetex
+sudo zypper --non-interactive install 'texlive-collection-lang*' texlive-collection-fontsextra texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexextra texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-latexmk texlive-tex-gyre texlive-xetex
diff --git a/documentation/tools/host_packages_scripts/opensuse_essential.sh b/documentation/tools/host_packages_scripts/opensuse_essential.sh
index a784f4a5dc05f3734177f08ed97663a04d20a978..19e85f4a736532b890fc55492721cf345c2b4083 100644
--- a/documentation/tools/host_packages_scripts/opensuse_essential.sh
+++ b/documentation/tools/host_packages_scripts/opensuse_essential.sh
@@ -1,2 +1,2 @@
-sudo zypper install bzip2 chrpath diffstat gcc gcc-c++ git gzip hostname libacl1 lz4 make makeinfo patch python python-curses python-xml python3 python3-Jinja2 python3-curses python3-pexpect python3-pip rpcgen socat tar wget which xz zstd
+sudo zypper --non-interactive install bzip2 chrpath diffstat gcc gcc-c++ git gzip hostname libacl1 lz4 make makeinfo patch python python-curses python-xml python3 python3-Jinja2 python3-curses python3-pexpect python3-pip rpcgen socat tar wget which xz zstd
sudo pip3 install GitPython
--
2.47.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [docs] [PATCH RFC v2 2/2] tools: add script for building documentation inside containers
2024-12-17 16:08 ` [PATCH RFC v2 2/2] tools: add script for building documentation inside containers Quentin Schulz
@ 2024-12-26 10:41 ` Antonin Godard
2025-01-13 16:43 ` Quentin Schulz
0 siblings, 1 reply; 6+ messages in thread
From: Antonin Godard @ 2024-12-26 10:41 UTC (permalink / raw)
To: Quentin Schulz, Quentin Schulz, docs
Hi Quentin,
On Tue Dec 17, 2024 at 5:08 PM CET, Quentin Schulz wrote:
> From: Quentin Schulz <quentin.schulz@cherry.de>
>
> This adds a script for building a container and building the
> documentation within that new container image.
>
> The openSUSE instructions now require a --non-interactive flag otherwise
> they fail to run. Sadly there doesn't seem to be a way to have this in
> an environment variable à-la DEBIAN_FRONTEND=noninteractive.
Hmm, this means that we end up instructing to pass "--non-interactive" in the
docs? That's not ideal :/ I suggested a workaround below.
I hope there won't be too many cases where we have to do things differently in
the context of a containerfile in the future. Apart from this case I cannot
think of anything else that could happen. Anything comes to mind for you?
> Somehow tzdata package in Ubuntu doesn't respect
> DEBIAN_FRONTEND=noninteractive hence why the timezone needs to be set by
> hand.
>
> Note that only instructions for Debian Bookworm (12), Ubuntu 22.04 and
> 24.04, and Fedora 38, 39 and 40, currently work.
>
> Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
> ---
> documentation/tools/Containerfile.almalinux | 1 +
> documentation/tools/Containerfile.apt | 26 +++++
> documentation/tools/Containerfile.debian | 1 +
> documentation/tools/Containerfile.dnf | 25 +++++
> documentation/tools/Containerfile.fedora | 1 +
> documentation/tools/Containerfile.ubuntu | 1 +
> documentation/tools/Containerfile.zypper | 24 +++++
> documentation/tools/build-docs-container | 113 +++++++++++++++++++++
> .../tools/host_packages_scripts/opensuse_docs.sh | 2 +-
> .../host_packages_scripts/opensuse_docs_pdf.sh | 2 +-
> .../host_packages_scripts/opensuse_essential.sh | 2 +-
> 11 files changed, 195 insertions(+), 3 deletions(-)
>
> diff --git a/documentation/tools/Containerfile.almalinux b/documentation/tools/Containerfile.almalinux
> new file mode 120000
> index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74
> --- /dev/null
> +++ b/documentation/tools/Containerfile.almalinux
> @@ -0,0 +1 @@
> +Containerfile.dnf
> \ No newline at end of file
> diff --git a/documentation/tools/Containerfile.apt b/documentation/tools/Containerfile.apt
> new file mode 100644
> index 0000000000000000000000000000000000000000..5e30b65eb8f81a7764c00eb1078dabaf59b64517
> --- /dev/null
> +++ b/documentation/tools/Containerfile.apt
> @@ -0,0 +1,26 @@
> +ARG ARG_FROM=debian:12 # default value to avoid warning
> +FROM $ARG_FROM
> +
> +ARG DOCS=ubuntu_docs.sh
> +ARG DOCS_PDF=ubuntu_docs_pdf.sh
> +
> +ENV DEBIAN_FRONTEND=noninteractive
> +ARG TZ=Europe/Vienna
> +
> +# relative to the location of the dockerfile
> +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh
> +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
> +
> +RUN ln -fs "/usr/share/zoneinfo/$TZ" /etc/localtime \
> + && apt-get update \
> + && apt-get install -y sudo \
> + && yes | /temp/host_packages_docs.sh \
> + && yes | /temp/host_packages_docs_pdf.sh \
> + && apt-get --yes autoremove \
> + && apt-get clean \
> + && rm -rf /temp
> +
> +RUN git config --global --add safe.directory /docs
> +
> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
> +CMD ["publish"]
> diff --git a/documentation/tools/Containerfile.debian b/documentation/tools/Containerfile.debian
> new file mode 120000
> index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a
> --- /dev/null
> +++ b/documentation/tools/Containerfile.debian
> @@ -0,0 +1 @@
> +Containerfile.apt
> \ No newline at end of file
> diff --git a/documentation/tools/Containerfile.dnf b/documentation/tools/Containerfile.dnf
> new file mode 100644
> index 0000000000000000000000000000000000000000..3dae74445585961a6f2d29b8acde09f2456dd886
> --- /dev/null
> +++ b/documentation/tools/Containerfile.dnf
> @@ -0,0 +1,25 @@
> +ARG ARG_FROM=fedora:40 # default value to avoid warning
> +FROM $ARG_FROM
> +
> +ARG DOCS=fedora_docs.sh
> +ARG DOCS_PDF=fedora_docs_pdf.sh
> +ARG PIP3=pip3_docs.sh
> +
> +# relative to the location of the dockerfile
> +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh
> +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
> +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh
> +
> +RUN dnf update -y \
> + && dnf install -y sudo \
> + && yes | /temp/host_packages_docs.sh \
> + && yes | /temp/host_packages_docs_pdf.sh \
> + && yes | /temp/pip3_docs.sh \
> + && dnf autoremove -y \
> + && dnf clean all -y \
> + && rm -rf /temp
> +
> +RUN git config --global --add safe.directory /docs
> +
> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
> +CMD ["publish"]
> diff --git a/documentation/tools/Containerfile.fedora b/documentation/tools/Containerfile.fedora
> new file mode 120000
> index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74
> --- /dev/null
> +++ b/documentation/tools/Containerfile.fedora
> @@ -0,0 +1 @@
> +Containerfile.dnf
> \ No newline at end of file
> diff --git a/documentation/tools/Containerfile.ubuntu b/documentation/tools/Containerfile.ubuntu
> new file mode 120000
> index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a
> --- /dev/null
> +++ b/documentation/tools/Containerfile.ubuntu
> @@ -0,0 +1 @@
> +Containerfile.apt
> \ No newline at end of file
> diff --git a/documentation/tools/Containerfile.zypper b/documentation/tools/Containerfile.zypper
> new file mode 100644
> index 0000000000000000000000000000000000000000..bcda5261ae98e8df16a3bcef17a7bf204033da99
> --- /dev/null
> +++ b/documentation/tools/Containerfile.zypper
> @@ -0,0 +1,24 @@
> +ARG ARG_FROM=opensuse/leap:15.4 # default value to avoid warning
> +FROM $ARG_FROM
> +
> +ARG DOCS=opensuse_docs.sh
> +ARG DOCS_PDF=opensuse_docs_pdf.sh
> +ARG PIP3=pip3_docs.sh
> +
> +# relative to the location of the dockerfile
> +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh
> +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
> +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh
A bit of a hacky workaround, but maybe here run
sed -i 's/zypper/zypper --non-interactive/ ...' on the scripts? So we don't need
to provide this option in the doc.
> +
> +RUN zypper update -y \
> + && zypper install -y sudo \
> + && yes | /temp/host_packages_docs.sh \
> + && yes | /temp/host_packages_docs_pdf.sh \
> + && yes | /temp/pip3_docs.sh \
> + && zypper clean --all \
> + && rm -rf /temp
> +
> +RUN git config --global --add safe.directory /docs
> +
> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
> +CMD ["publish"]
> diff --git a/documentation/tools/build-docs-container b/documentation/tools/build-docs-container
> new file mode 100755
> index 0000000000000000000000000000000000000000..f382af75a7e89cfcc8a6080fdbc65fb68ff9ea8d
> --- /dev/null
> +++ b/documentation/tools/build-docs-container
> @@ -0,0 +1,113 @@
> +#!/usr/bin/env bash
> +#
> +# Build a container ready to build the documentation be reading the dependencies
> +# listed in shell scripts in documentation/tools/host_packages_scripts, and
> +# start a documentation build in this container.
> +#
> +# Usage:
> +#
> +# ./documentation/tools/build-docs-container <image> [<make target>]
> +#
> +# e.g.:
> +#
> +# ./documentation/tools/build-docs-container ubuntu:24.04 html
> +#
> +# Will build the docs in an Ubuntu 24.04 container in html.
> +#
> +# The container engine can be selected by exporting CONTAINERCMD in the
> +# environment. The default is docker, but podman can also be used.
> +
> +set -eu -o pipefail
> +
> +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
> +CONTAINERCMD=${CONTAINERCMD:-docker}
> +DOCS_DIR="$SCRIPT_DIR/../.."
> +SH_DIR="$SCRIPT_DIR/host_packages_scripts"
> +
> +main ()
> +{
I would be good to have a check here to avoid
$ ./tools/build-docs-container
./tools/build-docs-container: line 29: $1: unbound variable
You could probably copypaste what I did in my version of the script.
> + local image="$1"
> + shift
> +
> + OCI=$(which "$CONTAINERCMD")
> +
> + # docker build doesn't accept 2 colons, so "sanitize" the name
> + local sanitized_dockername
> + sanitized_dockername=$(echo "$image" | tr ':.' '-')
> +
> + local version
> + version=$(echo "$image" | awk -F: '{print $NF}')
> +
> + case $image in
> + almalinux*)
> + containerfile=Containerfile.almalinux
> + docs=almalinux_docs.sh
> + docs_pdf=almalinux_docs_pdf.sh
> + pip3=pip3_docs.sh
> + ;;
> + debian*)
> + containerfile=Containerfile.debian
> + docs=ubuntu_docs.sh
> + docs_pdf=ubuntu_docs_pdf.sh
> + ;;
> + fedora*)
> + containerfile=Containerfile.fedora
> + docs=fedora_docs.sh
> + docs_pdf=fedora_docs_pdf.sh
> + pip3=pip3_docs.sh
> + ;;
> + opensuse* | leap*)
> + image=opensuse/leap:$version
> + containerfile=Containerfile.zypper
> + docs=opensuse_docs.sh
> + docs_pdf=opensuse_docs_pdf.sh
> + pip3=pip3_docs.sh
> + ;;
> + ubuntu*)
> + containerfile=Containerfile.ubuntu
> + docs=ubuntu_docs.sh
> + docs_pdf=ubuntu_docs_pdf.sh
> + ;;
> + *)
> + echo "$image not supported"
Can we have a way of listing images here so a user doesn't need to read the
script and guess which one to use?
> + exit 1
> + ;;
> + esac
> +
> + $OCI build \
> + --tag "yocto-docs-$sanitized_dockername:latest" \
> + --build-arg ARG_FROM="docker.io/$image" \
> + --build-arg DOCS="$docs" \
> + --build-arg DOCS_PDF="$docs_pdf" \
> + --build-arg PIP3="${pip3:-}" \
> + --file "$SCRIPT_DIR/$containerfile" \
> + "$SH_DIR/"
> +
> + local -a args_run=(
> + --rm
> + --interactive
> + --tty
> + --volume="$DOCS_DIR:/docs:rw"
> + --workdir=/docs
> + --security-opt label=disable
> + )
> +
> + if [ "$OCI" = "docker" ]; then
> + args_run+=(
> + --user="$(id -u)":"$(id -g)"
> + )
> + elif [ "$OCI" = "podman" ]; then
> + # we need net access to fetch bitbake terms
> + args_run+=(
> + --cap-add=NET_RAW
> + --userns=keep-id
> + )
> + fi
> +
> + $OCI run \
> + "${args_run[@]}" \
> + "yocto-docs-$sanitized_dockername" \
> + "$@"
> +}
> +
> +main "$@"
> diff --git a/documentation/tools/host_packages_scripts/opensuse_docs.sh b/documentation/tools/host_packages_scripts/opensuse_docs.sh
> index 7d36eb0f99668168dc4fed44d71181338bdc59a3..896ad52b5042a9679555a30e582fa32076c3a3e7 100644
> --- a/documentation/tools/host_packages_scripts/opensuse_docs.sh
> +++ b/documentation/tools/host_packages_scripts/opensuse_docs.sh
> @@ -1 +1 @@
> -sudo zypper install git glibc-i18ndata make python3-pip rsvg-convert which
> +sudo zypper --non-interactive install git glibc-i18ndata make python3-pip rsvg-convert which
> diff --git a/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh b/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh
> index ee9f07886ce1c0eac8e1d1118cda8ddc496f0139..f8a3b800d9790c0434f0c3f577ea1916a7fe1b28 100644
> --- a/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh
> +++ b/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh
> @@ -1 +1 @@
> -sudo zypper install 'texlive-collection-lang*' texlive-collection-fontsextra texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexextra texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-latexmk texlive-tex-gyre texlive-xetex
> +sudo zypper --non-interactive install 'texlive-collection-lang*' texlive-collection-fontsextra texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexextra texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-latexmk texlive-tex-gyre texlive-xetex
> diff --git a/documentation/tools/host_packages_scripts/opensuse_essential.sh b/documentation/tools/host_packages_scripts/opensuse_essential.sh
> index a784f4a5dc05f3734177f08ed97663a04d20a978..19e85f4a736532b890fc55492721cf345c2b4083 100644
> --- a/documentation/tools/host_packages_scripts/opensuse_essential.sh
> +++ b/documentation/tools/host_packages_scripts/opensuse_essential.sh
> @@ -1,2 +1,2 @@
> -sudo zypper install bzip2 chrpath diffstat gcc gcc-c++ git gzip hostname libacl1 lz4 make makeinfo patch python python-curses python-xml python3 python3-Jinja2 python3-curses python3-pexpect python3-pip rpcgen socat tar wget which xz zstd
> +sudo zypper --non-interactive install bzip2 chrpath diffstat gcc gcc-c++ git gzip hostname libacl1 lz4 make makeinfo patch python python-curses python-xml python3 python3-Jinja2 python3-curses python3-pexpect python3-pip rpcgen socat tar wget which xz zstd
> sudo pip3 install GitPython
Thanks,
Antonin
--
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [docs] [PATCH RFC v2 2/2] tools: add script for building documentation inside containers
2024-12-26 10:41 ` [docs] " Antonin Godard
@ 2025-01-13 16:43 ` Quentin Schulz
2025-01-24 11:27 ` Antonin Godard
0 siblings, 1 reply; 6+ messages in thread
From: Quentin Schulz @ 2025-01-13 16:43 UTC (permalink / raw)
To: Antonin Godard, Quentin Schulz, docs
Hi Antonin,
On 12/26/24 11:41 AM, Antonin Godard wrote:
> Hi Quentin,
>
> On Tue Dec 17, 2024 at 5:08 PM CET, Quentin Schulz wrote:
>> From: Quentin Schulz <quentin.schulz@cherry.de>
>>
>> This adds a script for building a container and building the
>> documentation within that new container image.
>>
>> The openSUSE instructions now require a --non-interactive flag otherwise
>> they fail to run. Sadly there doesn't seem to be a way to have this in
>> an environment variable à-la DEBIAN_FRONTEND=noninteractive.
>
> Hmm, this means that we end up instructing to pass "--non-interactive" in the
> docs? That's not ideal :/ I suggested a workaround below.
>
> I hope there won't be too many cases where we have to do things differently in
> the context of a containerfile in the future. Apart from this case I cannot
> think of anything else that could happen. Anything comes to mind for you?
>
There are always interesting corner cases in unattended CI :)
>> Somehow tzdata package in Ubuntu doesn't respect
>> DEBIAN_FRONTEND=noninteractive hence why the timezone needs to be set by
>> hand.
>>
>> Note that only instructions for Debian Bookworm (12), Ubuntu 22.04 and
>> 24.04, and Fedora 38, 39 and 40, currently work.
>>
>> Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
>> ---
>> documentation/tools/Containerfile.almalinux | 1 +
>> documentation/tools/Containerfile.apt | 26 +++++
>> documentation/tools/Containerfile.debian | 1 +
>> documentation/tools/Containerfile.dnf | 25 +++++
>> documentation/tools/Containerfile.fedora | 1 +
>> documentation/tools/Containerfile.ubuntu | 1 +
>> documentation/tools/Containerfile.zypper | 24 +++++
>> documentation/tools/build-docs-container | 113 +++++++++++++++++++++
>> .../tools/host_packages_scripts/opensuse_docs.sh | 2 +-
>> .../host_packages_scripts/opensuse_docs_pdf.sh | 2 +-
>> .../host_packages_scripts/opensuse_essential.sh | 2 +-
>> 11 files changed, 195 insertions(+), 3 deletions(-)
>>
>> diff --git a/documentation/tools/Containerfile.almalinux b/documentation/tools/Containerfile.almalinux
>> new file mode 120000
>> index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74
>> --- /dev/null
>> +++ b/documentation/tools/Containerfile.almalinux
>> @@ -0,0 +1 @@
>> +Containerfile.dnf
>> \ No newline at end of file
>> diff --git a/documentation/tools/Containerfile.apt b/documentation/tools/Containerfile.apt
>> new file mode 100644
>> index 0000000000000000000000000000000000000000..5e30b65eb8f81a7764c00eb1078dabaf59b64517
>> --- /dev/null
>> +++ b/documentation/tools/Containerfile.apt
>> @@ -0,0 +1,26 @@
>> +ARG ARG_FROM=debian:12 # default value to avoid warning
>> +FROM $ARG_FROM
>> +
>> +ARG DOCS=ubuntu_docs.sh
>> +ARG DOCS_PDF=ubuntu_docs_pdf.sh
>> +
>> +ENV DEBIAN_FRONTEND=noninteractive
>> +ARG TZ=Europe/Vienna
>> +
>> +# relative to the location of the dockerfile
>> +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh
>> +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
>> +
>> +RUN ln -fs "/usr/share/zoneinfo/$TZ" /etc/localtime \
>> + && apt-get update \
>> + && apt-get install -y sudo \
>> + && yes | /temp/host_packages_docs.sh \
>> + && yes | /temp/host_packages_docs_pdf.sh \
>> + && apt-get --yes autoremove \
>> + && apt-get clean \
>> + && rm -rf /temp
>> +
>> +RUN git config --global --add safe.directory /docs
>> +
>> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
>> +CMD ["publish"]
>> diff --git a/documentation/tools/Containerfile.debian b/documentation/tools/Containerfile.debian
>> new file mode 120000
>> index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a
>> --- /dev/null
>> +++ b/documentation/tools/Containerfile.debian
>> @@ -0,0 +1 @@
>> +Containerfile.apt
>> \ No newline at end of file
>> diff --git a/documentation/tools/Containerfile.dnf b/documentation/tools/Containerfile.dnf
>> new file mode 100644
>> index 0000000000000000000000000000000000000000..3dae74445585961a6f2d29b8acde09f2456dd886
>> --- /dev/null
>> +++ b/documentation/tools/Containerfile.dnf
>> @@ -0,0 +1,25 @@
>> +ARG ARG_FROM=fedora:40 # default value to avoid warning
>> +FROM $ARG_FROM
>> +
>> +ARG DOCS=fedora_docs.sh
>> +ARG DOCS_PDF=fedora_docs_pdf.sh
>> +ARG PIP3=pip3_docs.sh
>> +
>> +# relative to the location of the dockerfile
>> +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh
>> +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
>> +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh
>> +
>> +RUN dnf update -y \
>> + && dnf install -y sudo \
>> + && yes | /temp/host_packages_docs.sh \
>> + && yes | /temp/host_packages_docs_pdf.sh \
>> + && yes | /temp/pip3_docs.sh \
>> + && dnf autoremove -y \
>> + && dnf clean all -y \
>> + && rm -rf /temp
>> +
>> +RUN git config --global --add safe.directory /docs
>> +
>> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
>> +CMD ["publish"]
>> diff --git a/documentation/tools/Containerfile.fedora b/documentation/tools/Containerfile.fedora
>> new file mode 120000
>> index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74
>> --- /dev/null
>> +++ b/documentation/tools/Containerfile.fedora
>> @@ -0,0 +1 @@
>> +Containerfile.dnf
>> \ No newline at end of file
>> diff --git a/documentation/tools/Containerfile.ubuntu b/documentation/tools/Containerfile.ubuntu
>> new file mode 120000
>> index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a
>> --- /dev/null
>> +++ b/documentation/tools/Containerfile.ubuntu
>> @@ -0,0 +1 @@
>> +Containerfile.apt
>> \ No newline at end of file
>> diff --git a/documentation/tools/Containerfile.zypper b/documentation/tools/Containerfile.zypper
>> new file mode 100644
>> index 0000000000000000000000000000000000000000..bcda5261ae98e8df16a3bcef17a7bf204033da99
>> --- /dev/null
>> +++ b/documentation/tools/Containerfile.zypper
>> @@ -0,0 +1,24 @@
>> +ARG ARG_FROM=opensuse/leap:15.4 # default value to avoid warning
>> +FROM $ARG_FROM
>> +
>> +ARG DOCS=opensuse_docs.sh
>> +ARG DOCS_PDF=opensuse_docs_pdf.sh
>> +ARG PIP3=pip3_docs.sh
>> +
>> +# relative to the location of the dockerfile
>> +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh
>> +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
>> +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh
>
> A bit of a hacky workaround, but maybe here run
> sed -i 's/zypper/zypper --non-interactive/ ...' on the scripts? So we don't need
> to provide this option in the doc.
>
Another dirty hack is to have a shell zypper wrapper that calls
/usr/bin/zypper --non-interactive.
Up to you.
>> +
>> +RUN zypper update -y \
>> + && zypper install -y sudo \
>> + && yes | /temp/host_packages_docs.sh \
>> + && yes | /temp/host_packages_docs_pdf.sh \
>> + && yes | /temp/pip3_docs.sh \
>> + && zypper clean --all \
>> + && rm -rf /temp
>> +
>> +RUN git config --global --add safe.directory /docs
>> +
>> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
>> +CMD ["publish"]
>> diff --git a/documentation/tools/build-docs-container b/documentation/tools/build-docs-container
>> new file mode 100755
>> index 0000000000000000000000000000000000000000..f382af75a7e89cfcc8a6080fdbc65fb68ff9ea8d
>> --- /dev/null
>> +++ b/documentation/tools/build-docs-container
>> @@ -0,0 +1,113 @@
>> +#!/usr/bin/env bash
>> +#
>> +# Build a container ready to build the documentation be reading the dependencies
>> +# listed in shell scripts in documentation/tools/host_packages_scripts, and
>> +# start a documentation build in this container.
>> +#
>> +# Usage:
>> +#
>> +# ./documentation/tools/build-docs-container <image> [<make target>]
>> +#
>> +# e.g.:
>> +#
>> +# ./documentation/tools/build-docs-container ubuntu:24.04 html
>> +#
>> +# Will build the docs in an Ubuntu 24.04 container in html.
>> +#
>> +# The container engine can be selected by exporting CONTAINERCMD in the
>> +# environment. The default is docker, but podman can also be used.
>> +
>> +set -eu -o pipefail
>> +
>> +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
>> +CONTAINERCMD=${CONTAINERCMD:-docker}
>> +DOCS_DIR="$SCRIPT_DIR/../.."
>> +SH_DIR="$SCRIPT_DIR/host_packages_scripts"
>> +
>> +main ()
>> +{
>
> I would be good to have a check here to avoid
>
> $ ./tools/build-docs-container
> ./tools/build-docs-container: line 29: $1: unbound variable
>
> You could probably copypaste what I did in my version of the script.
>
Will do.
>> + local image="$1"
>> + shift
>> +
>> + OCI=$(which "$CONTAINERCMD")
>> +
>> + # docker build doesn't accept 2 colons, so "sanitize" the name
>> + local sanitized_dockername
>> + sanitized_dockername=$(echo "$image" | tr ':.' '-')
>> +
>> + local version
>> + version=$(echo "$image" | awk -F: '{print $NF}')
>> +
>> + case $image in
>> + almalinux*)
>> + containerfile=Containerfile.almalinux
>> + docs=almalinux_docs.sh
>> + docs_pdf=almalinux_docs_pdf.sh
>> + pip3=pip3_docs.sh
>> + ;;
>> + debian*)
>> + containerfile=Containerfile.debian
>> + docs=ubuntu_docs.sh
>> + docs_pdf=ubuntu_docs_pdf.sh
>> + ;;
>> + fedora*)
>> + containerfile=Containerfile.fedora
>> + docs=fedora_docs.sh
>> + docs_pdf=fedora_docs_pdf.sh
>> + pip3=pip3_docs.sh
>> + ;;
>> + opensuse* | leap*)
>> + image=opensuse/leap:$version
>> + containerfile=Containerfile.zypper
>> + docs=opensuse_docs.sh
>> + docs_pdf=opensuse_docs_pdf.sh
>> + pip3=pip3_docs.sh
>> + ;;
>> + ubuntu*)
>> + containerfile=Containerfile.ubuntu
>> + docs=ubuntu_docs.sh
>> + docs_pdf=ubuntu_docs_pdf.sh
>> + ;;
>> + *)
>> + echo "$image not supported"
>
> Can we have a way of listing images here so a user doesn't need to read the
> script and guess which one to use?
>
The ones in
https://docs.yoctoproject.org/ref-manual/system-requirements.html#supported-linux-distributions?
Though I guess Rocky and CentOS are still missing. Do we want the name
of the distro or the name + the version? Basically we can end up with
two places where this information will be, in the docs and in the container.
Cheers,
Quentin
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [docs] [PATCH RFC v2 2/2] tools: add script for building documentation inside containers
2025-01-13 16:43 ` Quentin Schulz
@ 2025-01-24 11:27 ` Antonin Godard
0 siblings, 0 replies; 6+ messages in thread
From: Antonin Godard @ 2025-01-24 11:27 UTC (permalink / raw)
To: Quentin Schulz, Quentin Schulz, docs
Hi Quentin,
On Mon Jan 13, 2025 at 5:43 PM CET, Quentin Schulz wrote:
[...]
>>> +# relative to the location of the dockerfile
>>> +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh
>>> +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
>>> +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh
>>
>> A bit of a hacky workaround, but maybe here run
>> sed -i 's/zypper/zypper --non-interactive/ ...' on the scripts? So we don't need
>> to provide this option in the doc.
>>
>
> Another dirty hack is to have a shell zypper wrapper that calls
> /usr/bin/zypper --non-interactive.
>
> Up to you.
Let's go for a sed in the Containerfile.
[...]
>>
>> Can we have a way of listing images here so a user doesn't need to read the
>> script and guess which one to use?
>>
>
> The ones in
> https://docs.yoctoproject.org/ref-manual/system-requirements.html#supported-linux-distributions?
> Though I guess Rocky and CentOS are still missing. Do we want the name
> of the distro or the name + the version? Basically we can end up with
> two places where this information will be, in the docs and in the container.
I was thinking of just listing what the script supports, not from the docs. I
did that in my version of the script. Something like `./build-docs-container
list`. In your case, it would probably extract that information from the
Containerfiles.
Antonin
--
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2025-01-24 11:27 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-12-17 16:08 [PATCH RFC v2 0/2] add script for containers building the documentation Quentin Schulz
2024-12-17 16:08 ` [PATCH RFC v2 1/2] docs: use literalinclude for system requirements Quentin Schulz
2024-12-17 16:08 ` [PATCH RFC v2 2/2] tools: add script for building documentation inside containers Quentin Schulz
2024-12-26 10:41 ` [docs] " Antonin Godard
2025-01-13 16:43 ` Quentin Schulz
2025-01-24 11:27 ` Antonin Godard
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox