[PATCH RFC 0/2] add script for containers building the documentation

[PATCH RFC 0/2] add script for containers building the documentation

[Quentin Schulz]

Based on initial work done by Antonin Godard[1].

This does two things:
- move all the installation steps to fulfil the system requirements for
  each distro into separate files,
- add a script for using those separate files for building a container
  which allows to build the documentation within,

Note that Ubuntu (at least 24.04) and openSUSE instructions do not work
right now. Ubuntu has an issue with locales which are already reported
and fixes sent to the ML[2] (I have not tested the possible fix).
openSUSE seems to be running the sphinx build from a 3.6 Python
interpreter which has an incompatible API with the Python calls used in
Sphinx for our documentation.

[1] https://lore.kernel.org/yocto-docs/20241205-docs-build-dockerfiles-v2-1-047cb3245adf@bootlin.com/
[2] https://lore.kernel.org/yocto-docs/20241202110128.27711-1-guenael.muller@smile.fr/

Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
---
Quentin Schulz (2):
      docs: use literalinclude for system requirements
      tools: add script for building documentation inside containers

 documentation/poky.yaml.in                         |  26 ------
 documentation/ref-manual/system-requirements.rst   |  45 ++++-----
 documentation/tools/Containerfile.almalinux        |   1 +
 documentation/tools/Containerfile.apt              |  21 +++++
 documentation/tools/Containerfile.debian           |   1 +
 documentation/tools/Containerfile.dnf              |  19 ++++
 documentation/tools/Containerfile.fedora           |   1 +
 documentation/tools/Containerfile.ubuntu           |   1 +
 documentation/tools/Containerfile.zypper           |  18 ++++
 .../tools/almalinux_host_packages_docs.sh          |   2 +
 .../tools/almalinux_host_packages_essential.sh     |   5 +
 documentation/tools/build-docs-container           | 101 +++++++++++++++++++++
 documentation/tools/fedora_host_packages_docs.sh   |   2 +
 .../tools/fedora_host_packages_essential.sh        |   1 +
 documentation/tools/opensuse_host_packages_docs.sh |   2 +
 .../tools/opensuse_host_packages_essential.sh      |   2 +
 documentation/tools/ubuntu_host_packages_docs.sh   |   2 +
 .../tools/ubuntu_host_packages_essential.sh        |   2 +
 18 files changed, 205 insertions(+), 47 deletions(-)
---
base-commit: 5835cb574881d57785f099c768467177d077e867
change-id: 20241210-instructions-shell-container-c5a84c043a35

Best regards,
-- 
Quentin Schulz <quentin.schulz@cherry.de>

[PATCH RFC 1/2] docs: use literalinclude for system requirements

[Quentin Schulz]

From: Quentin Schulz <quentin.schulz@cherry.de>

The YAML variables for the host dependencies are updated by hand and
actually only used inside code blocks.

It's also slowly getting more difficult to make sense of the content of
those variables as they are both the list of packages to install with
the distro's package manager and sometimes additional commands, which
all need to have the indentation of the block where the variable is used
matched. This is impossible so we just guess it's going to be at a
specific indentation.

Instead, let's migrate all instructions into separate shell scripts that
are then literalinclude'd into the Sphinx documentation.

This allows a few things:
- ability to run shellcheck on the scripts if we ever want to
- manually calling the appropriate script from a supported distro to
  build stuff (distro or bitbake/yocto stuff)
- use this script to create containers to do CI of documentation on
  different distros, to make sure our instructions are all up to date,

Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
---
 documentation/poky.yaml.in                         | 26 -------------
 documentation/ref-manual/system-requirements.rst   | 45 ++++++++++++----------
 .../tools/almalinux_host_packages_docs.sh          |  2 +
 .../tools/almalinux_host_packages_essential.sh     |  5 +++
 documentation/tools/fedora_host_packages_docs.sh   |  2 +
 .../tools/fedora_host_packages_essential.sh        |  1 +
 documentation/tools/opensuse_host_packages_docs.sh |  2 +
 .../tools/opensuse_host_packages_essential.sh      |  2 +
 documentation/tools/ubuntu_host_packages_docs.sh   |  2 +
 .../tools/ubuntu_host_packages_essential.sh        |  2 +
 10 files changed, 42 insertions(+), 47 deletions(-)

diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in
index c770318f2be76cca3ea23a25d819ace46d157b4c..4f4684be4bf7f95ff0263a01136f8ee5a30f1937 100644
--- a/documentation/poky.yaml.in
+++ b/documentation/poky.yaml.in
@@ -10,32 +10,6 @@ BITBAKE_SERIES : ""
 YOCTO_DL_URL : "https://downloads.yoctoproject.org"
 YOCTO_AB_URL : "https://autobuilder.yoctoproject.org"
 YOCTO_RELEASE_DL_URL : "&YOCTO_DL_URL;/releases/yocto/yocto-&DISTRO;"
-UBUNTU_HOST_PACKAGES_ESSENTIAL : "gawk wget git diffstat unzip texinfo gcc \
-     build-essential chrpath socat cpio python3 python3-pip python3-pexpect \
-     xz-utils debianutils iputils-ping python3-git python3-jinja2 \
-     python3-subunit zstd liblz4-tool file locales libacl1
-     \n\   $ sudo locale-gen en_US.UTF-8"
-FEDORA_HOST_PACKAGES_ESSENTIAL : "gawk make wget tar bzip2 gzip python3 unzip perl patch \
-     diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath \
-     ccache perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue perl-bignum socat \
-     python3-pexpect findutils which file cpio python python3-pip xz python3-GitPython \
-     python3-jinja2 rpcgen perl-FindBin perl-File-Compare \
-     perl-File-Copy perl-locale zstd lz4 hostname glibc-langpack-en libacl"
-OPENSUSE_HOST_PACKAGES_ESSENTIAL : "python gcc gcc-c++ git chrpath make wget python-xml \
-     diffstat makeinfo python-curses patch socat python3 python3-curses tar python3-pip \
-     python3-pexpect xz which python3-Jinja2 rpcgen \
-     zstd lz4 bzip2 gzip hostname libacl1
-     \n\   $ sudo pip3 install GitPython"
-ALMALINUX_HOST_PACKAGES_ESSENTIAL : "-y epel-release
-     \n\   $ sudo yum install dnf-plugins-core
-     \n\   $ sudo dnf config-manager --set-enabled crb
-     \n\   $ sudo dnf makecache
-     \n\   $ sudo dnf install gawk make wget tar bzip2 gzip python3 unzip perl patch \
-     diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath ccache \
-     socat perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue python3-pip \
-     python3-GitPython python3-jinja2 python3-pexpect xz which \
-     rpcgen zstd lz4 cpio glibc-langpack-en libacl"
-PIP3_HOST_PACKAGES_DOC : "$ sudo pip3 install sphinx sphinx_rtd_theme pyyaml"
 MIN_PYTHON_VERSION : "3.8.0"
 MIN_TAR_VERSION : "1.28"
 MIN_GIT_VERSION : "1.8.3.1"
diff --git a/documentation/ref-manual/system-requirements.rst b/documentation/ref-manual/system-requirements.rst
index 0fc92550a51ac62c39859acc63002ac065559e2a..0660d830ec2dc903fa4660a3ac62043661d955b1 100644
--- a/documentation/ref-manual/system-requirements.rst
+++ b/documentation/ref-manual/system-requirements.rst
@@ -148,9 +148,10 @@ Ubuntu and Debian
 -----------------
 
 Here are the packages needed to build an image on a headless system
-with a supported Ubuntu or Debian Linux distribution::
+with a supported Ubuntu or Debian Linux distribution:
 
-   $ sudo apt install &UBUNTU_HOST_PACKAGES_ESSENTIAL;
+   .. literalinclude:: ../tools/ubuntu_host_packages_essential.sh
+      :language: shell
 
 .. note::
 
@@ -162,45 +163,47 @@ with a supported Ubuntu or Debian Linux distribution::
          $ sudo apt build-dep qemu
          $ sudo apt remove oss4-dev
 
-Here are the packages needed to build Project documentation manuals::
+Here are the packages needed to build Project documentation manuals:
 
-   $ sudo apt install git make inkscape texlive-latex-extra
-   $ sudo apt install sphinx python3-saneyaml python3-sphinx-rtd-theme
+   .. literalinclude:: ../tools/ubuntu_host_packages_docs.sh
+      :language: shell
 
 Fedora Packages
 ---------------
 
 Here are the packages needed to build an image on a headless system
-with a supported Fedora Linux distribution::
+with a supported Fedora Linux distribution:
 
-   $ sudo dnf install &FEDORA_HOST_PACKAGES_ESSENTIAL;
+   .. literalinclude:: ../tools/fedora_host_packages_essential.sh
+      :language: shell
 
-Here are the packages needed to build Project documentation manuals::
+Here are the packages needed to build Project documentation manuals:
 
-   $ sudo dnf install git make python3-pip which inkscape texlive-fncychap
-   &PIP3_HOST_PACKAGES_DOC;
+   .. literalinclude:: ../tools/fedora_host_packages_docs.sh
+      :language: shell
 
 openSUSE Packages
 -----------------
 
 Here are the packages needed to build an image on a headless system
-with a supported openSUSE distribution::
+with a supported openSUSE distribution:
 
-   $ sudo zypper install &OPENSUSE_HOST_PACKAGES_ESSENTIAL;
+   .. literalinclude:: ../tools/opensuse_host_packages_essential.sh
+      :language: shell
 
-Here are the packages needed to build Project documentation manuals::
-
-   $ sudo zypper install git make python3-pip which inkscape texlive-fncychap
-   &PIP3_HOST_PACKAGES_DOC;
+Here are the packages needed to build Project documentation manuals:
 
+   .. literalinclude:: ../tools/opensuse_host_packages_docs.sh
+      :language: shell
 
 AlmaLinux Packages
 ------------------
 
 Here are the packages needed to build an image on a headless system
-with a supported AlmaLinux distribution::
+with a supported AlmaLinux distribution:
 
-   $ sudo dnf install &ALMALINUX_HOST_PACKAGES_ESSENTIAL;
+   .. literalinclude:: ../tools/almalinux_host_packages_essential.sh
+      :language: shell
 
 .. note::
 
@@ -215,10 +218,10 @@ with a supported AlmaLinux distribution::
    -  The ``makecache`` command consumes additional Metadata from
       ``epel-release``.
 
-Here are the packages needed to build Project documentation manuals::
+Here are the packages needed to build Project documentation manuals:
 
-   $ sudo dnf install git make python3-pip which inkscape texlive-fncychap
-   &PIP3_HOST_PACKAGES_DOC;
+   .. literalinclude:: ../tools/almalinux_host_packages_docs.sh
+      :language: shell
 
 .. _system-requirements-buildtools:
 
diff --git a/documentation/tools/almalinux_host_packages_docs.sh b/documentation/tools/almalinux_host_packages_docs.sh
new file mode 100644
index 0000000000000000000000000000000000000000..93ed34ce01d803e85c67e01f75dc82ed53d864f0
--- /dev/null
+++ b/documentation/tools/almalinux_host_packages_docs.sh
@@ -0,0 +1,2 @@
+sudo dnf install git make python3-pip which inkscape texlive-fncychap
+sudo pip3 install sphinx sphinx_rtd_theme pyyaml
diff --git a/documentation/tools/almalinux_host_packages_essential.sh b/documentation/tools/almalinux_host_packages_essential.sh
new file mode 100644
index 0000000000000000000000000000000000000000..d2c5168e0bb53bfe14785b2f36220917ac55282e
--- /dev/null
+++ b/documentation/tools/almalinux_host_packages_essential.sh
@@ -0,0 +1,5 @@
+sudo dnf install -y epel-release
+sudo yum install dnf-plugins-core
+sudo dnf config-manager --set-enabled crb
+sudo dnf makecache
+sudo dnf install gawk make wget tar bzip2 gzip python3 unzip perl patch diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath ccache socat perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue python3-pip python3-GitPython python3-jinja2 python3-pexpect xz which rpcgen zstd lz4 cpio glibc-langpack-en libacl
diff --git a/documentation/tools/fedora_host_packages_docs.sh b/documentation/tools/fedora_host_packages_docs.sh
new file mode 100644
index 0000000000000000000000000000000000000000..93ed34ce01d803e85c67e01f75dc82ed53d864f0
--- /dev/null
+++ b/documentation/tools/fedora_host_packages_docs.sh
@@ -0,0 +1,2 @@
+sudo dnf install git make python3-pip which inkscape texlive-fncychap
+sudo pip3 install sphinx sphinx_rtd_theme pyyaml
diff --git a/documentation/tools/fedora_host_packages_essential.sh b/documentation/tools/fedora_host_packages_essential.sh
new file mode 100644
index 0000000000000000000000000000000000000000..448a6e5f1038838132603db0d993fb80eabf75ef
--- /dev/null
+++ b/documentation/tools/fedora_host_packages_essential.sh
@@ -0,0 +1 @@
+sudo dnf install gawk make wget tar bzip2 gzip python3 unzip perl patch diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath ccache perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue perl-bignum socat python3-pexpect findutils which file cpio python python3-pip xz python3-GitPython python3-jinja2 rpcgen perl-FindBin perl-File-Compare perl-File-Copy perl-locale zstd lz4 hostname glibc-langpack-en libacl
diff --git a/documentation/tools/opensuse_host_packages_docs.sh b/documentation/tools/opensuse_host_packages_docs.sh
new file mode 100644
index 0000000000000000000000000000000000000000..90f52d576451ee0a7c481b31b7605e4a870496cd
--- /dev/null
+++ b/documentation/tools/opensuse_host_packages_docs.sh
@@ -0,0 +1,2 @@
+sudo zypper install git make python3-pip which inkscape texlive-fncychap
+sudo pip3 install sphinx sphinx_rtd_theme pyyaml
diff --git a/documentation/tools/opensuse_host_packages_essential.sh b/documentation/tools/opensuse_host_packages_essential.sh
new file mode 100644
index 0000000000000000000000000000000000000000..ef24fcefd99c9ecc335c594965965bb36e43cd05
--- /dev/null
+++ b/documentation/tools/opensuse_host_packages_essential.sh
@@ -0,0 +1,2 @@
+sudo zypper install python gcc gcc-c++ git chrpath make wget python-xml diffstat makeinfo python-curses patch socat python3 python3-curses tar python3-pip python3-pexpect xz which python3-Jinja2 rpcgen zstd lz4 bzip2 gzip hostname libacl1
+sudo pip3 install GitPython
diff --git a/documentation/tools/ubuntu_host_packages_docs.sh b/documentation/tools/ubuntu_host_packages_docs.sh
new file mode 100644
index 0000000000000000000000000000000000000000..1bfb256e34ba931784bcf97a60e211520c70a199
--- /dev/null
+++ b/documentation/tools/ubuntu_host_packages_docs.sh
@@ -0,0 +1,2 @@
+sudo apt install git make inkscape texlive-latex-extra
+sudo apt install sphinx python3-saneyaml python3-sphinx-rtd-theme
diff --git a/documentation/tools/ubuntu_host_packages_essential.sh b/documentation/tools/ubuntu_host_packages_essential.sh
new file mode 100644
index 0000000000000000000000000000000000000000..5cab5b34068e3886fb2fb06ea66120a2ae12e3a3
--- /dev/null
+++ b/documentation/tools/ubuntu_host_packages_essential.sh
@@ -0,0 +1,2 @@
+sudo apt-get install gawk wget git diffstat unzip texinfo gcc build-essential chrpath socat cpio python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping python3-git python3-jinja2 python3-subunit zstd liblz4-tool file locales libacl1
+sudo locale-gen en_US.UTF-8

-- 
2.47.1

Re: [docs] [PATCH RFC 1/2] docs: use literalinclude for system requirements

[Antonin Godard]

Hi Quentin,

On Tue Dec 10, 2024 at 4:26 PM CET, Quentin Schulz wrote:
> From: Quentin Schulz <quentin.schulz@cherry.de>
>
> The YAML variables for the host dependencies are updated by hand and
> actually only used inside code blocks.
>
> It's also slowly getting more difficult to make sense of the content of
> those variables as they are both the list of packages to install with
> the distro's package manager and sometimes additional commands, which
> all need to have the indentation of the block where the variable is used
> matched. This is impossible so we just guess it's going to be at a
> specific indentation.
>
> Instead, let's migrate all instructions into separate shell scripts that
> are then literalinclude'd into the Sphinx documentation.
>
> This allows a few things:
> - ability to run shellcheck on the scripts if we ever want to
> - manually calling the appropriate script from a supported distro to
>   build stuff (distro or bitbake/yocto stuff)
> - use this script to create containers to do CI of documentation on
>   different distros, to make sure our instructions are all up to date,
>
> Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
> ---
>  documentation/poky.yaml.in                         | 26 -------------
>  documentation/ref-manual/system-requirements.rst   | 45 ++++++++++++----------
>  .../tools/almalinux_host_packages_docs.sh          |  2 +

Maybe I would put the scripts in a subdirectory called "host_packages_scripts"
and rename the scripts "almalinux_docs.sh", "almalinux_essential.sh", etc. for
shorter names. Same for Containerfiles in the next patch, maybe storing them in
a subdirectory would make sense.

If putting them at the same level is because of the build context requiring
files to be all at the same place, let's leave it like this.


Otherwise, this needs to be rebased and updated after the dependencies were
updated on master.

Thanks,
Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH RFC 1/2] docs: use literalinclude for system requirements

[Quentin Schulz]

Hi Antonin,

On 12/17/24 11:31 AM, Antonin Godard wrote:
> Hi Quentin,
> 
> On Tue Dec 10, 2024 at 4:26 PM CET, Quentin Schulz wrote:
>> From: Quentin Schulz <quentin.schulz@cherry.de>
>>
>> The YAML variables for the host dependencies are updated by hand and
>> actually only used inside code blocks.
>>
>> It's also slowly getting more difficult to make sense of the content of
>> those variables as they are both the list of packages to install with
>> the distro's package manager and sometimes additional commands, which
>> all need to have the indentation of the block where the variable is used
>> matched. This is impossible so we just guess it's going to be at a
>> specific indentation.
>>
>> Instead, let's migrate all instructions into separate shell scripts that
>> are then literalinclude'd into the Sphinx documentation.
>>
>> This allows a few things:
>> - ability to run shellcheck on the scripts if we ever want to
>> - manually calling the appropriate script from a supported distro to
>>    build stuff (distro or bitbake/yocto stuff)
>> - use this script to create containers to do CI of documentation on
>>    different distros, to make sure our instructions are all up to date,
>>
>> Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
>> ---
>>   documentation/poky.yaml.in                         | 26 -------------
>>   documentation/ref-manual/system-requirements.rst   | 45 ++++++++++++----------
>>   .../tools/almalinux_host_packages_docs.sh          |  2 +
> 
> Maybe I would put the scripts in a subdirectory called "host_packages_scripts"
> and rename the scripts "almalinux_docs.sh", "almalinux_essential.sh", etc. for
> shorter names. Same for Containerfiles in the next patch, maybe storing them in
> a subdirectory would make sense.
> 
> If putting them at the same level is because of the build context requiring
> files to be all at the same place, let's leave it like this.
> 

Will check for a v3 but should be doable I believe.

> 
> Otherwise, this needs to be rebased and updated after the dependencies were
> updated on master.
> 

That's already done in the v2 I sent yesterday :)

Cheers,
Quentin

Re: [docs] [PATCH RFC 1/2] docs: use literalinclude for system requirements

[Antonin Godard]

Hi Quentin,

On Tue Dec 17, 2024 at 1:58 PM CET, Quentin Schulz wrote:
> Hi Antonin,
>
> On 12/17/24 11:31 AM, Antonin Godard wrote:
>> Hi Quentin,
>> 
>> On Tue Dec 10, 2024 at 4:26 PM CET, Quentin Schulz wrote:
>>> From: Quentin Schulz <quentin.schulz@cherry.de>
>>>
>>> The YAML variables for the host dependencies are updated by hand and
>>> actually only used inside code blocks.
>>>
>>> It's also slowly getting more difficult to make sense of the content of
>>> those variables as they are both the list of packages to install with
>>> the distro's package manager and sometimes additional commands, which
>>> all need to have the indentation of the block where the variable is used
>>> matched. This is impossible so we just guess it's going to be at a
>>> specific indentation.
>>>
>>> Instead, let's migrate all instructions into separate shell scripts that
>>> are then literalinclude'd into the Sphinx documentation.
>>>
>>> This allows a few things:
>>> - ability to run shellcheck on the scripts if we ever want to
>>> - manually calling the appropriate script from a supported distro to
>>>    build stuff (distro or bitbake/yocto stuff)
>>> - use this script to create containers to do CI of documentation on
>>>    different distros, to make sure our instructions are all up to date,
>>>
>>> Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
>>> ---
>>>   documentation/poky.yaml.in                         | 26 -------------
>>>   documentation/ref-manual/system-requirements.rst   | 45 ++++++++++++----------
>>>   .../tools/almalinux_host_packages_docs.sh          |  2 +
>> 
>> Maybe I would put the scripts in a subdirectory called "host_packages_scripts"
>> and rename the scripts "almalinux_docs.sh", "almalinux_essential.sh", etc. for
>> shorter names. Same for Containerfiles in the next patch, maybe storing them in
>> a subdirectory would make sense.
>> 
>> If putting them at the same level is because of the build context requiring
>> files to be all at the same place, let's leave it like this.
>> 
>
> Will check for a v3 but should be doable I believe.
>
>> 
>> Otherwise, this needs to be rebased and updated after the dependencies were
>> updated on master.
>> 
>
> That's already done in the v2 I sent yesterday :)

Strange, I did not receive it, and I can't find it on
lore.kernel.org/yocto-docs. Is the issue on my side?

Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH RFC 1/2] docs: use literalinclude for system requirements

[Quentin Schulz]

Hi Antonin,

On 12/17/24 2:10 PM, Antonin Godard wrote:
> Hi Quentin,
> 
> On Tue Dec 17, 2024 at 1:58 PM CET, Quentin Schulz wrote:
>> Hi Antonin,
>>
>> On 12/17/24 11:31 AM, Antonin Godard wrote:
>>> Hi Quentin,
>>>
>>> On Tue Dec 10, 2024 at 4:26 PM CET, Quentin Schulz wrote:
>>>> From: Quentin Schulz <quentin.schulz@cherry.de>
>>>>
>>>> The YAML variables for the host dependencies are updated by hand and
>>>> actually only used inside code blocks.
>>>>
>>>> It's also slowly getting more difficult to make sense of the content of
>>>> those variables as they are both the list of packages to install with
>>>> the distro's package manager and sometimes additional commands, which
>>>> all need to have the indentation of the block where the variable is used
>>>> matched. This is impossible so we just guess it's going to be at a
>>>> specific indentation.
>>>>
>>>> Instead, let's migrate all instructions into separate shell scripts that
>>>> are then literalinclude'd into the Sphinx documentation.
>>>>
>>>> This allows a few things:
>>>> - ability to run shellcheck on the scripts if we ever want to
>>>> - manually calling the appropriate script from a supported distro to
>>>>     build stuff (distro or bitbake/yocto stuff)
>>>> - use this script to create containers to do CI of documentation on
>>>>     different distros, to make sure our instructions are all up to date,
>>>>
>>>> Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
>>>> ---
>>>>    documentation/poky.yaml.in                         | 26 -------------
>>>>    documentation/ref-manual/system-requirements.rst   | 45 ++++++++++++----------
>>>>    .../tools/almalinux_host_packages_docs.sh          |  2 +
>>>
>>> Maybe I would put the scripts in a subdirectory called "host_packages_scripts"
>>> and rename the scripts "almalinux_docs.sh", "almalinux_essential.sh", etc. for
>>> shorter names. Same for Containerfiles in the next patch, maybe storing them in
>>> a subdirectory would make sense.
>>>
>>> If putting them at the same level is because of the build context requiring
>>> files to be all at the same place, let's leave it like this.
>>>
>>
>> Will check for a v3 but should be doable I believe.
>>
>>>
>>> Otherwise, this needs to be rebased and updated after the dependencies were
>>> updated on master.
>>>
>>
>> That's already done in the v2 I sent yesterday :)
> 
> Strange, I did not receive it, and I can't find it on
> lore.kernel.org/yocto-docs. Is the issue on my side?
> 

Typical PEBKAC, forgot to run b4 prep --auto-to-cc on the patch series 
before sending it (it warned me I hadn't run the command yet but I had 
sent the v1 already so I thought it was a spurious warning message) so I 
sent it to a whopping amount of one recipient... myself.

Will send with the requested changes as v2 once ready :) Thanks for the 
heads up.

Cheers,
Quentin

[PATCH RFC 2/2] tools: add script for building documentation inside containers

[Quentin Schulz]

From: Quentin Schulz <quentin.schulz@cherry.de>

This adds a script for building a container and building the
documentation within that new container image.

The openSUSE instructions now require a --non-interactive flag otherwise
they fail to run. Sadly there doesn't seem to be a way to have this in
an environment variable à-la DEBIAN_FRONTEND=noninteractive.

Note that the Ubuntu and openSUSE instructions currently do not work.

Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
---
 documentation/tools/Containerfile.almalinux        |   1 +
 documentation/tools/Containerfile.apt              |  21 +++++
 documentation/tools/Containerfile.debian           |   1 +
 documentation/tools/Containerfile.dnf              |  19 ++++
 documentation/tools/Containerfile.fedora           |   1 +
 documentation/tools/Containerfile.ubuntu           |   1 +
 documentation/tools/Containerfile.zypper           |  18 ++++
 documentation/tools/build-docs-container           | 101 +++++++++++++++++++++
 documentation/tools/opensuse_host_packages_docs.sh |   2 +-
 .../tools/opensuse_host_packages_essential.sh      |   2 +-
 10 files changed, 165 insertions(+), 2 deletions(-)

diff --git a/documentation/tools/Containerfile.almalinux b/documentation/tools/Containerfile.almalinux
new file mode 120000
index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74
--- /dev/null
+++ b/documentation/tools/Containerfile.almalinux
@@ -0,0 +1 @@
+Containerfile.dnf
\ No newline at end of file
diff --git a/documentation/tools/Containerfile.apt b/documentation/tools/Containerfile.apt
new file mode 100644
index 0000000000000000000000000000000000000000..a601699ce4005b3bc5288990dbf2e7fa235ea431
--- /dev/null
+++ b/documentation/tools/Containerfile.apt
@@ -0,0 +1,21 @@
+ARG ARG_FROM=debian:12 # default value to avoid warning
+FROM $ARG_FROM
+
+ARG HOST_DOCS_PACKAGES=ubuntu_host_packages_docs.sh
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+# relative to the location of the dockerfile
+COPY --chmod=777 ${HOST_DOCS_PACKAGES} /temp/host_packages_docs.sh
+
+RUN apt-get update \
+ && apt-get install -y sudo \
+ && yes | /temp/host_packages_docs.sh \
+ && apt-get --yes autoremove \
+ && apt-get clean \
+ && rm -rf /temp
+
+RUN git config --global --add safe.directory /docs
+
+ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
+CMD ["publish"]
diff --git a/documentation/tools/Containerfile.debian b/documentation/tools/Containerfile.debian
new file mode 120000
index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a
--- /dev/null
+++ b/documentation/tools/Containerfile.debian
@@ -0,0 +1 @@
+Containerfile.apt
\ No newline at end of file
diff --git a/documentation/tools/Containerfile.dnf b/documentation/tools/Containerfile.dnf
new file mode 100644
index 0000000000000000000000000000000000000000..29ea670905ac011966d7bf5340c10be0660149a6
--- /dev/null
+++ b/documentation/tools/Containerfile.dnf
@@ -0,0 +1,19 @@
+ARG ARG_FROM=fedora:40 # default value to avoid warning
+FROM $ARG_FROM
+
+ARG HOST_DOCS_PACKAGES=fedora_host_packages_docs.sh
+
+# relative to the location of the dockerfile
+COPY --chmod=777 ${HOST_DOCS_PACKAGES} /temp/host_packages_docs.sh
+
+RUN dnf update -y \
+ && dnf install -y sudo \
+ && yes | /temp/host_packages_docs.sh \
+ && dnf autoremove -y \
+ && dnf clean all -y \
+ && rm -rf /temp
+
+RUN git config --global --add safe.directory /docs
+
+ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
+CMD ["publish"]
diff --git a/documentation/tools/Containerfile.fedora b/documentation/tools/Containerfile.fedora
new file mode 120000
index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74
--- /dev/null
+++ b/documentation/tools/Containerfile.fedora
@@ -0,0 +1 @@
+Containerfile.dnf
\ No newline at end of file
diff --git a/documentation/tools/Containerfile.ubuntu b/documentation/tools/Containerfile.ubuntu
new file mode 120000
index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a
--- /dev/null
+++ b/documentation/tools/Containerfile.ubuntu
@@ -0,0 +1 @@
+Containerfile.apt
\ No newline at end of file
diff --git a/documentation/tools/Containerfile.zypper b/documentation/tools/Containerfile.zypper
new file mode 100644
index 0000000000000000000000000000000000000000..567256d65b580aab45e27a7b7a386e7a8dbe107f
--- /dev/null
+++ b/documentation/tools/Containerfile.zypper
@@ -0,0 +1,18 @@
+ARG ARG_FROM=opensuse/leap:15.4 # default value to avoid warning
+FROM $ARG_FROM
+
+ARG HOST_DOCS_PACKAGES=opensuse_host_packages_docs.sh
+
+# relative to the location of the dockerfile
+COPY --chmod=777 ${HOST_DOCS_PACKAGES} /temp/host_packages_docs.sh
+
+RUN zypper update -y \
+ && zypper install -y sudo \
+ && yes | /temp/host_packages_docs.sh \
+ && zypper clean --all \
+ && rm -rf /temp
+
+RUN git config --global --add safe.directory /docs
+
+ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
+CMD ["publish"]
diff --git a/documentation/tools/build-docs-container b/documentation/tools/build-docs-container
new file mode 100755
index 0000000000000000000000000000000000000000..4a799bd8e795f1ab7d55eb99116fe859aff2692c
--- /dev/null
+++ b/documentation/tools/build-docs-container
@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+#
+# Build a container ready to build the documentation be reading the dependencies
+# listed in poky.yaml.in, and start a documentation build in this container.
+#
+# Usage:
+#
+#   ./documentation/tools/build-docs-container <image> [<make target>]
+#
+# e.g.:
+#
+#   ./documentation/tools/build-docs-container ubuntu:24.04 html
+#
+# Will build the docs in an Ubuntu 24.04 container in html.
+#
+# The container engine can be selected by exporting CONTAINERCMD in the
+# environment. The default is docker, but podman can also be used.
+
+set -eu -o pipefail
+
+SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
+CONTAINERCMD=${CONTAINERCMD:-docker}
+DOCS_DIR="$SCRIPT_DIR/../.."
+
+main ()
+{
+  local image="$1"
+  shift
+
+  OCI=$(which "$CONTAINERCMD")
+
+  # docker build doesn't accept 2 colons, so "sanitize" the name
+  local sanitized_dockername
+  sanitized_dockername=$(echo "$image" | tr ':.' '-')
+
+  local version
+  version=$(echo "$image" | awk -F: '{print $NF}')
+
+  case $image in
+    almalinux*)
+      containerfile=Containerfile.almalinux
+      host_packages_docs=almalinux_host_packages_docs.sh
+      ;;
+    debian*)
+      containerfile=Containerfile.debian
+      host_packages_docs=ubuntu_host_packages_docs.sh
+      ;;
+    fedora*)
+      containerfile=Containerfile.fedora
+      host_packages_docs=fedora_host_packages_docs.sh
+      ;;
+    opensuse* | leap*)
+      image=opensuse/leap:$version
+      containerfile=Containerfile.zypper
+      host_packages_docs=opensuse_host_packages_docs.sh
+      ;;
+    ubuntu*)
+      containerfile=Containerfile.ubuntu
+      host_packages_docs=ubuntu_host_packages_docs.sh
+      ;;
+    *)
+      echo "$image not supported"
+      exit 1
+      ;;
+  esac
+
+  $OCI build \
+    --tag "yocto-docs-$sanitized_dockername:latest" \
+    --build-arg ARG_FROM="docker.io/$image" \
+    --build-arg HOST_DOCS_PACKAGES="$host_packages_docs" \
+    --file "$SCRIPT_DIR/$containerfile" \
+    "$SCRIPT_DIR/"
+
+  local -a args_run=(
+    --rm
+    --interactive
+    --tty
+    --volume="$DOCS_DIR:/docs:rw"
+    --workdir=/docs
+    --security-opt label=disable
+  )
+
+  if [ "$OCI" = "docker" ]; then
+    args_run+=(
+      --user="$(id -u)":"$(id -g)"
+    )
+  elif [ "$OCI" = "podman" ]; then
+    # we need net access to fetch bitbake terms
+    args_run+=(
+      --cap-add=NET_RAW
+      --userns=keep-id
+    )
+  fi
+
+  $OCI run \
+    "${args_run[@]}" \
+    "yocto-docs-$sanitized_dockername" \
+    "$*"
+}
+
+main "$@"
diff --git a/documentation/tools/opensuse_host_packages_docs.sh b/documentation/tools/opensuse_host_packages_docs.sh
index 90f52d576451ee0a7c481b31b7605e4a870496cd..18975bd8d2db93eaadc663fac738e9400fd11ffb 100644
--- a/documentation/tools/opensuse_host_packages_docs.sh
+++ b/documentation/tools/opensuse_host_packages_docs.sh
@@ -1,2 +1,2 @@
-sudo zypper install git make python3-pip which inkscape texlive-fncychap
+sudo zypper --non-interactive install git make python3-pip which inkscape texlive-fncychap
 sudo pip3 install sphinx sphinx_rtd_theme pyyaml
diff --git a/documentation/tools/opensuse_host_packages_essential.sh b/documentation/tools/opensuse_host_packages_essential.sh
index ef24fcefd99c9ecc335c594965965bb36e43cd05..20d934c34b833f547463d2e635ee16db935de3dc 100644
--- a/documentation/tools/opensuse_host_packages_essential.sh
+++ b/documentation/tools/opensuse_host_packages_essential.sh
@@ -1,2 +1,2 @@
-sudo zypper install python gcc gcc-c++ git chrpath make wget python-xml diffstat makeinfo python-curses patch socat python3 python3-curses tar python3-pip python3-pexpect xz which python3-Jinja2 rpcgen zstd lz4 bzip2 gzip hostname libacl1
+sudo zypper --non-interactive install python gcc gcc-c++ git chrpath make wget python-xml diffstat makeinfo python-curses patch socat python3 python3-curses tar python3-pip python3-pexpect xz which python3-Jinja2 rpcgen zstd lz4 bzip2 gzip hostname libacl1
 sudo pip3 install GitPython

-- 
2.47.1

Re: [docs] [PATCH RFC 2/2] tools: add script for building documentation inside containers

[Antonin Godard]

Hi Quentin,

On Tue Dec 10, 2024 at 4:26 PM CET, Quentin Schulz wrote:
> From: Quentin Schulz <quentin.schulz@cherry.de>
>
> This adds a script for building a container and building the
> documentation within that new container image.
>
> The openSUSE instructions now require a --non-interactive flag otherwise
> they fail to run. Sadly there doesn't seem to be a way to have this in
> an environment variable à-la DEBIAN_FRONTEND=noninteractive.
>
> Note that the Ubuntu and openSUSE instructions currently do not work.
>
> Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
> ---
>  documentation/tools/Containerfile.almalinux        |   1 +
>  documentation/tools/Containerfile.apt              |  21 +++++
>  documentation/tools/Containerfile.debian           |   1 +
>  documentation/tools/Containerfile.dnf              |  19 ++++
>  documentation/tools/Containerfile.fedora           |   1 +
>  documentation/tools/Containerfile.ubuntu           |   1 +
>  documentation/tools/Containerfile.zypper           |  18 ++++
>  documentation/tools/build-docs-container           | 101 +++++++++++++++++++++
>  documentation/tools/opensuse_host_packages_docs.sh |   2 +-
>  .../tools/opensuse_host_packages_essential.sh      |   2 +-
>  10 files changed, 165 insertions(+), 2 deletions(-)
>
> diff --git a/documentation/tools/Containerfile.almalinux b/documentation/tools/Containerfile.almalinux
> new file mode 120000
> index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74
> --- /dev/null
> +++ b/documentation/tools/Containerfile.almalinux
> @@ -0,0 +1 @@
> +Containerfile.dnf
> \ No newline at end of file
> diff --git a/documentation/tools/Containerfile.apt b/documentation/tools/Containerfile.apt
> new file mode 100644
> index 0000000000000000000000000000000000000000..a601699ce4005b3bc5288990dbf2e7fa235ea431
> --- /dev/null
> +++ b/documentation/tools/Containerfile.apt
> @@ -0,0 +1,21 @@
> +ARG ARG_FROM=debian:12 # default value to avoid warning
> +FROM $ARG_FROM
> +
> +ARG HOST_DOCS_PACKAGES=ubuntu_host_packages_docs.sh
> +
> +ENV DEBIAN_FRONTEND=noninteractive
> +
> +# relative to the location of the dockerfile
> +COPY --chmod=777 ${HOST_DOCS_PACKAGES} /temp/host_packages_docs.sh
> +
> +RUN apt-get update \
> + && apt-get install -y sudo \
> + && yes | /temp/host_packages_docs.sh \
> + && apt-get --yes autoremove \
> + && apt-get clean \
> + && rm -rf /temp
> +
> +RUN git config --global --add safe.directory /docs
> +
> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
> +CMD ["publish"]

Are we able to override this from the command-line, with the
build-docs-container script? I assume everything passed in "$*" below replaces
CMD?

> diff --git a/documentation/tools/Containerfile.debian b/documentation/tools/Containerfile.debian
> new file mode 120000
> index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a
> --- /dev/null
> +++ b/documentation/tools/Containerfile.debian
> @@ -0,0 +1 @@
> +Containerfile.apt
> \ No newline at end of file
> diff --git a/documentation/tools/Containerfile.dnf b/documentation/tools/Containerfile.dnf
> new file mode 100644
> index 0000000000000000000000000000000000000000..29ea670905ac011966d7bf5340c10be0660149a6
> --- /dev/null
> +++ b/documentation/tools/Containerfile.dnf
> @@ -0,0 +1,19 @@
> +ARG ARG_FROM=fedora:40 # default value to avoid warning
> +FROM $ARG_FROM
> +
> +ARG HOST_DOCS_PACKAGES=fedora_host_packages_docs.sh
> +
> +# relative to the location of the dockerfile
> +COPY --chmod=777 ${HOST_DOCS_PACKAGES} /temp/host_packages_docs.sh
> +
> +RUN dnf update -y \
> + && dnf install -y sudo \
> + && yes | /temp/host_packages_docs.sh \
> + && dnf autoremove -y \
> + && dnf clean all -y \
> + && rm -rf /temp
> +
> +RUN git config --global --add safe.directory /docs
> +
> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
> +CMD ["publish"]
> diff --git a/documentation/tools/Containerfile.fedora b/documentation/tools/Containerfile.fedora
> new file mode 120000
> index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74
> --- /dev/null
> +++ b/documentation/tools/Containerfile.fedora
> @@ -0,0 +1 @@
> +Containerfile.dnf
> \ No newline at end of file
> diff --git a/documentation/tools/Containerfile.ubuntu b/documentation/tools/Containerfile.ubuntu
> new file mode 120000
> index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a
> --- /dev/null
> +++ b/documentation/tools/Containerfile.ubuntu
> @@ -0,0 +1 @@
> +Containerfile.apt
> \ No newline at end of file
> diff --git a/documentation/tools/Containerfile.zypper b/documentation/tools/Containerfile.zypper
> new file mode 100644
> index 0000000000000000000000000000000000000000..567256d65b580aab45e27a7b7a386e7a8dbe107f
> --- /dev/null
> +++ b/documentation/tools/Containerfile.zypper
> @@ -0,0 +1,18 @@
> +ARG ARG_FROM=opensuse/leap:15.4 # default value to avoid warning
> +FROM $ARG_FROM
> +
> +ARG HOST_DOCS_PACKAGES=opensuse_host_packages_docs.sh
> +
> +# relative to the location of the dockerfile
> +COPY --chmod=777 ${HOST_DOCS_PACKAGES} /temp/host_packages_docs.sh
> +
> +RUN zypper update -y \
> + && zypper install -y sudo \
> + && yes | /temp/host_packages_docs.sh \
> + && zypper clean --all \
> + && rm -rf /temp
> +
> +RUN git config --global --add safe.directory /docs
> +
> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
> +CMD ["publish"]
> diff --git a/documentation/tools/build-docs-container b/documentation/tools/build-docs-container
> new file mode 100755
> index 0000000000000000000000000000000000000000..4a799bd8e795f1ab7d55eb99116fe859aff2692c
> --- /dev/null
> +++ b/documentation/tools/build-docs-container
> @@ -0,0 +1,101 @@
> +#!/usr/bin/env bash
> +#
> +# Build a container ready to build the documentation be reading the dependencies
> +# listed in poky.yaml.in, and start a documentation build in this container.

Needs an update, since we don't read poky.yaml.in here.

> +#
> +# Usage:
> +#
> +#   ./documentation/tools/build-docs-container <image> [<make target>]
> +#
> +# e.g.:
> +#
> +#   ./documentation/tools/build-docs-container ubuntu:24.04 html
> +#
> +# Will build the docs in an Ubuntu 24.04 container in html.
> +#
> +# The container engine can be selected by exporting CONTAINERCMD in the
> +# environment. The default is docker, but podman can also be used.
> +
> +set -eu -o pipefail
> +
> +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
> +CONTAINERCMD=${CONTAINERCMD:-docker}
> +DOCS_DIR="$SCRIPT_DIR/../.."
> +
> +main ()
> +{
> +  local image="$1"
> +  shift
> +
> +  OCI=$(which "$CONTAINERCMD")
> +
> +  # docker build doesn't accept 2 colons, so "sanitize" the name
> +  local sanitized_dockername
> +  sanitized_dockername=$(echo "$image" | tr ':.' '-')
> +
> +  local version
> +  version=$(echo "$image" | awk -F: '{print $NF}')
> +
> +  case $image in
> +    almalinux*)
> +      containerfile=Containerfile.almalinux
> +      host_packages_docs=almalinux_host_packages_docs.sh
> +      ;;
> +    debian*)
> +      containerfile=Containerfile.debian
> +      host_packages_docs=ubuntu_host_packages_docs.sh
> +      ;;
> +    fedora*)
> +      containerfile=Containerfile.fedora
> +      host_packages_docs=fedora_host_packages_docs.sh
> +      ;;
> +    opensuse* | leap*)
> +      image=opensuse/leap:$version
> +      containerfile=Containerfile.zypper
> +      host_packages_docs=opensuse_host_packages_docs.sh
> +      ;;
> +    ubuntu*)
> +      containerfile=Containerfile.ubuntu
> +      host_packages_docs=ubuntu_host_packages_docs.sh
> +      ;;
> +    *)
> +      echo "$image not supported"
> +      exit 1
> +      ;;
> +  esac
> +
> +  $OCI build \
> +    --tag "yocto-docs-$sanitized_dockername:latest" \
> +    --build-arg ARG_FROM="docker.io/$image" \
> +    --build-arg HOST_DOCS_PACKAGES="$host_packages_docs" \
> +    --file "$SCRIPT_DIR/$containerfile" \
> +    "$SCRIPT_DIR/"
> +
> +  local -a args_run=(
> +    --rm
> +    --interactive
> +    --tty
> +    --volume="$DOCS_DIR:/docs:rw"
> +    --workdir=/docs
> +    --security-opt label=disable
> +  )
> +
> +  if [ "$OCI" = "docker" ]; then
> +    args_run+=(
> +      --user="$(id -u)":"$(id -g)"
> +    )
> +  elif [ "$OCI" = "podman" ]; then
> +    # we need net access to fetch bitbake terms
> +    args_run+=(
> +      --cap-add=NET_RAW
> +      --userns=keep-id
> +    )
> +  fi
> +
> +  $OCI run \
> +    "${args_run[@]}" \
> +    "yocto-docs-$sanitized_dockername" \
> +    "$*"
> +}
> +
> +main "$@"
> diff --git a/documentation/tools/opensuse_host_packages_docs.sh b/documentation/tools/opensuse_host_packages_docs.sh
> index 90f52d576451ee0a7c481b31b7605e4a870496cd..18975bd8d2db93eaadc663fac738e9400fd11ffb 100644
> --- a/documentation/tools/opensuse_host_packages_docs.sh
> +++ b/documentation/tools/opensuse_host_packages_docs.sh
> @@ -1,2 +1,2 @@
> -sudo zypper install git make python3-pip which inkscape texlive-fncychap
> +sudo zypper --non-interactive install git make python3-pip which inkscape texlive-fncychap
>  sudo pip3 install sphinx sphinx_rtd_theme pyyaml
> diff --git a/documentation/tools/opensuse_host_packages_essential.sh b/documentation/tools/opensuse_host_packages_essential.sh
> index ef24fcefd99c9ecc335c594965965bb36e43cd05..20d934c34b833f547463d2e635ee16db935de3dc 100644
> --- a/documentation/tools/opensuse_host_packages_essential.sh
> +++ b/documentation/tools/opensuse_host_packages_essential.sh
> @@ -1,2 +1,2 @@
> -sudo zypper install python gcc gcc-c++ git chrpath make wget python-xml diffstat makeinfo python-curses patch socat python3 python3-curses tar python3-pip python3-pexpect xz which python3-Jinja2 rpcgen zstd lz4 bzip2 gzip hostname libacl1
> +sudo zypper --non-interactive install python gcc gcc-c++ git chrpath make wget python-xml diffstat makeinfo python-curses patch socat python3 python3-curses tar python3-pip python3-pexpect xz which python3-Jinja2 rpcgen zstd lz4 bzip2 gzip hostname libacl1
>  sudo pip3 install GitPython

Looks good to me otherwise!

With the rebase I'd take these patches :)

Thanks,
Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH RFC 2/2] tools: add script for building documentation inside containers

[Quentin Schulz]

Hi Antonin,

On 12/17/24 11:31 AM, Antonin Godard wrote:
> Hi Quentin,
> 
> On Tue Dec 10, 2024 at 4:26 PM CET, Quentin Schulz wrote:
>> From: Quentin Schulz <quentin.schulz@cherry.de>
>>
>> This adds a script for building a container and building the
>> documentation within that new container image.
>>
>> The openSUSE instructions now require a --non-interactive flag otherwise
>> they fail to run. Sadly there doesn't seem to be a way to have this in
>> an environment variable à-la DEBIAN_FRONTEND=noninteractive.
>>
>> Note that the Ubuntu and openSUSE instructions currently do not work.
>>
>> Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
>> ---
>>   documentation/tools/Containerfile.almalinux        |   1 +
>>   documentation/tools/Containerfile.apt              |  21 +++++
>>   documentation/tools/Containerfile.debian           |   1 +
>>   documentation/tools/Containerfile.dnf              |  19 ++++
>>   documentation/tools/Containerfile.fedora           |   1 +
>>   documentation/tools/Containerfile.ubuntu           |   1 +
>>   documentation/tools/Containerfile.zypper           |  18 ++++
>>   documentation/tools/build-docs-container           | 101 +++++++++++++++++++++
>>   documentation/tools/opensuse_host_packages_docs.sh |   2 +-
>>   .../tools/opensuse_host_packages_essential.sh      |   2 +-
>>   10 files changed, 165 insertions(+), 2 deletions(-)
>>
>> diff --git a/documentation/tools/Containerfile.almalinux b/documentation/tools/Containerfile.almalinux
>> new file mode 120000
>> index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74
>> --- /dev/null
>> +++ b/documentation/tools/Containerfile.almalinux
>> @@ -0,0 +1 @@
>> +Containerfile.dnf
>> \ No newline at end of file
>> diff --git a/documentation/tools/Containerfile.apt b/documentation/tools/Containerfile.apt
>> new file mode 100644
>> index 0000000000000000000000000000000000000000..a601699ce4005b3bc5288990dbf2e7fa235ea431
>> --- /dev/null
>> +++ b/documentation/tools/Containerfile.apt
>> @@ -0,0 +1,21 @@
>> +ARG ARG_FROM=debian:12 # default value to avoid warning
>> +FROM $ARG_FROM
>> +
>> +ARG HOST_DOCS_PACKAGES=ubuntu_host_packages_docs.sh
>> +
>> +ENV DEBIAN_FRONTEND=noninteractive
>> +
>> +# relative to the location of the dockerfile
>> +COPY --chmod=777 ${HOST_DOCS_PACKAGES} /temp/host_packages_docs.sh
>> +
>> +RUN apt-get update \
>> + && apt-get install -y sudo \
>> + && yes | /temp/host_packages_docs.sh \
>> + && apt-get --yes autoremove \
>> + && apt-get clean \
>> + && rm -rf /temp
>> +
>> +RUN git config --global --add safe.directory /docs
>> +
>> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
>> +CMD ["publish"]
> 
> Are we able to override this from the command-line, with the
> build-docs-container script? I assume everything passed in "$*" below replaces
> CMD?
> 

Yes, I'm currently calling the script like that:

./tools/build-docs-container fedora:40 html
./tools/build-docs-container fedora:40 latexpdf

or ./tools/build-docs-container fedora:40 latexpdf html (in v2, broken 
in v1).

If you omit it, it'll call "publish", which is what you decided would be 
the default in **your** v1. We can have nothing instead which would call 
"all" I guess instead? Up to you.

Cheers,
Quentin

Re: [docs] [PATCH RFC 2/2] tools: add script for building documentation inside containers

[Antonin Godard]

Hi Quentin,

On Tue Dec 17, 2024 at 2:50 PM CET, Quentin Schulz wrote:
> Hi Antonin,
>
> On 12/17/24 11:31 AM, Antonin Godard wrote:
>> Hi Quentin,
>> 
>> On Tue Dec 10, 2024 at 4:26 PM CET, Quentin Schulz wrote:
>>> From: Quentin Schulz <quentin.schulz@cherry.de>
>>>
>>> This adds a script for building a container and building the
>>> documentation within that new container image.
>>>
>>> The openSUSE instructions now require a --non-interactive flag otherwise
>>> they fail to run. Sadly there doesn't seem to be a way to have this in
>>> an environment variable à-la DEBIAN_FRONTEND=noninteractive.
>>>
>>> Note that the Ubuntu and openSUSE instructions currently do not work.
>>>
>>> Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
>>> ---
>>>   documentation/tools/Containerfile.almalinux        |   1 +
>>>   documentation/tools/Containerfile.apt              |  21 +++++
>>>   documentation/tools/Containerfile.debian           |   1 +
>>>   documentation/tools/Containerfile.dnf              |  19 ++++
>>>   documentation/tools/Containerfile.fedora           |   1 +
>>>   documentation/tools/Containerfile.ubuntu           |   1 +
>>>   documentation/tools/Containerfile.zypper           |  18 ++++
>>>   documentation/tools/build-docs-container           | 101 +++++++++++++++++++++
>>>   documentation/tools/opensuse_host_packages_docs.sh |   2 +-
>>>   .../tools/opensuse_host_packages_essential.sh      |   2 +-
>>>   10 files changed, 165 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/documentation/tools/Containerfile.almalinux b/documentation/tools/Containerfile.almalinux
>>> new file mode 120000
>>> index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74
>>> --- /dev/null
>>> +++ b/documentation/tools/Containerfile.almalinux
>>> @@ -0,0 +1 @@
>>> +Containerfile.dnf
>>> \ No newline at end of file
>>> diff --git a/documentation/tools/Containerfile.apt b/documentation/tools/Containerfile.apt
>>> new file mode 100644
>>> index 0000000000000000000000000000000000000000..a601699ce4005b3bc5288990dbf2e7fa235ea431
>>> --- /dev/null
>>> +++ b/documentation/tools/Containerfile.apt
>>> @@ -0,0 +1,21 @@
>>> +ARG ARG_FROM=debian:12 # default value to avoid warning
>>> +FROM $ARG_FROM
>>> +
>>> +ARG HOST_DOCS_PACKAGES=ubuntu_host_packages_docs.sh
>>> +
>>> +ENV DEBIAN_FRONTEND=noninteractive
>>> +
>>> +# relative to the location of the dockerfile
>>> +COPY --chmod=777 ${HOST_DOCS_PACKAGES} /temp/host_packages_docs.sh
>>> +
>>> +RUN apt-get update \
>>> + && apt-get install -y sudo \
>>> + && yes | /temp/host_packages_docs.sh \
>>> + && apt-get --yes autoremove \
>>> + && apt-get clean \
>>> + && rm -rf /temp
>>> +
>>> +RUN git config --global --add safe.directory /docs
>>> +
>>> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
>>> +CMD ["publish"]
>> 
>> Are we able to override this from the command-line, with the
>> build-docs-container script? I assume everything passed in "$*" below replaces
>> CMD?
>> 
>
> Yes, I'm currently calling the script like that:
>
> ./tools/build-docs-container fedora:40 html
> ./tools/build-docs-container fedora:40 latexpdf
>
> or ./tools/build-docs-container fedora:40 latexpdf html (in v2, broken 
> in v1).
>
> If you omit it, it'll call "publish", which is what you decided would be 
> the default in **your** v1. We can have nothing instead which would call 
> "all" I guess instead? Up to you.

"publish" is similar to all, but goes a bit further to assemble the documents
and gives a better representation of the final state of the docs IMO (how they're
actually published). I think the main advantage is the "final" directory in
the build dir, which gathers everything in one place.

TLDR: the extra cost of running publish instead of all is low, we might as well
just run it by default :)

Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com