Re: [docs] [PATCH RFC v2 2/2] tools: add script for building documentation inside containers

[Quentin Schulz <quentin.schulz@cherry.de>]

Hi Antonin,

On 12/26/24 11:41 AM, Antonin Godard wrote:
> Hi Quentin,
> 
> On Tue Dec 17, 2024 at 5:08 PM CET, Quentin Schulz wrote:
>> From: Quentin Schulz <quentin.schulz@cherry.de>
>>
>> This adds a script for building a container and building the
>> documentation within that new container image.
>>
>> The openSUSE instructions now require a --non-interactive flag otherwise
>> they fail to run. Sadly there doesn't seem to be a way to have this in
>> an environment variable à-la DEBIAN_FRONTEND=noninteractive.
> 
> Hmm, this means that we end up instructing to pass "--non-interactive" in the
> docs? That's not ideal :/ I suggested a workaround below.
> 
> I hope there won't be too many cases where we have to do things differently in
> the context of a containerfile in the future. Apart from this case I cannot
> think of anything else that could happen. Anything comes to mind for you?
> 

There are always interesting corner cases in unattended CI :)

>> Somehow tzdata package in Ubuntu doesn't respect
>> DEBIAN_FRONTEND=noninteractive hence why the timezone needs to be set by
>> hand.
>>
>> Note that only instructions for Debian Bookworm (12), Ubuntu 22.04 and
>> 24.04, and Fedora 38, 39 and 40, currently work.
>>
>> Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
>> ---
>>   documentation/tools/Containerfile.almalinux        |   1 +
>>   documentation/tools/Containerfile.apt              |  26 +++++
>>   documentation/tools/Containerfile.debian           |   1 +
>>   documentation/tools/Containerfile.dnf              |  25 +++++
>>   documentation/tools/Containerfile.fedora           |   1 +
>>   documentation/tools/Containerfile.ubuntu           |   1 +
>>   documentation/tools/Containerfile.zypper           |  24 +++++
>>   documentation/tools/build-docs-container           | 113 +++++++++++++++++++++
>>   .../tools/host_packages_scripts/opensuse_docs.sh   |   2 +-
>>   .../host_packages_scripts/opensuse_docs_pdf.sh     |   2 +-
>>   .../host_packages_scripts/opensuse_essential.sh    |   2 +-
>>   11 files changed, 195 insertions(+), 3 deletions(-)
>>
>> diff --git a/documentation/tools/Containerfile.almalinux b/documentation/tools/Containerfile.almalinux
>> new file mode 120000
>> index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74
>> --- /dev/null
>> +++ b/documentation/tools/Containerfile.almalinux
>> @@ -0,0 +1 @@
>> +Containerfile.dnf
>> \ No newline at end of file
>> diff --git a/documentation/tools/Containerfile.apt b/documentation/tools/Containerfile.apt
>> new file mode 100644
>> index 0000000000000000000000000000000000000000..5e30b65eb8f81a7764c00eb1078dabaf59b64517
>> --- /dev/null
>> +++ b/documentation/tools/Containerfile.apt
>> @@ -0,0 +1,26 @@
>> +ARG ARG_FROM=debian:12 # default value to avoid warning
>> +FROM $ARG_FROM
>> +
>> +ARG DOCS=ubuntu_docs.sh
>> +ARG DOCS_PDF=ubuntu_docs_pdf.sh
>> +
>> +ENV DEBIAN_FRONTEND=noninteractive
>> +ARG TZ=Europe/Vienna
>> +
>> +# relative to the location of the dockerfile
>> +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh
>> +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
>> +
>> +RUN ln -fs "/usr/share/zoneinfo/$TZ" /etc/localtime \
>> + && apt-get update \
>> + && apt-get install -y sudo \
>> + && yes | /temp/host_packages_docs.sh \
>> + && yes | /temp/host_packages_docs_pdf.sh \
>> + && apt-get --yes autoremove \
>> + && apt-get clean \
>> + && rm -rf /temp
>> +
>> +RUN git config --global --add safe.directory /docs
>> +
>> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
>> +CMD ["publish"]
>> diff --git a/documentation/tools/Containerfile.debian b/documentation/tools/Containerfile.debian
>> new file mode 120000
>> index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a
>> --- /dev/null
>> +++ b/documentation/tools/Containerfile.debian
>> @@ -0,0 +1 @@
>> +Containerfile.apt
>> \ No newline at end of file
>> diff --git a/documentation/tools/Containerfile.dnf b/documentation/tools/Containerfile.dnf
>> new file mode 100644
>> index 0000000000000000000000000000000000000000..3dae74445585961a6f2d29b8acde09f2456dd886
>> --- /dev/null
>> +++ b/documentation/tools/Containerfile.dnf
>> @@ -0,0 +1,25 @@
>> +ARG ARG_FROM=fedora:40 # default value to avoid warning
>> +FROM $ARG_FROM
>> +
>> +ARG DOCS=fedora_docs.sh
>> +ARG DOCS_PDF=fedora_docs_pdf.sh
>> +ARG PIP3=pip3_docs.sh
>> +
>> +# relative to the location of the dockerfile
>> +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh
>> +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
>> +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh
>> +
>> +RUN dnf update -y \
>> + && dnf install -y sudo \
>> + && yes | /temp/host_packages_docs.sh \
>> + && yes | /temp/host_packages_docs_pdf.sh \
>> + && yes | /temp/pip3_docs.sh \
>> + && dnf autoremove -y \
>> + && dnf clean all -y \
>> + && rm -rf /temp
>> +
>> +RUN git config --global --add safe.directory /docs
>> +
>> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
>> +CMD ["publish"]
>> diff --git a/documentation/tools/Containerfile.fedora b/documentation/tools/Containerfile.fedora
>> new file mode 120000
>> index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74
>> --- /dev/null
>> +++ b/documentation/tools/Containerfile.fedora
>> @@ -0,0 +1 @@
>> +Containerfile.dnf
>> \ No newline at end of file
>> diff --git a/documentation/tools/Containerfile.ubuntu b/documentation/tools/Containerfile.ubuntu
>> new file mode 120000
>> index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a
>> --- /dev/null
>> +++ b/documentation/tools/Containerfile.ubuntu
>> @@ -0,0 +1 @@
>> +Containerfile.apt
>> \ No newline at end of file
>> diff --git a/documentation/tools/Containerfile.zypper b/documentation/tools/Containerfile.zypper
>> new file mode 100644
>> index 0000000000000000000000000000000000000000..bcda5261ae98e8df16a3bcef17a7bf204033da99
>> --- /dev/null
>> +++ b/documentation/tools/Containerfile.zypper
>> @@ -0,0 +1,24 @@
>> +ARG ARG_FROM=opensuse/leap:15.4 # default value to avoid warning
>> +FROM $ARG_FROM
>> +
>> +ARG DOCS=opensuse_docs.sh
>> +ARG DOCS_PDF=opensuse_docs_pdf.sh
>> +ARG PIP3=pip3_docs.sh
>> +
>> +# relative to the location of the dockerfile
>> +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh
>> +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
>> +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh
> 
> A bit of a hacky workaround, but maybe here run
> sed -i 's/zypper/zypper --non-interactive/ ...' on the scripts? So we don't need
> to provide this option in the doc.
> 

Another dirty hack is to have a shell zypper wrapper that calls 
/usr/bin/zypper --non-interactive.

Up to you.

>> +
>> +RUN zypper update -y \
>> + && zypper install -y sudo \
>> + && yes | /temp/host_packages_docs.sh \
>> + && yes | /temp/host_packages_docs_pdf.sh \
>> + && yes | /temp/pip3_docs.sh \
>> + && zypper clean --all \
>> + && rm -rf /temp
>> +
>> +RUN git config --global --add safe.directory /docs
>> +
>> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
>> +CMD ["publish"]
>> diff --git a/documentation/tools/build-docs-container b/documentation/tools/build-docs-container
>> new file mode 100755
>> index 0000000000000000000000000000000000000000..f382af75a7e89cfcc8a6080fdbc65fb68ff9ea8d
>> --- /dev/null
>> +++ b/documentation/tools/build-docs-container
>> @@ -0,0 +1,113 @@
>> +#!/usr/bin/env bash
>> +#
>> +# Build a container ready to build the documentation be reading the dependencies
>> +# listed in shell scripts in documentation/tools/host_packages_scripts, and
>> +# start a documentation build in this container.
>> +#
>> +# Usage:
>> +#
>> +#   ./documentation/tools/build-docs-container <image> [<make target>]
>> +#
>> +# e.g.:
>> +#
>> +#   ./documentation/tools/build-docs-container ubuntu:24.04 html
>> +#
>> +# Will build the docs in an Ubuntu 24.04 container in html.
>> +#
>> +# The container engine can be selected by exporting CONTAINERCMD in the
>> +# environment. The default is docker, but podman can also be used.
>> +
>> +set -eu -o pipefail
>> +
>> +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
>> +CONTAINERCMD=${CONTAINERCMD:-docker}
>> +DOCS_DIR="$SCRIPT_DIR/../.."
>> +SH_DIR="$SCRIPT_DIR/host_packages_scripts"
>> +
>> +main ()
>> +{
> 
> I would be good to have a check here to avoid
> 
>    $ ./tools/build-docs-container
>    ./tools/build-docs-container: line 29: $1: unbound variable
> 
> You could probably copypaste what I did in my version of the script.
> 

Will do.

>> +  local image="$1"
>> +  shift
>> +
>> +  OCI=$(which "$CONTAINERCMD")
>> +
>> +  # docker build doesn't accept 2 colons, so "sanitize" the name
>> +  local sanitized_dockername
>> +  sanitized_dockername=$(echo "$image" | tr ':.' '-')
>> +
>> +  local version
>> +  version=$(echo "$image" | awk -F: '{print $NF}')
>> +
>> +  case $image in
>> +    almalinux*)
>> +      containerfile=Containerfile.almalinux
>> +      docs=almalinux_docs.sh
>> +      docs_pdf=almalinux_docs_pdf.sh
>> +      pip3=pip3_docs.sh
>> +      ;;
>> +    debian*)
>> +      containerfile=Containerfile.debian
>> +      docs=ubuntu_docs.sh
>> +      docs_pdf=ubuntu_docs_pdf.sh
>> +      ;;
>> +    fedora*)
>> +      containerfile=Containerfile.fedora
>> +      docs=fedora_docs.sh
>> +      docs_pdf=fedora_docs_pdf.sh
>> +      pip3=pip3_docs.sh
>> +      ;;
>> +    opensuse* | leap*)
>> +      image=opensuse/leap:$version
>> +      containerfile=Containerfile.zypper
>> +      docs=opensuse_docs.sh
>> +      docs_pdf=opensuse_docs_pdf.sh
>> +      pip3=pip3_docs.sh
>> +      ;;
>> +    ubuntu*)
>> +      containerfile=Containerfile.ubuntu
>> +      docs=ubuntu_docs.sh
>> +      docs_pdf=ubuntu_docs_pdf.sh
>> +      ;;
>> +    *)
>> +      echo "$image not supported"
> 
> Can we have a way of listing images here so a user doesn't need to read the
> script and guess which one to use?
> 

The ones in 
https://docs.yoctoproject.org/ref-manual/system-requirements.html#supported-linux-distributions? 
Though I guess Rocky and CentOS are still missing. Do we want the name 
of the distro or the name + the version? Basically we can end up with 
two places where this information will be, in the docs and in the container.

Cheers,
Quentin