Re: [docs] [PATCH RFC v2 2/2] tools: add script for building documentation inside containers

["Antonin Godard" <antonin.godard@bootlin.com>]

Hi Quentin,

On Tue Dec 17, 2024 at 5:08 PM CET, Quentin Schulz wrote:
> From: Quentin Schulz <quentin.schulz@cherry.de>
>
> This adds a script for building a container and building the
> documentation within that new container image.
>
> The openSUSE instructions now require a --non-interactive flag otherwise
> they fail to run. Sadly there doesn't seem to be a way to have this in
> an environment variable à-la DEBIAN_FRONTEND=noninteractive.

Hmm, this means that we end up instructing to pass "--non-interactive" in the
docs? That's not ideal :/ I suggested a workaround below.

I hope there won't be too many cases where we have to do things differently in
the context of a containerfile in the future. Apart from this case I cannot
think of anything else that could happen. Anything comes to mind for you?

> Somehow tzdata package in Ubuntu doesn't respect
> DEBIAN_FRONTEND=noninteractive hence why the timezone needs to be set by
> hand.
>
> Note that only instructions for Debian Bookworm (12), Ubuntu 22.04 and
> 24.04, and Fedora 38, 39 and 40, currently work.
>
> Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
> ---
>  documentation/tools/Containerfile.almalinux        |   1 +
>  documentation/tools/Containerfile.apt              |  26 +++++
>  documentation/tools/Containerfile.debian           |   1 +
>  documentation/tools/Containerfile.dnf              |  25 +++++
>  documentation/tools/Containerfile.fedora           |   1 +
>  documentation/tools/Containerfile.ubuntu           |   1 +
>  documentation/tools/Containerfile.zypper           |  24 +++++
>  documentation/tools/build-docs-container           | 113 +++++++++++++++++++++
>  .../tools/host_packages_scripts/opensuse_docs.sh   |   2 +-
>  .../host_packages_scripts/opensuse_docs_pdf.sh     |   2 +-
>  .../host_packages_scripts/opensuse_essential.sh    |   2 +-
>  11 files changed, 195 insertions(+), 3 deletions(-)
>
> diff --git a/documentation/tools/Containerfile.almalinux b/documentation/tools/Containerfile.almalinux
> new file mode 120000
> index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74
> --- /dev/null
> +++ b/documentation/tools/Containerfile.almalinux
> @@ -0,0 +1 @@
> +Containerfile.dnf
> \ No newline at end of file
> diff --git a/documentation/tools/Containerfile.apt b/documentation/tools/Containerfile.apt
> new file mode 100644
> index 0000000000000000000000000000000000000000..5e30b65eb8f81a7764c00eb1078dabaf59b64517
> --- /dev/null
> +++ b/documentation/tools/Containerfile.apt
> @@ -0,0 +1,26 @@
> +ARG ARG_FROM=debian:12 # default value to avoid warning
> +FROM $ARG_FROM
> +
> +ARG DOCS=ubuntu_docs.sh
> +ARG DOCS_PDF=ubuntu_docs_pdf.sh
> +
> +ENV DEBIAN_FRONTEND=noninteractive
> +ARG TZ=Europe/Vienna
> +
> +# relative to the location of the dockerfile
> +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh
> +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
> +
> +RUN ln -fs "/usr/share/zoneinfo/$TZ" /etc/localtime \
> + && apt-get update \
> + && apt-get install -y sudo \
> + && yes | /temp/host_packages_docs.sh \
> + && yes | /temp/host_packages_docs_pdf.sh \
> + && apt-get --yes autoremove \
> + && apt-get clean \
> + && rm -rf /temp
> +
> +RUN git config --global --add safe.directory /docs
> +
> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
> +CMD ["publish"]
> diff --git a/documentation/tools/Containerfile.debian b/documentation/tools/Containerfile.debian
> new file mode 120000
> index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a
> --- /dev/null
> +++ b/documentation/tools/Containerfile.debian
> @@ -0,0 +1 @@
> +Containerfile.apt
> \ No newline at end of file
> diff --git a/documentation/tools/Containerfile.dnf b/documentation/tools/Containerfile.dnf
> new file mode 100644
> index 0000000000000000000000000000000000000000..3dae74445585961a6f2d29b8acde09f2456dd886
> --- /dev/null
> +++ b/documentation/tools/Containerfile.dnf
> @@ -0,0 +1,25 @@
> +ARG ARG_FROM=fedora:40 # default value to avoid warning
> +FROM $ARG_FROM
> +
> +ARG DOCS=fedora_docs.sh
> +ARG DOCS_PDF=fedora_docs_pdf.sh
> +ARG PIP3=pip3_docs.sh
> +
> +# relative to the location of the dockerfile
> +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh
> +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
> +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh
> +
> +RUN dnf update -y \
> + && dnf install -y sudo \
> + && yes | /temp/host_packages_docs.sh \
> + && yes | /temp/host_packages_docs_pdf.sh \
> + && yes | /temp/pip3_docs.sh \
> + && dnf autoremove -y \
> + && dnf clean all -y \
> + && rm -rf /temp
> +
> +RUN git config --global --add safe.directory /docs
> +
> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
> +CMD ["publish"]
> diff --git a/documentation/tools/Containerfile.fedora b/documentation/tools/Containerfile.fedora
> new file mode 120000
> index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74
> --- /dev/null
> +++ b/documentation/tools/Containerfile.fedora
> @@ -0,0 +1 @@
> +Containerfile.dnf
> \ No newline at end of file
> diff --git a/documentation/tools/Containerfile.ubuntu b/documentation/tools/Containerfile.ubuntu
> new file mode 120000
> index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a
> --- /dev/null
> +++ b/documentation/tools/Containerfile.ubuntu
> @@ -0,0 +1 @@
> +Containerfile.apt
> \ No newline at end of file
> diff --git a/documentation/tools/Containerfile.zypper b/documentation/tools/Containerfile.zypper
> new file mode 100644
> index 0000000000000000000000000000000000000000..bcda5261ae98e8df16a3bcef17a7bf204033da99
> --- /dev/null
> +++ b/documentation/tools/Containerfile.zypper
> @@ -0,0 +1,24 @@
> +ARG ARG_FROM=opensuse/leap:15.4 # default value to avoid warning
> +FROM $ARG_FROM
> +
> +ARG DOCS=opensuse_docs.sh
> +ARG DOCS_PDF=opensuse_docs_pdf.sh
> +ARG PIP3=pip3_docs.sh
> +
> +# relative to the location of the dockerfile
> +COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh
> +COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
> +COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh

A bit of a hacky workaround, but maybe here run
sed -i 's/zypper/zypper --non-interactive/ ...' on the scripts? So we don't need
to provide this option in the doc.

> +
> +RUN zypper update -y \
> + && zypper install -y sudo \
> + && yes | /temp/host_packages_docs.sh \
> + && yes | /temp/host_packages_docs_pdf.sh \
> + && yes | /temp/pip3_docs.sh \
> + && zypper clean --all \
> + && rm -rf /temp
> +
> +RUN git config --global --add safe.directory /docs
> +
> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
> +CMD ["publish"]
> diff --git a/documentation/tools/build-docs-container b/documentation/tools/build-docs-container
> new file mode 100755
> index 0000000000000000000000000000000000000000..f382af75a7e89cfcc8a6080fdbc65fb68ff9ea8d
> --- /dev/null
> +++ b/documentation/tools/build-docs-container
> @@ -0,0 +1,113 @@
> +#!/usr/bin/env bash
> +#
> +# Build a container ready to build the documentation be reading the dependencies
> +# listed in shell scripts in documentation/tools/host_packages_scripts, and
> +# start a documentation build in this container.
> +#
> +# Usage:
> +#
> +#   ./documentation/tools/build-docs-container <image> [<make target>]
> +#
> +# e.g.:
> +#
> +#   ./documentation/tools/build-docs-container ubuntu:24.04 html
> +#
> +# Will build the docs in an Ubuntu 24.04 container in html.
> +#
> +# The container engine can be selected by exporting CONTAINERCMD in the
> +# environment. The default is docker, but podman can also be used.
> +
> +set -eu -o pipefail
> +
> +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
> +CONTAINERCMD=${CONTAINERCMD:-docker}
> +DOCS_DIR="$SCRIPT_DIR/../.."
> +SH_DIR="$SCRIPT_DIR/host_packages_scripts"
> +
> +main ()
> +{

I would be good to have a check here to avoid

  $ ./tools/build-docs-container
  ./tools/build-docs-container: line 29: $1: unbound variable

You could probably copypaste what I did in my version of the script.

> +  local image="$1"
> +  shift
> +
> +  OCI=$(which "$CONTAINERCMD")
> +
> +  # docker build doesn't accept 2 colons, so "sanitize" the name
> +  local sanitized_dockername
> +  sanitized_dockername=$(echo "$image" | tr ':.' '-')
> +
> +  local version
> +  version=$(echo "$image" | awk -F: '{print $NF}')
> +
> +  case $image in
> +    almalinux*)
> +      containerfile=Containerfile.almalinux
> +      docs=almalinux_docs.sh
> +      docs_pdf=almalinux_docs_pdf.sh
> +      pip3=pip3_docs.sh
> +      ;;
> +    debian*)
> +      containerfile=Containerfile.debian
> +      docs=ubuntu_docs.sh
> +      docs_pdf=ubuntu_docs_pdf.sh
> +      ;;
> +    fedora*)
> +      containerfile=Containerfile.fedora
> +      docs=fedora_docs.sh
> +      docs_pdf=fedora_docs_pdf.sh
> +      pip3=pip3_docs.sh
> +      ;;
> +    opensuse* | leap*)
> +      image=opensuse/leap:$version
> +      containerfile=Containerfile.zypper
> +      docs=opensuse_docs.sh
> +      docs_pdf=opensuse_docs_pdf.sh
> +      pip3=pip3_docs.sh
> +      ;;
> +    ubuntu*)
> +      containerfile=Containerfile.ubuntu
> +      docs=ubuntu_docs.sh
> +      docs_pdf=ubuntu_docs_pdf.sh
> +      ;;
> +    *)
> +      echo "$image not supported"

Can we have a way of listing images here so a user doesn't need to read the
script and guess which one to use?

> +      exit 1
> +      ;;
> +  esac
> +
> +  $OCI build \
> +    --tag "yocto-docs-$sanitized_dockername:latest" \
> +    --build-arg ARG_FROM="docker.io/$image" \
> +    --build-arg DOCS="$docs" \
> +    --build-arg DOCS_PDF="$docs_pdf" \
> +    --build-arg PIP3="${pip3:-}" \
> +    --file "$SCRIPT_DIR/$containerfile" \
> +    "$SH_DIR/"
> +
> +  local -a args_run=(
> +    --rm
> +    --interactive
> +    --tty
> +    --volume="$DOCS_DIR:/docs:rw"
> +    --workdir=/docs
> +    --security-opt label=disable
> +  )
> +
> +  if [ "$OCI" = "docker" ]; then
> +    args_run+=(
> +      --user="$(id -u)":"$(id -g)"
> +    )
> +  elif [ "$OCI" = "podman" ]; then
> +    # we need net access to fetch bitbake terms
> +    args_run+=(
> +      --cap-add=NET_RAW
> +      --userns=keep-id
> +    )
> +  fi
> +
> +  $OCI run \
> +    "${args_run[@]}" \
> +    "yocto-docs-$sanitized_dockername" \
> +    "$@"
> +}
> +
> +main "$@"
> diff --git a/documentation/tools/host_packages_scripts/opensuse_docs.sh b/documentation/tools/host_packages_scripts/opensuse_docs.sh
> index 7d36eb0f99668168dc4fed44d71181338bdc59a3..896ad52b5042a9679555a30e582fa32076c3a3e7 100644
> --- a/documentation/tools/host_packages_scripts/opensuse_docs.sh
> +++ b/documentation/tools/host_packages_scripts/opensuse_docs.sh
> @@ -1 +1 @@
> -sudo zypper install git glibc-i18ndata make python3-pip rsvg-convert which
> +sudo zypper --non-interactive install git glibc-i18ndata make python3-pip rsvg-convert which
> diff --git a/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh b/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh
> index ee9f07886ce1c0eac8e1d1118cda8ddc496f0139..f8a3b800d9790c0434f0c3f577ea1916a7fe1b28 100644
> --- a/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh
> +++ b/documentation/tools/host_packages_scripts/opensuse_docs_pdf.sh
> @@ -1 +1 @@
> -sudo zypper install 'texlive-collection-lang*' texlive-collection-fontsextra texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexextra texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-latexmk texlive-tex-gyre texlive-xetex
> +sudo zypper --non-interactive install 'texlive-collection-lang*' texlive-collection-fontsextra texlive-collection-fontsrecommended texlive-collection-latex texlive-collection-latexextra texlive-collection-latexrecommended texlive-collection-xetex texlive-fncychap texlive-gnu-freefont texlive-latexmk texlive-tex-gyre texlive-xetex
> diff --git a/documentation/tools/host_packages_scripts/opensuse_essential.sh b/documentation/tools/host_packages_scripts/opensuse_essential.sh
> index a784f4a5dc05f3734177f08ed97663a04d20a978..19e85f4a736532b890fc55492721cf345c2b4083 100644
> --- a/documentation/tools/host_packages_scripts/opensuse_essential.sh
> +++ b/documentation/tools/host_packages_scripts/opensuse_essential.sh
> @@ -1,2 +1,2 @@
> -sudo zypper install bzip2 chrpath diffstat gcc gcc-c++ git gzip hostname libacl1 lz4 make makeinfo patch python python-curses python-xml python3 python3-Jinja2 python3-curses python3-pexpect python3-pip rpcgen socat tar wget which xz zstd
> +sudo zypper --non-interactive install bzip2 chrpath diffstat gcc gcc-c++ git gzip hostname libacl1 lz4 make makeinfo patch python python-curses python-xml python3 python3-Jinja2 python3-curses python3-pexpect python3-pip rpcgen socat tar wget which xz zstd
>  sudo pip3 install GitPython

Thanks,
Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com