Re: [docs] [PATCH RFC 2/2] tools: add script for building documentation inside containers

["Antonin Godard" <antonin.godard@bootlin.com>]

Hi Quentin,

On Tue Dec 10, 2024 at 4:26 PM CET, Quentin Schulz wrote:
> From: Quentin Schulz <quentin.schulz@cherry.de>
>
> This adds a script for building a container and building the
> documentation within that new container image.
>
> The openSUSE instructions now require a --non-interactive flag otherwise
> they fail to run. Sadly there doesn't seem to be a way to have this in
> an environment variable à-la DEBIAN_FRONTEND=noninteractive.
>
> Note that the Ubuntu and openSUSE instructions currently do not work.
>
> Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
> ---
>  documentation/tools/Containerfile.almalinux        |   1 +
>  documentation/tools/Containerfile.apt              |  21 +++++
>  documentation/tools/Containerfile.debian           |   1 +
>  documentation/tools/Containerfile.dnf              |  19 ++++
>  documentation/tools/Containerfile.fedora           |   1 +
>  documentation/tools/Containerfile.ubuntu           |   1 +
>  documentation/tools/Containerfile.zypper           |  18 ++++
>  documentation/tools/build-docs-container           | 101 +++++++++++++++++++++
>  documentation/tools/opensuse_host_packages_docs.sh |   2 +-
>  .../tools/opensuse_host_packages_essential.sh      |   2 +-
>  10 files changed, 165 insertions(+), 2 deletions(-)
>
> diff --git a/documentation/tools/Containerfile.almalinux b/documentation/tools/Containerfile.almalinux
> new file mode 120000
> index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74
> --- /dev/null
> +++ b/documentation/tools/Containerfile.almalinux
> @@ -0,0 +1 @@
> +Containerfile.dnf
> \ No newline at end of file
> diff --git a/documentation/tools/Containerfile.apt b/documentation/tools/Containerfile.apt
> new file mode 100644
> index 0000000000000000000000000000000000000000..a601699ce4005b3bc5288990dbf2e7fa235ea431
> --- /dev/null
> +++ b/documentation/tools/Containerfile.apt
> @@ -0,0 +1,21 @@
> +ARG ARG_FROM=debian:12 # default value to avoid warning
> +FROM $ARG_FROM
> +
> +ARG HOST_DOCS_PACKAGES=ubuntu_host_packages_docs.sh
> +
> +ENV DEBIAN_FRONTEND=noninteractive
> +
> +# relative to the location of the dockerfile
> +COPY --chmod=777 ${HOST_DOCS_PACKAGES} /temp/host_packages_docs.sh
> +
> +RUN apt-get update \
> + && apt-get install -y sudo \
> + && yes | /temp/host_packages_docs.sh \
> + && apt-get --yes autoremove \
> + && apt-get clean \
> + && rm -rf /temp
> +
> +RUN git config --global --add safe.directory /docs
> +
> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
> +CMD ["publish"]

Are we able to override this from the command-line, with the
build-docs-container script? I assume everything passed in "$*" below replaces
CMD?

> diff --git a/documentation/tools/Containerfile.debian b/documentation/tools/Containerfile.debian
> new file mode 120000
> index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a
> --- /dev/null
> +++ b/documentation/tools/Containerfile.debian
> @@ -0,0 +1 @@
> +Containerfile.apt
> \ No newline at end of file
> diff --git a/documentation/tools/Containerfile.dnf b/documentation/tools/Containerfile.dnf
> new file mode 100644
> index 0000000000000000000000000000000000000000..29ea670905ac011966d7bf5340c10be0660149a6
> --- /dev/null
> +++ b/documentation/tools/Containerfile.dnf
> @@ -0,0 +1,19 @@
> +ARG ARG_FROM=fedora:40 # default value to avoid warning
> +FROM $ARG_FROM
> +
> +ARG HOST_DOCS_PACKAGES=fedora_host_packages_docs.sh
> +
> +# relative to the location of the dockerfile
> +COPY --chmod=777 ${HOST_DOCS_PACKAGES} /temp/host_packages_docs.sh
> +
> +RUN dnf update -y \
> + && dnf install -y sudo \
> + && yes | /temp/host_packages_docs.sh \
> + && dnf autoremove -y \
> + && dnf clean all -y \
> + && rm -rf /temp
> +
> +RUN git config --global --add safe.directory /docs
> +
> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
> +CMD ["publish"]
> diff --git a/documentation/tools/Containerfile.fedora b/documentation/tools/Containerfile.fedora
> new file mode 120000
> index 0000000000000000000000000000000000000000..7237e9b99f4132957c0ad5b11fa6cefe9daaec74
> --- /dev/null
> +++ b/documentation/tools/Containerfile.fedora
> @@ -0,0 +1 @@
> +Containerfile.dnf
> \ No newline at end of file
> diff --git a/documentation/tools/Containerfile.ubuntu b/documentation/tools/Containerfile.ubuntu
> new file mode 120000
> index 0000000000000000000000000000000000000000..5a7a425197afc2928802fcad5f34699b1ad3348a
> --- /dev/null
> +++ b/documentation/tools/Containerfile.ubuntu
> @@ -0,0 +1 @@
> +Containerfile.apt
> \ No newline at end of file
> diff --git a/documentation/tools/Containerfile.zypper b/documentation/tools/Containerfile.zypper
> new file mode 100644
> index 0000000000000000000000000000000000000000..567256d65b580aab45e27a7b7a386e7a8dbe107f
> --- /dev/null
> +++ b/documentation/tools/Containerfile.zypper
> @@ -0,0 +1,18 @@
> +ARG ARG_FROM=opensuse/leap:15.4 # default value to avoid warning
> +FROM $ARG_FROM
> +
> +ARG HOST_DOCS_PACKAGES=opensuse_host_packages_docs.sh
> +
> +# relative to the location of the dockerfile
> +COPY --chmod=777 ${HOST_DOCS_PACKAGES} /temp/host_packages_docs.sh
> +
> +RUN zypper update -y \
> + && zypper install -y sudo \
> + && yes | /temp/host_packages_docs.sh \
> + && zypper clean --all \
> + && rm -rf /temp
> +
> +RUN git config --global --add safe.directory /docs
> +
> +ENTRYPOINT ["/usr/bin/env", "make", "-C", "documentation/"]
> +CMD ["publish"]
> diff --git a/documentation/tools/build-docs-container b/documentation/tools/build-docs-container
> new file mode 100755
> index 0000000000000000000000000000000000000000..4a799bd8e795f1ab7d55eb99116fe859aff2692c
> --- /dev/null
> +++ b/documentation/tools/build-docs-container
> @@ -0,0 +1,101 @@
> +#!/usr/bin/env bash
> +#
> +# Build a container ready to build the documentation be reading the dependencies
> +# listed in poky.yaml.in, and start a documentation build in this container.

Needs an update, since we don't read poky.yaml.in here.

> +#
> +# Usage:
> +#
> +#   ./documentation/tools/build-docs-container <image> [<make target>]
> +#
> +# e.g.:
> +#
> +#   ./documentation/tools/build-docs-container ubuntu:24.04 html
> +#
> +# Will build the docs in an Ubuntu 24.04 container in html.
> +#
> +# The container engine can be selected by exporting CONTAINERCMD in the
> +# environment. The default is docker, but podman can also be used.
> +
> +set -eu -o pipefail
> +
> +SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
> +CONTAINERCMD=${CONTAINERCMD:-docker}
> +DOCS_DIR="$SCRIPT_DIR/../.."
> +
> +main ()
> +{
> +  local image="$1"
> +  shift
> +
> +  OCI=$(which "$CONTAINERCMD")
> +
> +  # docker build doesn't accept 2 colons, so "sanitize" the name
> +  local sanitized_dockername
> +  sanitized_dockername=$(echo "$image" | tr ':.' '-')
> +
> +  local version
> +  version=$(echo "$image" | awk -F: '{print $NF}')
> +
> +  case $image in
> +    almalinux*)
> +      containerfile=Containerfile.almalinux
> +      host_packages_docs=almalinux_host_packages_docs.sh
> +      ;;
> +    debian*)
> +      containerfile=Containerfile.debian
> +      host_packages_docs=ubuntu_host_packages_docs.sh
> +      ;;
> +    fedora*)
> +      containerfile=Containerfile.fedora
> +      host_packages_docs=fedora_host_packages_docs.sh
> +      ;;
> +    opensuse* | leap*)
> +      image=opensuse/leap:$version
> +      containerfile=Containerfile.zypper
> +      host_packages_docs=opensuse_host_packages_docs.sh
> +      ;;
> +    ubuntu*)
> +      containerfile=Containerfile.ubuntu
> +      host_packages_docs=ubuntu_host_packages_docs.sh
> +      ;;
> +    *)
> +      echo "$image not supported"
> +      exit 1
> +      ;;
> +  esac
> +
> +  $OCI build \
> +    --tag "yocto-docs-$sanitized_dockername:latest" \
> +    --build-arg ARG_FROM="docker.io/$image" \
> +    --build-arg HOST_DOCS_PACKAGES="$host_packages_docs" \
> +    --file "$SCRIPT_DIR/$containerfile" \
> +    "$SCRIPT_DIR/"
> +
> +  local -a args_run=(
> +    --rm
> +    --interactive
> +    --tty
> +    --volume="$DOCS_DIR:/docs:rw"
> +    --workdir=/docs
> +    --security-opt label=disable
> +  )
> +
> +  if [ "$OCI" = "docker" ]; then
> +    args_run+=(
> +      --user="$(id -u)":"$(id -g)"
> +    )
> +  elif [ "$OCI" = "podman" ]; then
> +    # we need net access to fetch bitbake terms
> +    args_run+=(
> +      --cap-add=NET_RAW
> +      --userns=keep-id
> +    )
> +  fi
> +
> +  $OCI run \
> +    "${args_run[@]}" \
> +    "yocto-docs-$sanitized_dockername" \
> +    "$*"
> +}
> +
> +main "$@"
> diff --git a/documentation/tools/opensuse_host_packages_docs.sh b/documentation/tools/opensuse_host_packages_docs.sh
> index 90f52d576451ee0a7c481b31b7605e4a870496cd..18975bd8d2db93eaadc663fac738e9400fd11ffb 100644
> --- a/documentation/tools/opensuse_host_packages_docs.sh
> +++ b/documentation/tools/opensuse_host_packages_docs.sh
> @@ -1,2 +1,2 @@
> -sudo zypper install git make python3-pip which inkscape texlive-fncychap
> +sudo zypper --non-interactive install git make python3-pip which inkscape texlive-fncychap
>  sudo pip3 install sphinx sphinx_rtd_theme pyyaml
> diff --git a/documentation/tools/opensuse_host_packages_essential.sh b/documentation/tools/opensuse_host_packages_essential.sh
> index ef24fcefd99c9ecc335c594965965bb36e43cd05..20d934c34b833f547463d2e635ee16db935de3dc 100644
> --- a/documentation/tools/opensuse_host_packages_essential.sh
> +++ b/documentation/tools/opensuse_host_packages_essential.sh
> @@ -1,2 +1,2 @@
> -sudo zypper install python gcc gcc-c++ git chrpath make wget python-xml diffstat makeinfo python-curses patch socat python3 python3-curses tar python3-pip python3-pexpect xz which python3-Jinja2 rpcgen zstd lz4 bzip2 gzip hostname libacl1
> +sudo zypper --non-interactive install python gcc gcc-c++ git chrpath make wget python-xml diffstat makeinfo python-curses patch socat python3 python3-curses tar python3-pip python3-pexpect xz which python3-Jinja2 rpcgen zstd lz4 bzip2 gzip hostname libacl1
>  sudo pip3 install GitPython

Looks good to me otherwise!

With the rebase I'd take these patches :)

Thanks,
Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com