* [XEN PATCH v1 1/6] CI: Refresh the Debian 12 x86_64 container
2024-10-24 10:04 [XEN PATCH v1 0/6] automation: Refresh the remaining Debian containers Javi Merino
@ 2024-10-24 10:04 ` Javi Merino
2024-10-24 14:04 ` Andrew Cooper
2024-10-24 10:04 ` [XEN PATCH v1 2/6] CI: Refresh the Debian 12 x86_32 container Javi Merino
` (4 subsequent siblings)
5 siblings, 1 reply; 19+ messages in thread
From: Javi Merino @ 2024-10-24 10:04 UTC (permalink / raw)
To: xen-devel; +Cc: Javi Merino, Doug Goldstein, Stefano Stabellini
Rework the container to use heredocs for readability, and use
apt-get --no-install-recommends to keep the size down.
This reduces the size of the (uncompressed) container from 3.44GB to 1.67GB.
Signed-off-by: Javi Merino <javi.merino@cloud.com>
---
automation/build/debian/12-x86_64.dockerfile | 68 ++++++++++++++++++++
automation/build/debian/bookworm.dockerfile | 57 ----------------
automation/gitlab-ci/build.yaml | 20 +++---
automation/gitlab-ci/test.yaml | 14 ++--
automation/scripts/containerize | 2 +-
5 files changed, 86 insertions(+), 75 deletions(-)
create mode 100644 automation/build/debian/12-x86_64.dockerfile
delete mode 100644 automation/build/debian/bookworm.dockerfile
diff --git a/automation/build/debian/12-x86_64.dockerfile b/automation/build/debian/12-x86_64.dockerfile
new file mode 100644
index 000000000000..e0ca8b7e9c91
--- /dev/null
+++ b/automation/build/debian/12-x86_64.dockerfile
@@ -0,0 +1,68 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 debian:bookworm
+LABEL maintainer.name="The Xen Project" \
+ maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+# build depends
+RUN <<EOF
+#!/bin/bash
+ set -eu
+
+ apt-get update
+ DEPS=(
+ # Xen
+ bison
+ build-essential
+ checkpolicy
+ clang
+ flex
+
+ # Tools (general)
+ ca-certificates
+ expect
+ git-core
+ libnl-3-dev
+ pkg-config
+ wget
+ # libxenguest dombuilder
+ liblzma-dev
+ zlib1g-dev
+ # libacpi
+ acpica-tools
+ # libxl
+ uuid-dev
+ libyajl-dev
+ # RomBIOS
+ bcc
+ bin86
+ # xentop
+ libncurses5-dev
+ # Python bindings
+ python3-dev
+ python3-setuptools
+ # Ocaml bindings/oxenstored
+ ocaml-nox
+ ocaml-findlib
+ # To build the documentation
+ pandoc
+
+ # for test phase, qemu-smoke-* jobs
+ qemu-system-x86
+
+ # for qemu-alpine-x86_64-gcc
+ busybox-static
+ cpio
+
+ # For *-efi jobs
+ ovmf
+ )
+
+ apt-get -y --no-install-recommends install "${DEPS[@]}"
+
+ rm -rf /var/lib/apt/lists*
+EOF
+
+USER root
+WORKDIR /build
diff --git a/automation/build/debian/bookworm.dockerfile b/automation/build/debian/bookworm.dockerfile
deleted file mode 100644
index 72e01aa58b55..000000000000
--- a/automation/build/debian/bookworm.dockerfile
+++ /dev/null
@@ -1,57 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 debian:bookworm
-LABEL maintainer.name="The Xen Project" \
- maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-# build depends
-RUN apt-get update && \
- apt-get --quiet --yes install \
- build-essential \
- zlib1g-dev \
- libncurses5-dev \
- python3-dev \
- python3-setuptools \
- uuid-dev \
- libyajl-dev \
- libaio-dev \
- libglib2.0-dev \
- clang \
- libpixman-1-dev \
- pkg-config \
- flex \
- bison \
- acpica-tools \
- bin86 \
- bcc \
- liblzma-dev \
- libnl-3-dev \
- ocaml-nox \
- libfindlib-ocaml-dev \
- markdown \
- transfig \
- pandoc \
- checkpolicy \
- wget \
- git \
- nasm \
- gnupg \
- apt-transport-https \
- golang \
- # for test phase, qemu-smoke-* jobs
- qemu-system-x86 \
- expect \
- # For *-efi jobs
- ovmf \
- # for test phase, qemu-alpine-* jobs
- cpio \
- busybox-static \
- && \
- apt-get autoremove -y && \
- apt-get clean && \
- rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 09dd9e6ccbd0..eb2c23619a2c 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -341,15 +341,15 @@ alpine-3.18-gcc-debug:
CONTAINER: alpine:3.18
BUILD_QEMU_XEN: y
-debian-bookworm-gcc-debug:
+debian-12-x86_64-gcc-debug:
extends: .gcc-x86-64-build-debug
variables:
- CONTAINER: debian:bookworm
+ CONTAINER: debian:12-x86_64
-debian-bookworm-clang-debug:
+debian-12-x86_64-clang-debug:
extends: .clang-x86-64-build-debug
variables:
- CONTAINER: debian:bookworm
+ CONTAINER: debian:12-x86_64
debian-12-ppc64le-gcc-debug:
extends: .gcc-ppc64le-cross-build-debug
@@ -553,20 +553,20 @@ debian-12-x86_64-gcc-ibt:
EXTRA_FIXED_RANDCONFIG: |
CONFIG_XEN_IBT=y
-debian-bookworm-clang:
+debian-12-x86_64-clang:
extends: .clang-x86-64-build
variables:
- CONTAINER: debian:bookworm
+ CONTAINER: debian:12-x86_64
-debian-bookworm-gcc:
+debian-12-x86_64-gcc:
extends: .gcc-x86-64-build
variables:
- CONTAINER: debian:bookworm
+ CONTAINER: debian:12-x86_64
-debian-bookworm-gcc-randconfig:
+debian-12-x86_64-gcc-randconfig:
extends: .gcc-x86-64-build
variables:
- CONTAINER: debian:bookworm
+ CONTAINER: debian:12-x86_64
RANDCONFIG: y
debian-bookworm-32-clang-debug:
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index e76a37bef32d..0812ddb42d9b 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -43,7 +43,7 @@
.qemu-x86-64:
extends: .test-jobs-common
variables:
- CONTAINER: debian:bookworm
+ CONTAINER: debian:12-x86_64
LOGFILE: qemu-smoke-x86-64.log
artifacts:
paths:
@@ -155,7 +155,7 @@
build-each-commit-gcc:
extends: .test-jobs-common
variables:
- CONTAINER: debian:bookworm
+ CONTAINER: debian:12-x86_64
XEN_TARGET_ARCH: x86_64
CC: gcc
script:
@@ -461,35 +461,35 @@ qemu-smoke-x86-64-gcc:
script:
- ./automation/scripts/qemu-smoke-x86-64.sh pv 2>&1 | tee ${LOGFILE}
needs:
- - debian-bookworm-gcc-debug
+ - debian-12-x86_64-gcc-debug
qemu-smoke-x86-64-clang:
extends: .qemu-smoke-x86-64
script:
- ./automation/scripts/qemu-smoke-x86-64.sh pv 2>&1 | tee ${LOGFILE}
needs:
- - debian-bookworm-clang-debug
+ - debian-12-x86_64-clang-debug
qemu-smoke-x86-64-gcc-pvh:
extends: .qemu-smoke-x86-64
script:
- ./automation/scripts/qemu-smoke-x86-64.sh pvh 2>&1 | tee ${LOGFILE}
needs:
- - debian-bookworm-gcc-debug
+ - debian-12-x86_64-gcc-debug
qemu-smoke-x86-64-clang-pvh:
extends: .qemu-smoke-x86-64
script:
- ./automation/scripts/qemu-smoke-x86-64.sh pvh 2>&1 | tee ${LOGFILE}
needs:
- - debian-bookworm-clang-debug
+ - debian-12-x86_64-clang-debug
qemu-smoke-x86-64-gcc-efi:
extends: .qemu-smoke-x86-64
script:
- ./automation/scripts/qemu-smoke-x86-64-efi.sh pv 2>&1 | tee ${LOGFILE}
needs:
- - debian-bookworm-gcc-debug
+ - debian-12-x86_64-gcc-debug
qemu-smoke-riscv64-gcc:
extends: .qemu-riscv64
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index 6ac02c42d124..ea6e1a9b18f4 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -34,7 +34,7 @@ case "_${CONTAINER}" in
_bullseye-riscv64) CONTAINER="${BASE}/debian:11-riscv64" ;;
_bookworm-riscv64) CONTAINER="${BASE}/debian:12-riscv64" ;;
_bookworm-x86_64-gcc-ibt) CONTAINER="${BASE}/debian:12-x86_64-gcc-ibt" ;;
- _bookworm|_) CONTAINER="${BASE}/debian:bookworm" ;;
+ _bookworm|_bookworm-x86_64|_) CONTAINER="${BASE}/debian:12-x86_64" ;;
_bookworm-i386) CONTAINER="${BASE}/debian:bookworm-i386" ;;
_bookworm-arm64v8-arm32-gcc) CONTAINER="${BASE}/debian:bookworm-arm64v8-arm32-gcc" ;;
_bookworm-arm64v8) CONTAINER="${BASE}/debian:bookworm-arm64v8" ;;
--
2.45.2
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [XEN PATCH v1 1/6] CI: Refresh the Debian 12 x86_64 container
2024-10-24 10:04 ` [XEN PATCH v1 1/6] CI: Refresh the Debian 12 x86_64 container Javi Merino
@ 2024-10-24 14:04 ` Andrew Cooper
2024-10-24 15:10 ` Javi Merino
0 siblings, 1 reply; 19+ messages in thread
From: Andrew Cooper @ 2024-10-24 14:04 UTC (permalink / raw)
To: Javi Merino, xen-devel; +Cc: Doug Goldstein, Stefano Stabellini
On 24/10/2024 11:04 am, Javi Merino wrote:
> Rework the container to use heredocs for readability, and use
> apt-get --no-install-recommends to keep the size down.
>
> This reduces the size of the (uncompressed) container from 3.44GB to 1.67GB.
!!
>
> Signed-off-by: Javi Merino <javi.merino@cloud.com>
> ---
> automation/build/debian/12-x86_64.dockerfile | 68 ++++++++++++++++++++
> automation/build/debian/bookworm.dockerfile | 57 ----------------
> automation/gitlab-ci/build.yaml | 20 +++---
> automation/gitlab-ci/test.yaml | 14 ++--
> automation/scripts/containerize | 2 +-
> 5 files changed, 86 insertions(+), 75 deletions(-)
> create mode 100644 automation/build/debian/12-x86_64.dockerfile
> delete mode 100644 automation/build/debian/bookworm.dockerfile
>
> diff --git a/automation/build/debian/12-x86_64.dockerfile b/automation/build/debian/12-x86_64.dockerfile
> new file mode 100644
> index 000000000000..e0ca8b7e9c91
> --- /dev/null
> +++ b/automation/build/debian/12-x86_64.dockerfile
> @@ -0,0 +1,68 @@
> +# syntax=docker/dockerfile:1
> +FROM --platform=linux/amd64 debian:bookworm
> +LABEL maintainer.name="The Xen Project" \
> + maintainer.email="xen-devel@lists.xenproject.org"
This wants to become two LABEL lines.
> +
> +ENV DEBIAN_FRONTEND=noninteractive
> +
> +# build depends
> +RUN <<EOF
> +#!/bin/bash
> + set -eu
Doesn't this need a `useradd --create-home user` here?
[Edit] Oh, no, because of the script change. In which case can you note
this in the commit message and says a root container for now, until some
other CI scripts can be adjusted.
> +
> + apt-get update
apt-get -y
> + DEPS=(
> + # Xen
> + bison
> + build-essential
> + checkpolicy
> + clang
> + flex
> +
> + # Tools (general)
> + ca-certificates
Interestingly, we've gained ca-certificates and dropped apt-transport-https.
ca-certificates is a side effect of --no-install-recommends, so is
fine. I recall there being a specific reason why we needed
apt-transport-https, but I can't recall why exactly. Something about
the LetsEncrypt Cert used by xenbits IIRC.
Anthony - do you remember?
> + expect
Expect is only for the test phase, so should move later.
> + git-core
> + libnl-3-dev
libnl-3-dev should be down in the #libxl section. It's only for COLO
support.
> + pkg-config
> + wget
> + # libxenguest dombuilder
> + liblzma-dev
> + zlib1g-dev
This is also fun. In Ubuntu, I've got:
libbz2-dev
libzstd-dev
liblzo2-dev
liblzma-dev
zlib1g-dev
which I think is all the algorithms we support in libxenguest.
Any decompressor which we don't find a suitable devel package gets the
hypervisor form instead.
> + # To build the documentation
> + pandoc
I know we had pandoc before, but I'd like to drop it.
I'm intending to turn off docs generally, and do them separately in a
single job that has *all* the docs build dependencies, not a misc subset
that the build system happens not to complain at.
I'm on the fence about the Qemu build things. It's off by default now,
but the container never previously had meson/ninja so it wouldn't have
built either. Perhaps leave them out until someone complains.
One thing you did drop which probably wants to stay is golang. We have
golang bindings for libxl which (like Ocaml) are built conditionally on
finding the toolchain.
~Andrew
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [XEN PATCH v1 1/6] CI: Refresh the Debian 12 x86_64 container
2024-10-24 14:04 ` Andrew Cooper
@ 2024-10-24 15:10 ` Javi Merino
2024-10-25 14:46 ` Andrew Cooper
0 siblings, 1 reply; 19+ messages in thread
From: Javi Merino @ 2024-10-24 15:10 UTC (permalink / raw)
To: Andrew Cooper; +Cc: xen-devel, Doug Goldstein, Stefano Stabellini
On Thu, Oct 24, 2024 at 03:04:10PM +0100, Andrew Cooper wrote:
> On 24/10/2024 11:04 am, Javi Merino wrote:
> > Rework the container to use heredocs for readability, and use
> > apt-get --no-install-recommends to keep the size down.
> >
> > This reduces the size of the (uncompressed) container from 3.44GB to 1.67GB.
>
> !!
>
> >
> > Signed-off-by: Javi Merino <javi.merino@cloud.com>
> > ---
> > automation/build/debian/12-x86_64.dockerfile | 68 ++++++++++++++++++++
> > automation/build/debian/bookworm.dockerfile | 57 ----------------
> > automation/gitlab-ci/build.yaml | 20 +++---
> > automation/gitlab-ci/test.yaml | 14 ++--
> > automation/scripts/containerize | 2 +-
> > 5 files changed, 86 insertions(+), 75 deletions(-)
> > create mode 100644 automation/build/debian/12-x86_64.dockerfile
> > delete mode 100644 automation/build/debian/bookworm.dockerfile
> >
> > diff --git a/automation/build/debian/12-x86_64.dockerfile b/automation/build/debian/12-x86_64.dockerfile
> > new file mode 100644
> > index 000000000000..e0ca8b7e9c91
> > --- /dev/null
> > +++ b/automation/build/debian/12-x86_64.dockerfile
> > @@ -0,0 +1,68 @@
> > +# syntax=docker/dockerfile:1
> > +FROM --platform=linux/amd64 debian:bookworm
> > +LABEL maintainer.name="The Xen Project" \
> > + maintainer.email="xen-devel@lists.xenproject.org"
>
> This wants to become two LABEL lines.
Yes, Anthony pointed it out in another patch. I have fixed all the
dockerfiles in these series.
> > +
> > +ENV DEBIAN_FRONTEND=noninteractive
> > +
> > +# build depends
> > +RUN <<EOF
> > +#!/bin/bash
> > + set -eu
>
> Doesn't this need a `useradd --create-home user` here?
>
> [Edit] Oh, no, because of the script change. In which case can you note
> this in the commit message and says a root container for now, until some
> other CI scripts can be adjusted.
I put it in the cover letter. I'll add it to the commit message as
well.
> > +
> > + apt-get update
>
> apt-get -y
apt-get update refreshes the package lists. -y doesn't do anything
here. It is needed for "apt-get install" below but not for
apt-get update. It would be needed for "apt-get upgrade", but
we don't.
> > + DEPS=(
> > + # Xen
> > + bison
> > + build-essential
> > + checkpolicy
> > + clang
> > + flex
> > +
> > + # Tools (general)
> > + ca-certificates
>
> Interestingly, we've gained ca-certificates and dropped apt-transport-https.
ca-certificates is needed for curl, wget or anything that tries to
validate tls certificates. It is a Recommends of libcurl, as
curl by default validates the ca certificate of https servers.
> ca-certificates is a side effect of --no-install-recommends, so is
> fine. I recall there being a specific reason why we needed
> apt-transport-https, but I can't recall why exactly. Something about
> the LetsEncrypt Cert used by xenbits IIRC.
I dropped apt-transport-https because it doesn't make sense to have
it. apt-transport-https allows apt to access package repositories over https,
but we were installing alongside all the other packages. apt is never
used again, so giving it the ability to install packages over https is
pointless.
> Anthony - do you remember?
>
>
> > + expect
>
> Expect is only for the test phase, so should move later.
I put it here because ./configure checks for it.
> > + git-core
> > + libnl-3-dev
>
> libnl-3-dev should be down in the #libxl section. It's only for COLO
> support.
Moved.
> > + pkg-config
> > + wget
> > + # libxenguest dombuilder
> > + liblzma-dev
> > + zlib1g-dev
>
> This is also fun. In Ubuntu, I've got:
>
> libbz2-dev
> libzstd-dev
> liblzo2-dev
> liblzma-dev
> zlib1g-dev
>
> which I think is all the algorithms we support in libxenguest.
I did this in the arm64v8 container and forgot to do it here. Fixed now.
> Any decompressor which we don't find a suitable devel package gets the
> hypervisor form instead.
>
> > + # To build the documentation
> > + pandoc
>
> I know we had pandoc before, but I'd like to drop it.
>
> I'm intending to turn off docs generally, and do them separately in a
> single job that has *all* the docs build dependencies, not a misc subset
> that the build system happens not to complain at.
I had the "build the docs as its own job" in my TODO list and was
going to drop pandoc from this dockerfile then. I can remove pandoc
in this commit if you prefer.
> I'm on the fence about the Qemu build things. It's off by default now,
> but the container never previously had meson/ninja so it wouldn't have
> built either. Perhaps leave them out until someone complains.
I thought I had removed them. Is there anything else that needs to
go?
> One thing you did drop which probably wants to stay is golang. We have
> golang bindings for libxl which (like Ocaml) are built conditionally on
> finding the toolchain.
Gah. Another one that I did in the arm64 container that I forgot to
move here. I will add golang-go in the next version of the series.
Thanks,
Javi
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [XEN PATCH v1 1/6] CI: Refresh the Debian 12 x86_64 container
2024-10-24 15:10 ` Javi Merino
@ 2024-10-25 14:46 ` Andrew Cooper
2024-11-04 15:40 ` Javi Merino
0 siblings, 1 reply; 19+ messages in thread
From: Andrew Cooper @ 2024-10-25 14:46 UTC (permalink / raw)
To: Javi Merino; +Cc: xen-devel, Doug Goldstein, Stefano Stabellini
On 24/10/2024 4:10 pm, Javi Merino wrote:
> On Thu, Oct 24, 2024 at 03:04:10PM +0100, Andrew Cooper wrote:
>> On 24/10/2024 11:04 am, Javi Merino wrote:
>>> +
>>> + apt-get update
>> apt-get -y
> apt-get update refreshes the package lists. -y doesn't do anything
> here. It is needed for "apt-get install" below but not for
> apt-get update. It would be needed for "apt-get upgrade", but
> we don't.
Hmm ok. We might want to adjust the others to match then.
>
>>> + DEPS=(
>>> + # Xen
>>> + bison
>>> + build-essential
>>> + checkpolicy
>>> + clang
>>> + flex
>>> +
>>> + # Tools (general)
>>> + ca-certificates
>> Interestingly, we've gained ca-certificates and dropped apt-transport-https.
> ca-certificates is needed for curl, wget or anything that tries to
> validate tls certificates. It is a Recommends of libcurl, as
> curl by default validates the ca certificate of https servers.
>
>> ca-certificates is a side effect of --no-install-recommends, so is
>> fine. I recall there being a specific reason why we needed
>> apt-transport-https, but I can't recall why exactly. Something about
>> the LetsEncrypt Cert used by xenbits IIRC.
> I dropped apt-transport-https because it doesn't make sense to have
> it. apt-transport-https allows apt to access package repositories over https,
> but we were installing alongside all the other packages. apt is never
> used again, so giving it the ability to install packages over https is
> pointless.
That is, as they say, an assumption.
fe746c26c0d2 ("automation/gitlab: add https transport support to Debian
images")
Although, subsequently the use of apt.llvm.org was removed:
a6b1e2b80fe2 ("automation: Remove clang-8 from Debian unstable container")
7a2983757216 ("CI: Remove llvm-8 from the Debian Stretch container")
So I guess we're back to being ok without it.
>>> + expect
>> Expect is only for the test phase, so should move later.
> I put it here because ./configure checks for it.
It does?
That's not necessary/expected.
>> Any decompressor which we don't find a suitable devel package gets the
>> hypervisor form instead.
>>
>>> + # To build the documentation
>>> + pandoc
>> I know we had pandoc before, but I'd like to drop it.
>>
>> I'm intending to turn off docs generally, and do them separately in a
>> single job that has *all* the docs build dependencies, not a misc subset
>> that the build system happens not to complain at.
> I had the "build the docs as its own job" in my TODO list and was
> going to drop pandoc from this dockerfile then. I can remove pandoc
> in this commit if you prefer.
Dropping packages from existing containers is complicated, because the
container (name) is shared with prior branches. You have to wait until
the oldest version of Xen which still uses the package leaves testing
(== leaves security support, == 3y), or we've backported changes to all
branches to drop the dependency.
The rename here gives us leeway because this change won't clobber any
older branches in Xen, but I don't want to set the precedent.
>
>> I'm on the fence about the Qemu build things. It's off by default now,
>> but the container never previously had meson/ninja so it wouldn't have
>> built either. Perhaps leave them out until someone complains.
> I thought I had removed them. Is there anything else that needs to
> go?
These containers are both for CI and human use, so "what happens in CI"
isn't the only consideration.
But, given that Qemu didn't build in the old container anyway, I'm not
overly fussed about keeping it working in the new container.
So yes, please keep the deps removed. We can always add them back in later.
~Andrew
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [XEN PATCH v1 1/6] CI: Refresh the Debian 12 x86_64 container
2024-10-25 14:46 ` Andrew Cooper
@ 2024-11-04 15:40 ` Javi Merino
0 siblings, 0 replies; 19+ messages in thread
From: Javi Merino @ 2024-11-04 15:40 UTC (permalink / raw)
To: Andrew Cooper; +Cc: xen-devel, Doug Goldstein, Stefano Stabellini
On Fri, Oct 25, 2024 at 03:46:35PM +0100, Andrew Cooper wrote:
> On 24/10/2024 4:10 pm, Javi Merino wrote:
> > On Thu, Oct 24, 2024 at 03:04:10PM +0100, Andrew Cooper wrote:
> >> On 24/10/2024 11:04 am, Javi Merino wrote:
> >>> +
> >>> + apt-get update
> >> apt-get -y
> > apt-get update refreshes the package lists. -y doesn't do anything
> > here. It is needed for "apt-get install" below but not for
> > apt-get update. It would be needed for "apt-get upgrade", but
> > we don't.
>
> Hmm ok. We might want to adjust the others to match then.
Ok, I will send a patch for the others.
> >>> + expect
> >> Expect is only for the test phase, so should move later.
> > I put it here because ./configure checks for it.
>
> It does?
>
> That's not necessary/expected.
Ok, I'll move it.
> >> Any decompressor which we don't find a suitable devel package gets the
> >> hypervisor form instead.
> >>
> >>> + # To build the documentation
> >>> + pandoc
> >> I know we had pandoc before, but I'd like to drop it.
> >>
> >> I'm intending to turn off docs generally, and do them separately in a
> >> single job that has *all* the docs build dependencies, not a misc subset
> >> that the build system happens not to complain at.
> > I had the "build the docs as its own job" in my TODO list and was
> > going to drop pandoc from this dockerfile then. I can remove pandoc
> > in this commit if you prefer.
>
> Dropping packages from existing containers is complicated, because the
> container (name) is shared with prior branches. You have to wait until
> the oldest version of Xen which still uses the package leaves testing
> (== leaves security support, == 3y), or we've backported changes to all
> branches to drop the dependency.
>
> The rename here gives us leeway because this change won't clobber any
> older branches in Xen, but I don't want to set the precedent.
Ok, this is another one to fix for the future. Having dependencies in
the dockerfile in this branch because it is used in another branch is
a recipe for disaster. CI for a branch should be defined on that
branch.
I'll drop pandoc.
> >> I'm on the fence about the Qemu build things. It's off by default now,
> >> but the container never previously had meson/ninja so it wouldn't have
> >> built either. Perhaps leave them out until someone complains.
> > I thought I had removed them. Is there anything else that needs to
> > go?
>
> These containers are both for CI and human use, so "what happens in CI"
> isn't the only consideration.
>
> But, given that Qemu didn't build in the old container anyway, I'm not
> overly fussed about keeping it working in the new container.
>
> So yes, please keep the deps removed. We can always add them back in later.
Ok.
Cheers,
Javi
^ permalink raw reply [flat|nested] 19+ messages in thread
* [XEN PATCH v1 2/6] CI: Refresh the Debian 12 x86_32 container
2024-10-24 10:04 [XEN PATCH v1 0/6] automation: Refresh the remaining Debian containers Javi Merino
2024-10-24 10:04 ` [XEN PATCH v1 1/6] CI: Refresh the Debian 12 x86_64 container Javi Merino
@ 2024-10-24 10:04 ` Javi Merino
2024-10-24 13:04 ` Javi Merino
2024-10-24 10:04 ` [XEN PATCH v1 3/6] CI: Refresh the Debian 12 arm64 container Javi Merino
` (3 subsequent siblings)
5 siblings, 1 reply; 19+ messages in thread
From: Javi Merino @ 2024-10-24 10:04 UTC (permalink / raw)
To: xen-devel; +Cc: Javi Merino, Doug Goldstein, Stefano Stabellini
Rework the container to be non-root, use heredocs for readability, and
use apt-get --no-install-recommends to keep the size down. Rename the
job to x86_32, to be consistent with XEN_TARGET_ARCH and the
naming scheme of all the other CI jobs:
${VERSION}-${ARCH}-${BUILD_NAME}
Remove build dependencies for building QEMU, as we don't do it since
e305256e69b1 ("CI: Stop building QEMU in general").
Remove build dependencies for the documentation as we don't have to
build it for every single arch.
This reduces the size of the container from 2.22GB to 1.32Gb.
Signed-off-by: Javi Merino <javi.merino@cloud.com>
---
automation/build/debian/12-x86_32.dockerfile | 51 +++++++++++++++++++
.../build/debian/bookworm-i386.dockerfile | 50 ------------------
automation/gitlab-ci/build.yaml | 8 +--
automation/scripts/containerize | 2 +-
4 files changed, 56 insertions(+), 55 deletions(-)
create mode 100644 automation/build/debian/12-x86_32.dockerfile
delete mode 100644 automation/build/debian/bookworm-i386.dockerfile
diff --git a/automation/build/debian/12-x86_32.dockerfile b/automation/build/debian/12-x86_32.dockerfile
new file mode 100644
index 000000000000..d3c67a256212
--- /dev/null
+++ b/automation/build/debian/12-x86_32.dockerfile
@@ -0,0 +1,51 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/i386 debian:bookworm
+LABEL maintainer.name="The Xen Project" \
+ maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV USER root
+
+# build depends
+RUN <<EOF
+#!/bin/bash
+ set -eu
+
+ useradd --create-home user
+
+ apt-get update
+ DEPS=(
+ # Xen
+ bison
+ build-essential
+ checkpolicy
+ clang
+ flex
+
+ # Tools (general)
+ ca-certificates
+ git-core
+ pkg-config
+ wget
+ # libacpi
+ acpica-tools
+ # libxl
+ uuid-dev
+ libyajl-dev
+ # xentop
+ libncurses5-dev
+ # Python bindings
+ python3-dev
+ python3-setuptools
+ # Ocaml bindings/oxenstored
+ ocaml-nox
+ ocaml-findlib
+ )
+ apt-get -y --no-install-recommends install "${DEPS[@]}"
+
+ rm -rf /var/lib/apt/lists*
+EOF
+
+USER user
+WORKDIR /build
+ENTRYPOINT ["linux32"]
diff --git a/automation/build/debian/bookworm-i386.dockerfile b/automation/build/debian/bookworm-i386.dockerfile
deleted file mode 100644
index 66fa3121c54e..000000000000
--- a/automation/build/debian/bookworm-i386.dockerfile
+++ /dev/null
@@ -1,50 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/i386 debian:bookworm
-LABEL maintainer.name="The Xen Project" \
- maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-ENTRYPOINT ["linux32"]
-
-# build depends
-RUN apt-get update && \
- apt-get --quiet --yes install \
- build-essential \
- zlib1g-dev \
- libncurses5-dev \
- python3-dev \
- python3-setuptools \
- uuid-dev \
- libyajl-dev \
- libaio-dev \
- libglib2.0-dev \
- clang \
- libpixman-1-dev \
- pkg-config \
- flex \
- bison \
- acpica-tools \
- bin86 \
- bcc \
- liblzma-dev \
- libc6-dev \
- libnl-3-dev \
- ocaml-nox \
- libfindlib-ocaml-dev \
- markdown \
- transfig \
- pandoc \
- checkpolicy \
- wget \
- git \
- nasm \
- apt-transport-https \
- && \
- apt-get autoremove -y && \
- apt-get clean && \
- rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index eb2c23619a2c..1b322044ab19 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -569,15 +569,15 @@ debian-12-x86_64-gcc-randconfig:
CONTAINER: debian:12-x86_64
RANDCONFIG: y
-debian-bookworm-32-clang-debug:
+debian-12-x86_32-clang-debug:
extends: .clang-x86-32-build-debug
variables:
- CONTAINER: debian:bookworm-i386
+ CONTAINER: debian:12-x86_32
-debian-bookworm-32-gcc-debug:
+debian-12-x86_32-gcc-debug:
extends: .gcc-x86-32-build-debug
variables:
- CONTAINER: debian:bookworm-i386
+ CONTAINER: debian:12-x86_32
fedora-40-x86_64-gcc:
extends: .gcc-x86-64-build
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index ea6e1a9b18f4..d72c22c103ff 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -35,7 +35,7 @@ case "_${CONTAINER}" in
_bookworm-riscv64) CONTAINER="${BASE}/debian:12-riscv64" ;;
_bookworm-x86_64-gcc-ibt) CONTAINER="${BASE}/debian:12-x86_64-gcc-ibt" ;;
_bookworm|_bookworm-x86_64|_) CONTAINER="${BASE}/debian:12-x86_64" ;;
- _bookworm-i386) CONTAINER="${BASE}/debian:bookworm-i386" ;;
+ _bookworm-i386|_bookworm-x86_32) CONTAINER="${BASE}/debian:12-x86_32" ;;
_bookworm-arm64v8-arm32-gcc) CONTAINER="${BASE}/debian:bookworm-arm64v8-arm32-gcc" ;;
_bookworm-arm64v8) CONTAINER="${BASE}/debian:bookworm-arm64v8" ;;
_bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
--
2.45.2
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [XEN PATCH v1 2/6] CI: Refresh the Debian 12 x86_32 container
2024-10-24 10:04 ` [XEN PATCH v1 2/6] CI: Refresh the Debian 12 x86_32 container Javi Merino
@ 2024-10-24 13:04 ` Javi Merino
0 siblings, 0 replies; 19+ messages in thread
From: Javi Merino @ 2024-10-24 13:04 UTC (permalink / raw)
To: xen-devel; +Cc: Doug Goldstein, Stefano Stabellini
On Thu, Oct 24, 2024 at 11:04:17AM +0100, Javi Merino wrote:
> Rework the container to be non-root, use heredocs for readability, and
> use apt-get --no-install-recommends to keep the size down. Rename the
> job to x86_32, to be consistent with XEN_TARGET_ARCH and the
> naming scheme of all the other CI jobs:
> ${VERSION}-${ARCH}-${BUILD_NAME}
>
> Remove build dependencies for building QEMU, as we don't do it since
> e305256e69b1 ("CI: Stop building QEMU in general").
>
> Remove build dependencies for the documentation as we don't have to
> build it for every single arch.
>
> This reduces the size of the container from 2.22GB to 1.32Gb.
>
> Signed-off-by: Javi Merino <javi.merino@cloud.com>
> ---
> automation/build/debian/12-x86_32.dockerfile | 51 +++++++++++++++++++
> .../build/debian/bookworm-i386.dockerfile | 50 ------------------
> automation/gitlab-ci/build.yaml | 8 +--
> automation/scripts/containerize | 2 +-
> 4 files changed, 56 insertions(+), 55 deletions(-)
> create mode 100644 automation/build/debian/12-x86_32.dockerfile
> delete mode 100644 automation/build/debian/bookworm-i386.dockerfile
>
> diff --git a/automation/build/debian/12-x86_32.dockerfile b/automation/build/debian/12-x86_32.dockerfile
> new file mode 100644
> index 000000000000..d3c67a256212
> --- /dev/null
> +++ b/automation/build/debian/12-x86_32.dockerfile
> @@ -0,0 +1,51 @@
> +# syntax=docker/dockerfile:1
> +FROM --platform=linux/i386 debian:bookworm
> +LABEL maintainer.name="The Xen Project" \
> + maintainer.email="xen-devel@lists.xenproject.org"
> +
> +ENV DEBIAN_FRONTEND=noninteractive
> +ENV USER root
This line shouldn't be here. I forgot to remove it. The container
has a USER line at the end to set it.
Cheers,
Javi
^ permalink raw reply [flat|nested] 19+ messages in thread
* [XEN PATCH v1 3/6] CI: Refresh the Debian 12 arm64 container
2024-10-24 10:04 [XEN PATCH v1 0/6] automation: Refresh the remaining Debian containers Javi Merino
2024-10-24 10:04 ` [XEN PATCH v1 1/6] CI: Refresh the Debian 12 x86_64 container Javi Merino
2024-10-24 10:04 ` [XEN PATCH v1 2/6] CI: Refresh the Debian 12 x86_32 container Javi Merino
@ 2024-10-24 10:04 ` Javi Merino
2024-10-24 23:03 ` Stefano Stabellini
2024-10-24 10:04 ` [XEN PATCH v1 4/6] CI: Refresh the Debian 12 arm32 cross compile container Javi Merino
` (2 subsequent siblings)
5 siblings, 1 reply; 19+ messages in thread
From: Javi Merino @ 2024-10-24 10:04 UTC (permalink / raw)
To: xen-devel; +Cc: Javi Merino, Doug Goldstein, Stefano Stabellini
Rework the container to use heredocs for readability and use apt-get
--no-install-recommends to keep the size down. Rename the job to
debian-12-arm64-* to follow the naming scheme of all the other CI jobs.
This reduces the size of the debian:12-arm64v8 from 2.25GB down to 1.62GB.
Signed-off-by: Javi Merino <javi.merino@cloud.com>
---
automation/build/debian/12-arm64v8.dockerfile | 68 +++++++++++++++++++
.../build/debian/bookworm-arm64v8.dockerfile | 55 ---------------
automation/gitlab-ci/build.yaml | 8 +--
automation/gitlab-ci/test.yaml | 4 +-
automation/scripts/containerize | 2 +-
5 files changed, 75 insertions(+), 62 deletions(-)
create mode 100644 automation/build/debian/12-arm64v8.dockerfile
delete mode 100644 automation/build/debian/bookworm-arm64v8.dockerfile
diff --git a/automation/build/debian/12-arm64v8.dockerfile b/automation/build/debian/12-arm64v8.dockerfile
new file mode 100644
index 000000000000..4da1b074aedb
--- /dev/null
+++ b/automation/build/debian/12-arm64v8.dockerfile
@@ -0,0 +1,68 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/arm64/v8 debian:bookworm
+LABEL maintainer.name="The Xen Project" \
+ maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+# build depends
+RUN <<EOF
+#!/bin/bash
+ set -eu
+
+ apt-get update
+ DEPS=(
+ # Xen
+ bison
+ build-essential
+ flex
+ # Flask
+ checkpolicy
+
+ # Tools (general)
+ git-core
+ libext2fs-dev
+ libfdt-dev
+ libglib2.0-dev
+ libpixman-1-dev
+ pkg-config
+ wget
+ # libxenguest dombuilder
+ libbz2-dev
+ liblzma-dev
+ liblzo2-dev
+ libzstd-dev
+ # libacpi
+ acpica-tools
+ # libxl
+ uuid-dev
+ libyajl-dev
+ # Python bindings
+ python3-dev
+ python3-setuptools
+ # Golang bindings
+ golang-go
+ # Ocaml bindings/oxenstored
+ ocaml-nox
+ ocaml-findlib
+
+ # for test phase, qemu-smoke-* jobs
+ busybox-static
+ ca-certificates
+ cpio
+ curl
+ device-tree-compiler
+ expect
+ u-boot-qemu
+ # for imagebuilder
+ file
+ u-boot-tools
+ )
+
+ apt-get -y --no-install-recommends install "${DEPS[@]}"
+
+ rm -rf /var/lib/apt/lists*
+EOF
+
+USER root
+WORKDIR /build
diff --git a/automation/build/debian/bookworm-arm64v8.dockerfile b/automation/build/debian/bookworm-arm64v8.dockerfile
deleted file mode 100644
index da8ae7512589..000000000000
--- a/automation/build/debian/bookworm-arm64v8.dockerfile
+++ /dev/null
@@ -1,55 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/arm64/v8 debian:bookworm
-LABEL maintainer.name="The Xen Project" \
- maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-# build depends
-RUN apt-get update && \
- apt-get --quiet --yes install \
- build-essential \
- zlib1g-dev \
- libncurses5-dev \
- python3-dev \
- python3-setuptools \
- uuid-dev \
- libyajl-dev \
- libaio-dev \
- libglib2.0-dev \
- clang \
- libpixman-1-dev \
- pkg-config \
- flex \
- bison \
- acpica-tools \
- libfdt-dev \
- bin86 \
- bcc \
- liblzma-dev \
- libnl-3-dev \
- ocaml-nox \
- libfindlib-ocaml-dev \
- markdown \
- transfig \
- pandoc \
- checkpolicy \
- wget \
- git \
- nasm \
- # for test phase, qemu-smoke-* jobs
- u-boot-qemu \
- u-boot-tools \
- device-tree-compiler \
- curl \
- cpio \
- busybox-static \
- expect \
- && \
- apt-get autoremove -y && \
- apt-get clean && \
- rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 1b322044ab19..c687298cc924 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -407,15 +407,15 @@ debian-bookworm-gcc-arm32-debug-earlyprintk:
# Arm builds
-debian-bookworm-gcc-arm64:
+debian-12-arm64-gcc:
extends: .gcc-arm64-build
variables:
- CONTAINER: debian:bookworm-arm64v8
+ CONTAINER: debian:12-arm64v8
-debian-bookworm-gcc-debug-arm64:
+debian-12-arm64-gcc-debug:
extends: .gcc-arm64-build-debug
variables:
- CONTAINER: debian:bookworm-arm64v8
+ CONTAINER: debian:12-arm64v8
alpine-3.18-gcc-arm64:
extends: .gcc-arm64-build
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 0812ddb42d9b..248281f47942 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -17,7 +17,7 @@
.qemu-arm64:
extends: .test-jobs-common
variables:
- CONTAINER: debian:bookworm-arm64v8
+ CONTAINER: debian:12-arm64v8
LOGFILE: qemu-smoke-arm64.log
artifacts:
paths:
@@ -30,7 +30,7 @@
.qemu-arm32:
extends: .test-jobs-common
variables:
- CONTAINER: debian:bookworm-arm64v8
+ CONTAINER: debian:12-arm64v8
LOGFILE: qemu-smoke-arm32.log
artifacts:
paths:
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index d72c22c103ff..1b75c8d93724 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -37,7 +37,7 @@ case "_${CONTAINER}" in
_bookworm|_bookworm-x86_64|_) CONTAINER="${BASE}/debian:12-x86_64" ;;
_bookworm-i386|_bookworm-x86_32) CONTAINER="${BASE}/debian:12-x86_32" ;;
_bookworm-arm64v8-arm32-gcc) CONTAINER="${BASE}/debian:bookworm-arm64v8-arm32-gcc" ;;
- _bookworm-arm64v8) CONTAINER="${BASE}/debian:bookworm-arm64v8" ;;
+ _bookworm-arm64v8) CONTAINER="${BASE}/debian:12-arm64v8" ;;
_bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
_opensuse-leap|_leap) CONTAINER="${BASE}/opensuse:leap-15.6-x86_64" ;;
_opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
--
2.45.2
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [XEN PATCH v1 3/6] CI: Refresh the Debian 12 arm64 container
2024-10-24 10:04 ` [XEN PATCH v1 3/6] CI: Refresh the Debian 12 arm64 container Javi Merino
@ 2024-10-24 23:03 ` Stefano Stabellini
2024-11-04 16:17 ` Javi Merino
0 siblings, 1 reply; 19+ messages in thread
From: Stefano Stabellini @ 2024-10-24 23:03 UTC (permalink / raw)
To: Javi Merino; +Cc: xen-devel, Doug Goldstein, Stefano Stabellini
On Thu, 24 Oct 2024, Javi Merino wrote:
> Rework the container to use heredocs for readability and use apt-get
> --no-install-recommends to keep the size down. Rename the job to
> debian-12-arm64-* to follow the naming scheme of all the other CI jobs.
>
> This reduces the size of the debian:12-arm64v8 from 2.25GB down to 1.62GB.
Hi Javi, I am a bit confused by the new list of packages. I diff'ed the
old list with the new list, and I appended the difference below. Maybe I
ask how did you manage to make the new list? As an example, are bcc and
bin86 necessary?
acpica-tools
+bcc
+bin86
bison
build-essential
busybox-static
-ca-certificates
checkpolicy
+clang
cpio
curl
device-tree-compiler
expect
-file
flex
-git-core
-golang-go
-libbz2-dev
-libext2fs-dev
+git
+libaio-dev
libfdt-dev
+libfindlib-ocaml-dev
libglib2.0-dev
liblzma-dev
-liblzo2-dev
+libncurses5-dev
+libnl-3-dev
libpixman-1-dev
libyajl-dev
-libzstd-dev
-ocaml-findlib
+markdown
+nasm
ocaml-nox
+pandoc
pkg-config
python3-dev
python3-setuptools
+transfig
u-boot-qemu
u-boot-tools
uuid-dev
wget
+zlib1g-dev
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [XEN PATCH v1 3/6] CI: Refresh the Debian 12 arm64 container
2024-10-24 23:03 ` Stefano Stabellini
@ 2024-11-04 16:17 ` Javi Merino
0 siblings, 0 replies; 19+ messages in thread
From: Javi Merino @ 2024-11-04 16:17 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: xen-devel, Doug Goldstein
On Thu, Oct 24, 2024 at 04:03:56PM -0700, Stefano Stabellini wrote:
> On Thu, 24 Oct 2024, Javi Merino wrote:
> > Rework the container to use heredocs for readability and use apt-get
> > --no-install-recommends to keep the size down. Rename the job to
> > debian-12-arm64-* to follow the naming scheme of all the other CI jobs.
> >
> > This reduces the size of the debian:12-arm64v8 from 2.25GB down to 1.62GB.
>
> Hi Javi, I am a bit confused by the new list of packages. I diff'ed the
> old list with the new list, and I appended the difference below. Maybe I
> ask how did you manage to make the new list?
I started from scratch. I dropped everything and only added the
packages that were needed.
> As an example, are bcc and
> bin86 necessary?
They are not, that's why I dropped them. The list you generated shows
as "+" the things that were in the old container and "-" are the ones
in the new container that were not in the old container.
If this helps, for the next series I will add a diff of the packages
added and removed.
Cheers,
Javi
> acpica-tools
> +bcc
> +bin86
> bison
> build-essential
> busybox-static
> -ca-certificates
> checkpolicy
> +clang
> cpio
> curl
> device-tree-compiler
> expect
> -file
> flex
> -git-core
> -golang-go
> -libbz2-dev
> -libext2fs-dev
> +git
> +libaio-dev
> libfdt-dev
> +libfindlib-ocaml-dev
> libglib2.0-dev
> liblzma-dev
> -liblzo2-dev
> +libncurses5-dev
> +libnl-3-dev
> libpixman-1-dev
> libyajl-dev
> -libzstd-dev
> -ocaml-findlib
> +markdown
> +nasm
> ocaml-nox
> +pandoc
> pkg-config
> python3-dev
> python3-setuptools
> +transfig
> u-boot-qemu
> u-boot-tools
> uuid-dev
> wget
> +zlib1g-dev
^ permalink raw reply [flat|nested] 19+ messages in thread
* [XEN PATCH v1 4/6] CI: Refresh the Debian 12 arm32 cross compile container
2024-10-24 10:04 [XEN PATCH v1 0/6] automation: Refresh the remaining Debian containers Javi Merino
` (2 preceding siblings ...)
2024-10-24 10:04 ` [XEN PATCH v1 3/6] CI: Refresh the Debian 12 arm64 container Javi Merino
@ 2024-10-24 10:04 ` Javi Merino
2024-10-24 11:51 ` Anthony PERARD
2024-10-24 10:04 ` [XEN PATCH v1 5/6] CI: Refresh the Debian 12 cppcheck container Javi Merino
2024-10-24 10:04 ` [XEN PATCH v1 6/6] CI: Run the builds and tests that use the Debian 12 containers as a normal user Javi Merino
5 siblings, 1 reply; 19+ messages in thread
From: Javi Merino @ 2024-10-24 10:04 UTC (permalink / raw)
To: xen-devel; +Cc: Javi Merino, Doug Goldstein, Stefano Stabellini
Rework the container to run as non-root, use heredocs for readability and use apt-get
--no-install-recommends to keep the size down. Rename the CI jobs to
debian-12-arm32-<variant> to follow the naming scheme of all the other
CI jobs.
Signed-off-by: Javi Merino <javi.merino@cloud.com>
---
.../debian/12-arm64v8-arm32-gcc.dockerfile | 28 +++++++++++++++++++
.../bookworm-arm64v8-arm32-gcc.dockerfile | 24 ----------------
automation/gitlab-ci/build.yaml | 20 ++++++-------
automation/gitlab-ci/test.yaml | 14 +++++-----
automation/scripts/containerize | 2 +-
5 files changed, 46 insertions(+), 42 deletions(-)
create mode 100644 automation/build/debian/12-arm64v8-arm32-gcc.dockerfile
delete mode 100644 automation/build/debian/bookworm-arm64v8-arm32-gcc.dockerfile
diff --git a/automation/build/debian/12-arm64v8-arm32-gcc.dockerfile b/automation/build/debian/12-arm64v8-arm32-gcc.dockerfile
new file mode 100644
index 000000000000..bdc935706bfa
--- /dev/null
+++ b/automation/build/debian/12-arm64v8-arm32-gcc.dockerfile
@@ -0,0 +1,28 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/arm64/v8 debian:bookworm
+LABEL maintainer.name="The Xen Project" \
+ maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV CROSS_COMPILE /usr/bin/arm-linux-gnueabihf-
+
+RUN <<EOF
+#!/bin/bash
+ set -eu
+
+ useradd --create-home user
+
+ apt-get update
+ DEPS=(
+ bison
+ build-essential
+ flex
+ gcc-arm-linux-gnueabihf
+ )
+ apt-get --yes --no-install-recommends install "${DEPS[@]}"
+
+ rm -rf /var/lib/apt/lists*
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/build/debian/bookworm-arm64v8-arm32-gcc.dockerfile b/automation/build/debian/bookworm-arm64v8-arm32-gcc.dockerfile
deleted file mode 100644
index 95b3f0428372..000000000000
--- a/automation/build/debian/bookworm-arm64v8-arm32-gcc.dockerfile
+++ /dev/null
@@ -1,24 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/arm64/v8 debian:bookworm
-LABEL maintainer.name="The Xen Project" \
- maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
-ENV CROSS_COMPILE /usr/bin/arm-linux-gnueabihf-
-
-RUN mkdir /build
-WORKDIR /build
-
-# build depends
-RUN apt-get update && \
- apt-get --quiet --yes install \
- build-essential \
- flex \
- bison \
- git \
- gcc-arm-linux-gnueabihf \
- && \
- apt-get autoremove -y && \
- apt-get clean && \
- rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index c687298cc924..380021d49f12 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -367,39 +367,39 @@ debian-12-riscv64-gcc-debug:
# Arm32 cross-build
-debian-bookworm-gcc-arm32:
+debian-12-arm32-gcc:
extends: .gcc-arm32-cross-build
variables:
- CONTAINER: debian:bookworm-arm64v8-arm32-gcc
+ CONTAINER: debian:12-arm64v8-arm32-gcc
HYPERVISOR_ONLY: y
-debian-bookworm-gcc-arm32-debug:
+debian-12-arm32-gcc-debug:
extends: .gcc-arm32-cross-build-debug
variables:
- CONTAINER: debian:bookworm-arm64v8-arm32-gcc
+ CONTAINER: debian:12-arm64v8-arm32-gcc
HYPERVISOR_ONLY: y
-debian-bookworm-gcc-arm32-randconfig:
+debian-12-arm32-gcc-randconfig:
extends: .gcc-arm32-cross-build
variables:
- CONTAINER: debian:bookworm-arm64v8-arm32-gcc
+ CONTAINER: debian:12-arm64v8-arm32-gcc
HYPERVISOR_ONLY: y
RANDCONFIG: y
-debian-bookworm-gcc-arm32-debug-staticmem:
+debian-12-arm32-gcc-debug-staticmem:
extends: .gcc-arm32-cross-build-debug
variables:
- CONTAINER: debian:bookworm-arm64v8-arm32-gcc
+ CONTAINER: debian:12-arm64v8-arm32-gcc
HYPERVISOR_ONLY: y
EXTRA_XEN_CONFIG: |
CONFIG_EXPERT=y
CONFIG_UNSUPPORTED=y
CONFIG_STATIC_MEMORY=y
-debian-bookworm-gcc-arm32-debug-earlyprintk:
+debian-12-arm32-gcc-debug-earlyprintk:
extends: .gcc-arm32-cross-build-debug
variables:
- CONTAINER: debian:bookworm-arm64v8-arm32-gcc
+ CONTAINER: debian:12-arm64v8-arm32-gcc
HYPERVISOR_ONLY: y
EXTRA_XEN_CONFIG: |
CONFIG_EARLY_UART_CHOICE_PL011=y
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 248281f47942..42baa82fe36f 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -398,7 +398,7 @@ qemu-smoke-dom0less-arm32-gcc:
- ./automation/scripts/qemu-smoke-dom0less-arm32.sh 2>&1 | tee ${LOGFILE}
needs:
- *arm32-test-needs
- - debian-bookworm-gcc-arm32
+ - debian-12-arm32-gcc
qemu-smoke-dom0less-arm32-gcc-debug:
extends: .qemu-arm32
@@ -406,7 +406,7 @@ qemu-smoke-dom0less-arm32-gcc-debug:
- ./automation/scripts/qemu-smoke-dom0less-arm32.sh 2>&1 | tee ${LOGFILE}
needs:
- *arm32-test-needs
- - debian-bookworm-gcc-arm32-debug
+ - debian-12-arm32-gcc-debug
qemu-smoke-dom0less-arm32-gcc-debug-staticmem:
extends: .qemu-arm32
@@ -414,7 +414,7 @@ qemu-smoke-dom0less-arm32-gcc-debug-staticmem:
- ./automation/scripts/qemu-smoke-dom0less-arm32.sh static-mem 2>&1 | tee ${LOGFILE}
needs:
- *arm32-test-needs
- - debian-bookworm-gcc-arm32-debug-staticmem
+ - debian-12-arm32-gcc-debug-staticmem
qemu-smoke-dom0less-arm32-gcc-debug-gzip:
extends: .qemu-arm32
@@ -422,7 +422,7 @@ qemu-smoke-dom0less-arm32-gcc-debug-gzip:
- ./automation/scripts/qemu-smoke-dom0less-arm32.sh gzip 2>&1 | tee ${LOGFILE}
needs:
- *arm32-test-needs
- - debian-bookworm-gcc-arm32-debug
+ - debian-12-arm32-gcc-debug
qemu-smoke-dom0less-arm32-gcc-without-dom0:
extends: .qemu-arm32
@@ -430,7 +430,7 @@ qemu-smoke-dom0less-arm32-gcc-without-dom0:
- ./automation/scripts/qemu-smoke-dom0less-arm32.sh without-dom0 2>&1 | tee ${LOGFILE}
needs:
- *arm32-test-needs
- - debian-bookworm-gcc-arm32
+ - debian-12-arm32-gcc
qemu-smoke-dom0less-arm32-gcc-debug-without-dom0:
extends: .qemu-arm32
@@ -438,7 +438,7 @@ qemu-smoke-dom0less-arm32-gcc-debug-without-dom0:
- ./automation/scripts/qemu-smoke-dom0less-arm32.sh without-dom0 2>&1 | tee ${LOGFILE}
needs:
- *arm32-test-needs
- - debian-bookworm-gcc-arm32-debug
+ - debian-12-arm32-gcc-debug
qemu-smoke-dom0less-arm32-gcc-debug-earlyprintk:
extends: .qemu-arm32
@@ -446,7 +446,7 @@ qemu-smoke-dom0less-arm32-gcc-debug-earlyprintk:
- ./automation/scripts/qemu-smoke-dom0less-arm32.sh earlyprintk 2>&1 | tee ${LOGFILE}
needs:
- *arm32-test-needs
- - debian-bookworm-gcc-arm32-debug-earlyprintk
+ - debian-12-arm32-gcc-debug-earlyprintk
qemu-alpine-x86_64-gcc:
extends: .qemu-x86-64
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index 1b75c8d93724..c9988bfe927d 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -36,7 +36,7 @@ case "_${CONTAINER}" in
_bookworm-x86_64-gcc-ibt) CONTAINER="${BASE}/debian:12-x86_64-gcc-ibt" ;;
_bookworm|_bookworm-x86_64|_) CONTAINER="${BASE}/debian:12-x86_64" ;;
_bookworm-i386|_bookworm-x86_32) CONTAINER="${BASE}/debian:12-x86_32" ;;
- _bookworm-arm64v8-arm32-gcc) CONTAINER="${BASE}/debian:bookworm-arm64v8-arm32-gcc" ;;
+ _bookworm-arm64v8-arm32-gcc) CONTAINER="${BASE}/debian:12-arm64v8-arm32-gcc" ;;
_bookworm-arm64v8) CONTAINER="${BASE}/debian:12-arm64v8" ;;
_bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
_opensuse-leap|_leap) CONTAINER="${BASE}/opensuse:leap-15.6-x86_64" ;;
--
2.45.2
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [XEN PATCH v1 4/6] CI: Refresh the Debian 12 arm32 cross compile container
2024-10-24 10:04 ` [XEN PATCH v1 4/6] CI: Refresh the Debian 12 arm32 cross compile container Javi Merino
@ 2024-10-24 11:51 ` Anthony PERARD
2024-10-24 12:59 ` Javi Merino
0 siblings, 1 reply; 19+ messages in thread
From: Anthony PERARD @ 2024-10-24 11:51 UTC (permalink / raw)
To: Javi Merino; +Cc: xen-devel, Doug Goldstein, Stefano Stabellini
On Thu, Oct 24, 2024 at 11:04:19AM +0100, Javi Merino wrote:
> diff --git a/automation/build/debian/12-arm64v8-arm32-gcc.dockerfile b/automation/build/debian/12-arm64v8-arm32-gcc.dockerfile
> new file mode 100644
> index 000000000000..bdc935706bfa
> --- /dev/null
> +++ b/automation/build/debian/12-arm64v8-arm32-gcc.dockerfile
> @@ -0,0 +1,28 @@
> +# syntax=docker/dockerfile:1
> +FROM --platform=linux/arm64/v8 debian:bookworm
> +LABEL maintainer.name="The Xen Project" \
> + maintainer.email="xen-devel@lists.xenproject.org"
Recent update of other dockerfile was repeating the "LABEL" intruction
for each label, maybe we should continue to do that (which avoid the
backslash at the end of lines).
> +
> +ENV DEBIAN_FRONTEND=noninteractive
> +ENV CROSS_COMPILE /usr/bin/arm-linux-gnueabihf-
While the syntax "ENV VAR value" is supported, it's been discouraged by
docker's doc (https://docs.docker.com/reference/dockerfile/#env). Also,
here we have two ENV instructions with two different syntax, could you
use the = for declaring CROSS_COMPILE as well?
Thanks,
--
Anthony Perard | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [XEN PATCH v1 4/6] CI: Refresh the Debian 12 arm32 cross compile container
2024-10-24 11:51 ` Anthony PERARD
@ 2024-10-24 12:59 ` Javi Merino
0 siblings, 0 replies; 19+ messages in thread
From: Javi Merino @ 2024-10-24 12:59 UTC (permalink / raw)
To: Anthony PERARD; +Cc: xen-devel, Doug Goldstein, Stefano Stabellini
On Thu, Oct 24, 2024 at 11:51:16AM +0000, Anthony PERARD wrote:
> On Thu, Oct 24, 2024 at 11:04:19AM +0100, Javi Merino wrote:
> > diff --git a/automation/build/debian/12-arm64v8-arm32-gcc.dockerfile b/automation/build/debian/12-arm64v8-arm32-gcc.dockerfile
> > new file mode 100644
> > index 000000000000..bdc935706bfa
> > --- /dev/null
> > +++ b/automation/build/debian/12-arm64v8-arm32-gcc.dockerfile
> > @@ -0,0 +1,28 @@
> > +# syntax=docker/dockerfile:1
> > +FROM --platform=linux/arm64/v8 debian:bookworm
> > +LABEL maintainer.name="The Xen Project" \
> > + maintainer.email="xen-devel@lists.xenproject.org"
>
> Recent update of other dockerfile was repeating the "LABEL" intruction
> for each label, maybe we should continue to do that (which avoid the
> backslash at the end of lines).
Sure. I will do that if there's a v2 (or for other dockerfiles).
> > +ENV DEBIAN_FRONTEND=noninteractive
> > +ENV CROSS_COMPILE /usr/bin/arm-linux-gnueabihf-
>
> While the syntax "ENV VAR value" is supported, it's been discouraged by
> docker's doc (https://docs.docker.com/reference/dockerfile/#env). Also,
> here we have two ENV instructions with two different syntax, could you
> use the = for declaring CROSS_COMPILE as well?
I hadn't even noticed that they were using different syntax. I'll use
ENV VAR=value from now on.
Thanks,
Javi
^ permalink raw reply [flat|nested] 19+ messages in thread
* [XEN PATCH v1 5/6] CI: Refresh the Debian 12 cppcheck container
2024-10-24 10:04 [XEN PATCH v1 0/6] automation: Refresh the remaining Debian containers Javi Merino
` (3 preceding siblings ...)
2024-10-24 10:04 ` [XEN PATCH v1 4/6] CI: Refresh the Debian 12 arm32 cross compile container Javi Merino
@ 2024-10-24 10:04 ` Javi Merino
2024-10-24 23:11 ` Stefano Stabellini
2024-10-24 10:04 ` [XEN PATCH v1 6/6] CI: Run the builds and tests that use the Debian 12 containers as a normal user Javi Merino
5 siblings, 1 reply; 19+ messages in thread
From: Javi Merino @ 2024-10-24 10:04 UTC (permalink / raw)
To: xen-devel; +Cc: Javi Merino, Doug Goldstein, Stefano Stabellini, Michal Orzel
Rework the container to build and run cppcheck as a normal user. Use
heredocs for readability and use apt-get --no-install-recommends to
keep the size down.
Changed the libpcre3-dev dependency to libpcre3, as the -dev package
is only needed for building, not for running.
With the cleanups, the size of the container is reduced from 882MB to
782MB.
Signed-off-by: Javi Merino <javi.merino@cloud.com>
---
I tried updating cppcheck to the latest upstream version (2.15.0) but
it reports misra errors like:
xen/common/sched/cpupool.c(0,0):internalError:error:Bailing out from analysis: Checking file failed: Failed to execute addon 'misra' - exitcode is 1
[...]
xen/arch/arm/decode.c(220,10):misra-config:error:Because of missing configuration, misra checking is incomplete. There can be false negatives! Variable 'type' is unknown
The output log xen-cppcheck.txt from job #8164274901 which uses
cppcheck 2.15.0 can be found at:
https://gitlab.com/xen-project/people/javimerino/xen/-/jobs/8164274901/artifacts/browse
Therefore, I'm keeping cppcheck at 2.7, which aligns with what
docs/misra/cppcheck.txt says is the last tested version.
.../debian/12-arm64v8-cppcheck.dockerfile | 80 +++++++++++++++++++
.../build/debian/bookworm-cppcheck.dockerfile | 54 -------------
automation/gitlab-ci/build.yaml | 12 +--
automation/scripts/containerize | 2 +-
4 files changed, 87 insertions(+), 61 deletions(-)
create mode 100644 automation/build/debian/12-arm64v8-cppcheck.dockerfile
delete mode 100644 automation/build/debian/bookworm-cppcheck.dockerfile
diff --git a/automation/build/debian/12-arm64v8-cppcheck.dockerfile b/automation/build/debian/12-arm64v8-cppcheck.dockerfile
new file mode 100644
index 000000000000..a839d4c57bda
--- /dev/null
+++ b/automation/build/debian/12-arm64v8-cppcheck.dockerfile
@@ -0,0 +1,80 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/arm64/v8 debian:bookworm AS builder
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV CPPCHECK_VERSION=2.7
+
+# dependencies for cppcheck build
+RUN <<EOF
+#!/bin/bash
+ set -eu
+
+ apt-get update
+ DEPS=(
+ build-essential
+ ca-certificates
+ curl
+ libpcre3-dev
+ python-is-python3
+ )
+
+ apt-get -y --no-install-recommends install "${DEPS[@]}"
+
+ rm -rf /var/lib/apt/lists*
+EOF
+
+RUN useradd --home /build --create-home user
+
+WORKDIR /build
+USER user
+
+# cppcheck release build (see cppcheck readme.md)
+RUN <<EOF
+#!/bin/bash
+ set -eu
+
+ curl -fsSL https://github.com/danmar/cppcheck/archive/"$CPPCHECK_VERSION".tar.gz | tar xvz
+ cd cppcheck-"$CPPCHECK_VERSION"
+
+ MAKE_OPTS=(
+ MATCHCOMPILER=yes
+ DESTDIR=/build/out
+ FILESDIR="/usr/share/cppcheck"
+ HAVE_RULES=yes CXXFLAGS="-O2 -DNDEBUG -Wall -Wno-sign-compare -Wno-unused-function"
+ )
+ make install -j$(nproc) "${MAKE_OPTS[@]}"
+EOF
+
+FROM --platform=linux/arm64/v8 debian:bookworm
+COPY --from=builder /build/out/usr/bin/cppcheck /usr/bin/cppcheck
+COPY --from=builder /build/out/usr/share/cppcheck /usr/share/cppcheck
+
+LABEL maintainer.name="The Xen Project" \
+ maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV USER root
+
+# dependencies for cppcheck analysis including Xen-only build/cross-build
+RUN <<EOF
+#!/bin/bash
+ set -eu
+
+ useradd --create-home user
+
+ apt-get update
+ DEPS=(
+ bison
+ build-essential
+ python-is-python3
+ libpcre3
+ flex
+ gcc-arm-linux-gnueabihf
+ gcc-x86-64-linux-gnu
+ )
+ apt-get --yes --no-install-recommends install "${DEPS[@]}"
+ rm -rf /var/lib/apt/lists*
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/build/debian/bookworm-cppcheck.dockerfile b/automation/build/debian/bookworm-cppcheck.dockerfile
deleted file mode 100644
index fe4cd4a1aaab..000000000000
--- a/automation/build/debian/bookworm-cppcheck.dockerfile
+++ /dev/null
@@ -1,54 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/arm64/v8 debian:bookworm AS builder
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV CPPCHECK_VERSION=2.7
-ENV USER root
-
-# dependencies for cppcheck build
-RUN apt-get update && \
- apt-get --quiet --yes install \
- curl \
- build-essential \
- python-is-python3 \
- libpcre3-dev
-
-RUN mkdir /build
-WORKDIR /build
-
-# cppcheck release build (see cppcheck readme.md)
-RUN curl -fsSLO https://github.com/danmar/cppcheck/archive/"$CPPCHECK_VERSION".tar.gz && \
- tar xvzf "$CPPCHECK_VERSION".tar.gz && \
- cd cppcheck-"$CPPCHECK_VERSION" && \
- make install -j$(nproc) \
- MATCHCOMPILER=yes \
- FILESDIR=/usr/share/cppcheck \
- HAVE_RULES=yes CXXFLAGS="-O2 -DNDEBUG -Wall -Wno-sign-compare -Wno-unused-function"
-
-FROM --platform=linux/arm64/v8 debian:bookworm
-COPY --from=builder /usr/bin/cppcheck /usr/bin/cppcheck
-COPY --from=builder /usr/share/cppcheck /usr/share/cppcheck
-
-LABEL maintainer.name="The Xen Project" \
- maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-# dependencies for cppcheck analysis including Xen-only build/cross-build
-RUN apt-get update && \
- apt-get --quiet --yes install \
- build-essential \
- python-is-python3 \
- libpcre3-dev \
- flex \
- bison \
- gcc-arm-linux-gnueabihf \
- gcc-x86-64-linux-gnu \
- && \
- apt-get autoremove -y && \
- apt-get clean && \
- rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 380021d49f12..c57882b8846e 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -488,26 +488,26 @@ yocto-qemux86-64:
# Cppcheck analysis jobs
-debian-bookworm-gcc-cppcheck:
+debian-12-x86_64-gcc-cppcheck:
extends: .gcc-x86-64-cross-build
variables:
- CONTAINER: debian:bookworm-cppcheck
+ CONTAINER: debian:12-arm64v8-cppcheck
CROSS_COMPILE: /usr/bin/x86_64-linux-gnu-
CPPCHECK: y
HYPERVISOR_ONLY: y
-debian-bookworm-gcc-arm32-cppcheck:
+debian-12-arm32-gcc-cppcheck:
extends: .gcc-arm32-cross-build
variables:
- CONTAINER: debian:bookworm-cppcheck
+ CONTAINER: debian:12-arm64v8-cppcheck
CROSS_COMPILE: /usr/bin/arm-linux-gnueabihf-
CPPCHECK: y
HYPERVISOR_ONLY: y
-debian-bookworm-gcc-arm64-cppcheck:
+debian-12-arm64-gcc-cppcheck:
extends: .gcc-arm64-build
variables:
- CONTAINER: debian:bookworm-cppcheck
+ CONTAINER: debian:12-arm64v8-cppcheck
CPPCHECK: y
HYPERVISOR_ONLY: y
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index c9988bfe927d..e5502c81a759 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -38,7 +38,7 @@ case "_${CONTAINER}" in
_bookworm-i386|_bookworm-x86_32) CONTAINER="${BASE}/debian:12-x86_32" ;;
_bookworm-arm64v8-arm32-gcc) CONTAINER="${BASE}/debian:12-arm64v8-arm32-gcc" ;;
_bookworm-arm64v8) CONTAINER="${BASE}/debian:12-arm64v8" ;;
- _bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
+ _bookworm-cppcheck) CONTAINER="${BASE}/debian:12-arm64v8-cppcheck" ;;
_opensuse-leap|_leap) CONTAINER="${BASE}/opensuse:leap-15.6-x86_64" ;;
_opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
_xenial) CONTAINER="${BASE}/ubuntu:16.04-x86_64" ;;
--
2.45.2
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [XEN PATCH v1 5/6] CI: Refresh the Debian 12 cppcheck container
2024-10-24 10:04 ` [XEN PATCH v1 5/6] CI: Refresh the Debian 12 cppcheck container Javi Merino
@ 2024-10-24 23:11 ` Stefano Stabellini
0 siblings, 0 replies; 19+ messages in thread
From: Stefano Stabellini @ 2024-10-24 23:11 UTC (permalink / raw)
To: Javi Merino; +Cc: xen-devel, Doug Goldstein, Stefano Stabellini, Michal Orzel
> heredocs for readability and use apt-get --no-install-recommends to
> keep the size down.
>
> Changed the libpcre3-dev dependency to libpcre3, as the -dev package
> is only needed for building, not for running.
>
> With the cleanups, the size of the container is reduced from 882MB to
> 782MB.
>
> Signed-off-by: Javi Merino <javi.merino@cloud.com>
Assuming everything works as expected:
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> ---
>
> I tried updating cppcheck to the latest upstream version (2.15.0) but
> it reports misra errors like:
>
> xen/common/sched/cpupool.c(0,0):internalError:error:Bailing out from analysis: Checking file failed: Failed to execute addon 'misra' - exitcode is 1
> [...]
> xen/arch/arm/decode.c(220,10):misra-config:error:Because of missing configuration, misra checking is incomplete. There can be false negatives! Variable 'type' is unknown
>
> The output log xen-cppcheck.txt from job #8164274901 which uses
> cppcheck 2.15.0 can be found at:
> https://gitlab.com/xen-project/people/javimerino/xen/-/jobs/8164274901/artifacts/browse
>
> Therefore, I'm keeping cppcheck at 2.7, which aligns with what
> docs/misra/cppcheck.txt says is the last tested version.
Yes, better to do the upgrade separately
> .../debian/12-arm64v8-cppcheck.dockerfile | 80 +++++++++++++++++++
> .../build/debian/bookworm-cppcheck.dockerfile | 54 -------------
> automation/gitlab-ci/build.yaml | 12 +--
> automation/scripts/containerize | 2 +-
> 4 files changed, 87 insertions(+), 61 deletions(-)
> create mode 100644 automation/build/debian/12-arm64v8-cppcheck.dockerfile
> delete mode 100644 automation/build/debian/bookworm-cppcheck.dockerfile
>
> diff --git a/automation/build/debian/12-arm64v8-cppcheck.dockerfile b/automation/build/debian/12-arm64v8-cppcheck.dockerfile
> new file mode 100644
> index 000000000000..a839d4c57bda
> --- /dev/null
> +++ b/automation/build/debian/12-arm64v8-cppcheck.dockerfile
> @@ -0,0 +1,80 @@
> +# syntax=docker/dockerfile:1
> +FROM --platform=linux/arm64/v8 debian:bookworm AS builder
> +
> +ENV DEBIAN_FRONTEND=noninteractive
> +ENV CPPCHECK_VERSION=2.7
> +
> +# dependencies for cppcheck build
> +RUN <<EOF
> +#!/bin/bash
> + set -eu
> +
> + apt-get update
> + DEPS=(
> + build-essential
> + ca-certificates
> + curl
> + libpcre3-dev
> + python-is-python3
> + )
> +
> + apt-get -y --no-install-recommends install "${DEPS[@]}"
> +
> + rm -rf /var/lib/apt/lists*
> +EOF
> +
> +RUN useradd --home /build --create-home user
> +
> +WORKDIR /build
> +USER user
> +
> +# cppcheck release build (see cppcheck readme.md)
> +RUN <<EOF
> +#!/bin/bash
> + set -eu
> +
> + curl -fsSL https://github.com/danmar/cppcheck/archive/"$CPPCHECK_VERSION".tar.gz | tar xvz
> + cd cppcheck-"$CPPCHECK_VERSION"
> +
> + MAKE_OPTS=(
> + MATCHCOMPILER=yes
> + DESTDIR=/build/out
> + FILESDIR="/usr/share/cppcheck"
> + HAVE_RULES=yes CXXFLAGS="-O2 -DNDEBUG -Wall -Wno-sign-compare -Wno-unused-function"
> + )
> + make install -j$(nproc) "${MAKE_OPTS[@]}"
> +EOF
> +
> +FROM --platform=linux/arm64/v8 debian:bookworm
> +COPY --from=builder /build/out/usr/bin/cppcheck /usr/bin/cppcheck
> +COPY --from=builder /build/out/usr/share/cppcheck /usr/share/cppcheck
> +
> +LABEL maintainer.name="The Xen Project" \
> + maintainer.email="xen-devel@lists.xenproject.org"
> +
> +ENV DEBIAN_FRONTEND=noninteractive
> +ENV USER root
> +
> +# dependencies for cppcheck analysis including Xen-only build/cross-build
> +RUN <<EOF
> +#!/bin/bash
> + set -eu
> +
> + useradd --create-home user
> +
> + apt-get update
> + DEPS=(
> + bison
> + build-essential
> + python-is-python3
> + libpcre3
> + flex
> + gcc-arm-linux-gnueabihf
> + gcc-x86-64-linux-gnu
> + )
> + apt-get --yes --no-install-recommends install "${DEPS[@]}"
> + rm -rf /var/lib/apt/lists*
> +EOF
> +
> +USER user
> +WORKDIR /build
> diff --git a/automation/build/debian/bookworm-cppcheck.dockerfile b/automation/build/debian/bookworm-cppcheck.dockerfile
> deleted file mode 100644
> index fe4cd4a1aaab..000000000000
> --- a/automation/build/debian/bookworm-cppcheck.dockerfile
> +++ /dev/null
> @@ -1,54 +0,0 @@
> -# syntax=docker/dockerfile:1
> -FROM --platform=linux/arm64/v8 debian:bookworm AS builder
> -
> -ENV DEBIAN_FRONTEND=noninteractive
> -ENV CPPCHECK_VERSION=2.7
> -ENV USER root
> -
> -# dependencies for cppcheck build
> -RUN apt-get update && \
> - apt-get --quiet --yes install \
> - curl \
> - build-essential \
> - python-is-python3 \
> - libpcre3-dev
> -
> -RUN mkdir /build
> -WORKDIR /build
> -
> -# cppcheck release build (see cppcheck readme.md)
> -RUN curl -fsSLO https://github.com/danmar/cppcheck/archive/"$CPPCHECK_VERSION".tar.gz && \
> - tar xvzf "$CPPCHECK_VERSION".tar.gz && \
> - cd cppcheck-"$CPPCHECK_VERSION" && \
> - make install -j$(nproc) \
> - MATCHCOMPILER=yes \
> - FILESDIR=/usr/share/cppcheck \
> - HAVE_RULES=yes CXXFLAGS="-O2 -DNDEBUG -Wall -Wno-sign-compare -Wno-unused-function"
> -
> -FROM --platform=linux/arm64/v8 debian:bookworm
> -COPY --from=builder /usr/bin/cppcheck /usr/bin/cppcheck
> -COPY --from=builder /usr/share/cppcheck /usr/share/cppcheck
> -
> -LABEL maintainer.name="The Xen Project" \
> - maintainer.email="xen-devel@lists.xenproject.org"
> -
> -ENV DEBIAN_FRONTEND=noninteractive
> -ENV USER root
> -
> -RUN mkdir /build
> -WORKDIR /build
> -
> -# dependencies for cppcheck analysis including Xen-only build/cross-build
> -RUN apt-get update && \
> - apt-get --quiet --yes install \
> - build-essential \
> - python-is-python3 \
> - libpcre3-dev \
> - flex \
> - bison \
> - gcc-arm-linux-gnueabihf \
> - gcc-x86-64-linux-gnu \
> - && \
> - apt-get autoremove -y && \
> - apt-get clean && \
> - rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> index 380021d49f12..c57882b8846e 100644
> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -488,26 +488,26 @@ yocto-qemux86-64:
>
> # Cppcheck analysis jobs
>
> -debian-bookworm-gcc-cppcheck:
> +debian-12-x86_64-gcc-cppcheck:
> extends: .gcc-x86-64-cross-build
> variables:
> - CONTAINER: debian:bookworm-cppcheck
> + CONTAINER: debian:12-arm64v8-cppcheck
> CROSS_COMPILE: /usr/bin/x86_64-linux-gnu-
> CPPCHECK: y
> HYPERVISOR_ONLY: y
>
> -debian-bookworm-gcc-arm32-cppcheck:
> +debian-12-arm32-gcc-cppcheck:
> extends: .gcc-arm32-cross-build
> variables:
> - CONTAINER: debian:bookworm-cppcheck
> + CONTAINER: debian:12-arm64v8-cppcheck
> CROSS_COMPILE: /usr/bin/arm-linux-gnueabihf-
> CPPCHECK: y
> HYPERVISOR_ONLY: y
>
> -debian-bookworm-gcc-arm64-cppcheck:
> +debian-12-arm64-gcc-cppcheck:
> extends: .gcc-arm64-build
> variables:
> - CONTAINER: debian:bookworm-cppcheck
> + CONTAINER: debian:12-arm64v8-cppcheck
> CPPCHECK: y
> HYPERVISOR_ONLY: y
>
> diff --git a/automation/scripts/containerize b/automation/scripts/containerize
> index c9988bfe927d..e5502c81a759 100755
> --- a/automation/scripts/containerize
> +++ b/automation/scripts/containerize
> @@ -38,7 +38,7 @@ case "_${CONTAINER}" in
> _bookworm-i386|_bookworm-x86_32) CONTAINER="${BASE}/debian:12-x86_32" ;;
> _bookworm-arm64v8-arm32-gcc) CONTAINER="${BASE}/debian:12-arm64v8-arm32-gcc" ;;
> _bookworm-arm64v8) CONTAINER="${BASE}/debian:12-arm64v8" ;;
> - _bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
> + _bookworm-cppcheck) CONTAINER="${BASE}/debian:12-arm64v8-cppcheck" ;;
> _opensuse-leap|_leap) CONTAINER="${BASE}/opensuse:leap-15.6-x86_64" ;;
> _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
> _xenial) CONTAINER="${BASE}/ubuntu:16.04-x86_64" ;;
> --
> 2.45.2
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* [XEN PATCH v1 6/6] CI: Run the builds and tests that use the Debian 12 containers as a normal user
2024-10-24 10:04 [XEN PATCH v1 0/6] automation: Refresh the remaining Debian containers Javi Merino
` (4 preceding siblings ...)
2024-10-24 10:04 ` [XEN PATCH v1 5/6] CI: Refresh the Debian 12 cppcheck container Javi Merino
@ 2024-10-24 10:04 ` Javi Merino
2024-10-24 23:47 ` Stefano Stabellini
5 siblings, 1 reply; 19+ messages in thread
From: Javi Merino @ 2024-10-24 10:04 UTC (permalink / raw)
To: xen-devel; +Cc: Javi Merino, Doug Goldstein, Stefano Stabellini
Use FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR so that GitLab CI clones xen
using the user in the image, instead of as root[0].
In qemu-smoke-dom0*.sh and qemu-alpine-x86_64.sh, use fakeroot to
create the rootfs images that untar a tarball that create character
devices. cpio replicates the block and character devices, as well as
preserving the uid and gid it sees in the current directory. fakeroot
lets tar think that it is creating block and character devices, and
all files are owned by root, but it is all smokes and mirrors for
cpio.
[0] https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1736
Signed-off-by: Javi Merino <javi.merino@cloud.com>
---
Regarding building the rootfs, I have chosen to use a fakeroot
subshell for the entire process. automation/scripts/qubes-x86-64.sh
takes a different approach, it just uses fakeroot for the tar/cpio
commands. I prefer to do it this way but I am happy to be overridden
if `fakeroot -s ../save tar` and `fakeroot -i ../save cpio` are
preferred.
automation/build/debian/12-arm64v8.dockerfile | 5 ++++-
automation/build/debian/12-x86_64.dockerfile | 5 ++++-
automation/gitlab-ci/test.yaml | 4 ++++
automation/scripts/qemu-alpine-x86_64.sh | 4 +++-
automation/scripts/qemu-smoke-dom0-arm64.sh | 7 +++++--
automation/scripts/qemu-smoke-dom0less-arm64.sh | 5 +++--
6 files changed, 23 insertions(+), 7 deletions(-)
diff --git a/automation/build/debian/12-arm64v8.dockerfile b/automation/build/debian/12-arm64v8.dockerfile
index 4da1b074aedb..c2617956ed77 100644
--- a/automation/build/debian/12-arm64v8.dockerfile
+++ b/automation/build/debian/12-arm64v8.dockerfile
@@ -10,6 +10,8 @@ RUN <<EOF
#!/bin/bash
set -eu
+ useradd --create-home user
+
apt-get update
DEPS=(
# Xen
@@ -53,6 +55,7 @@ RUN <<EOF
curl
device-tree-compiler
expect
+ fakeroot
u-boot-qemu
# for imagebuilder
file
@@ -64,5 +67,5 @@ RUN <<EOF
rm -rf /var/lib/apt/lists*
EOF
-USER root
+USER user
WORKDIR /build
diff --git a/automation/build/debian/12-x86_64.dockerfile b/automation/build/debian/12-x86_64.dockerfile
index e0ca8b7e9c91..98b23ea3eaa4 100644
--- a/automation/build/debian/12-x86_64.dockerfile
+++ b/automation/build/debian/12-x86_64.dockerfile
@@ -10,6 +10,8 @@ RUN <<EOF
#!/bin/bash
set -eu
+ useradd --create-home user
+
apt-get update
DEPS=(
# Xen
@@ -54,6 +56,7 @@ RUN <<EOF
# for qemu-alpine-x86_64-gcc
busybox-static
cpio
+ fakeroot
# For *-efi jobs
ovmf
@@ -64,5 +67,5 @@ RUN <<EOF
rm -rf /var/lib/apt/lists*
EOF
-USER root
+USER user
WORKDIR /build
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 42baa82fe36f..71f2beb68c4f 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -1,6 +1,10 @@
.test-jobs-common:
stage: test
image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+ variables:
+ # Clone xen as the user in the docker images, not root
+ # See https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1736
+ FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR: true
.arm64-test-needs: &arm64-test-needs
- alpine-3.18-arm64-rootfs-export
diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh
index 1ff689b577e3..2660403ab2b8 100755
--- a/automation/scripts/qemu-alpine-x86_64.sh
+++ b/automation/scripts/qemu-alpine-x86_64.sh
@@ -29,6 +29,7 @@ find . | cpio --create --format='newc' | gzip > ../initrd.cpio.gz
cd ..
# initrd.tar.gz is Dom0 rootfs
+fakeroot <<EOF
mkdir -p rootfs
cd rootfs
tar xvzf ../initrd.tar.gz
@@ -63,7 +64,8 @@ chmod +x etc/local.d/xen.start
echo "rc_verbose=yes" >> etc/rc.conf
# rebuild Dom0 rootfs
find . |cpio -H newc -o|gzip > ../xen-rootfs.cpio.gz
-cd ../..
+EOF
+cd ..
cat >> binaries/pxelinux.0 << EOF
#!ipxe
diff --git a/automation/scripts/qemu-smoke-dom0-arm64.sh b/automation/scripts/qemu-smoke-dom0-arm64.sh
index 4d22a124df11..8f9bfd620ee3 100755
--- a/automation/scripts/qemu-smoke-dom0-arm64.sh
+++ b/automation/scripts/qemu-smoke-dom0-arm64.sh
@@ -27,8 +27,10 @@ cd initrd
find . | cpio --create --format='newc' | gzip > ../initrd.cpio.gz
cd ..
-mkdir -p rootfs
+fakeroot <<EOF
+mkdir rootfs
cd rootfs
+
tar xvzf ../initrd.tar.gz
mkdir proc
mkdir run
@@ -58,7 +60,8 @@ xl create -c /root/test.cfg
chmod +x etc/local.d/xen.start
echo "rc_verbose=yes" >> etc/rc.conf
find . |cpio -H newc -o|gzip > ../xen-rootfs.cpio.gz
-cd ../..
+EOF
+cd ..
# XXX QEMU looks for "efi-virtio.rom" even if it is unneeded
curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
diff --git a/automation/scripts/qemu-smoke-dom0less-arm64.sh b/automation/scripts/qemu-smoke-dom0less-arm64.sh
index 83e1866ca6c5..2d60f22d68b7 100755
--- a/automation/scripts/qemu-smoke-dom0less-arm64.sh
+++ b/automation/scripts/qemu-smoke-dom0less-arm64.sh
@@ -112,7 +112,8 @@ find . | cpio --create --format='newc' | gzip > ../binaries/initrd
cd ..
# DOM0 rootfs
-mkdir -p rootfs
+fakeroot <<EOF
+mkdir rootfs
cd rootfs
tar xzf ../binaries/initrd.tar.gz
mkdir proc
@@ -141,7 +142,7 @@ ${dom0_check}
chmod +x etc/local.d/xen.start
echo "rc_verbose=yes" >> etc/rc.conf
find . | cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz
-cd ..
+EOF
# ImageBuilder
echo 'MEMORY_START="0x40000000"
--
2.45.2
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [XEN PATCH v1 6/6] CI: Run the builds and tests that use the Debian 12 containers as a normal user
2024-10-24 10:04 ` [XEN PATCH v1 6/6] CI: Run the builds and tests that use the Debian 12 containers as a normal user Javi Merino
@ 2024-10-24 23:47 ` Stefano Stabellini
2024-10-29 17:18 ` Javi Merino
0 siblings, 1 reply; 19+ messages in thread
From: Stefano Stabellini @ 2024-10-24 23:47 UTC (permalink / raw)
To: Javi Merino; +Cc: xen-devel, Doug Goldstein, Stefano Stabellini
On Thu, 24 Oct 2024, Javi Merino wrote:
> Use FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR so that GitLab CI clones xen
> using the user in the image, instead of as root[0].
>
> In qemu-smoke-dom0*.sh and qemu-alpine-x86_64.sh, use fakeroot to
> create the rootfs images that untar a tarball that create character
> devices. cpio replicates the block and character devices, as well as
> preserving the uid and gid it sees in the current directory. fakeroot
> lets tar think that it is creating block and character devices, and
> all files are owned by root, but it is all smokes and mirrors for
> cpio.
>
> [0] https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1736
>
> Signed-off-by: Javi Merino <javi.merino@cloud.com>
> ---
>
> Regarding building the rootfs, I have chosen to use a fakeroot
> subshell for the entire process. automation/scripts/qubes-x86-64.sh
> takes a different approach, it just uses fakeroot for the tar/cpio
> commands. I prefer to do it this way but I am happy to be overridden
> if `fakeroot -s ../save tar` and `fakeroot -i ../save cpio` are
> preferred.
>
> automation/build/debian/12-arm64v8.dockerfile | 5 ++++-
> automation/build/debian/12-x86_64.dockerfile | 5 ++++-
> automation/gitlab-ci/test.yaml | 4 ++++
> automation/scripts/qemu-alpine-x86_64.sh | 4 +++-
> automation/scripts/qemu-smoke-dom0-arm64.sh | 7 +++++--
> automation/scripts/qemu-smoke-dom0less-arm64.sh | 5 +++--
> 6 files changed, 23 insertions(+), 7 deletions(-)
>
> diff --git a/automation/build/debian/12-arm64v8.dockerfile b/automation/build/debian/12-arm64v8.dockerfile
> index 4da1b074aedb..c2617956ed77 100644
> --- a/automation/build/debian/12-arm64v8.dockerfile
> +++ b/automation/build/debian/12-arm64v8.dockerfile
> @@ -10,6 +10,8 @@ RUN <<EOF
> #!/bin/bash
> set -eu
>
> + useradd --create-home user
> +
> apt-get update
> DEPS=(
> # Xen
> @@ -53,6 +55,7 @@ RUN <<EOF
> curl
> device-tree-compiler
> expect
> + fakeroot
> u-boot-qemu
> # for imagebuilder
> file
> @@ -64,5 +67,5 @@ RUN <<EOF
> rm -rf /var/lib/apt/lists*
> EOF
>
> -USER root
> +USER user
> WORKDIR /build
> diff --git a/automation/build/debian/12-x86_64.dockerfile b/automation/build/debian/12-x86_64.dockerfile
> index e0ca8b7e9c91..98b23ea3eaa4 100644
> --- a/automation/build/debian/12-x86_64.dockerfile
> +++ b/automation/build/debian/12-x86_64.dockerfile
> @@ -10,6 +10,8 @@ RUN <<EOF
> #!/bin/bash
> set -eu
>
> + useradd --create-home user
> +
> apt-get update
> DEPS=(
> # Xen
> @@ -54,6 +56,7 @@ RUN <<EOF
> # for qemu-alpine-x86_64-gcc
> busybox-static
> cpio
> + fakeroot
>
> # For *-efi jobs
> ovmf
> @@ -64,5 +67,5 @@ RUN <<EOF
> rm -rf /var/lib/apt/lists*
> EOF
>
> -USER root
> +USER user
> WORKDIR /build
This breaks the xilinx hardware jobs both arm and x86 as they
require root inside the container at the moment
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index 42baa82fe36f..71f2beb68c4f 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -1,6 +1,10 @@
> .test-jobs-common:
> stage: test
> image: registry.gitlab.com/xen-project/xen/${CONTAINER}
> + variables:
> + # Clone xen as the user in the docker images, not root
> + # See https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1736
> + FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR: true
>
> .arm64-test-needs: &arm64-test-needs
> - alpine-3.18-arm64-rootfs-export
> diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh
> index 1ff689b577e3..2660403ab2b8 100755
> --- a/automation/scripts/qemu-alpine-x86_64.sh
> +++ b/automation/scripts/qemu-alpine-x86_64.sh
> @@ -29,6 +29,7 @@ find . | cpio --create --format='newc' | gzip > ../initrd.cpio.gz
> cd ..
>
> # initrd.tar.gz is Dom0 rootfs
> +fakeroot <<EOF
> mkdir -p rootfs
> cd rootfs
> tar xvzf ../initrd.tar.gz
> @@ -63,7 +64,8 @@ chmod +x etc/local.d/xen.start
> echo "rc_verbose=yes" >> etc/rc.conf
> # rebuild Dom0 rootfs
> find . |cpio -H newc -o|gzip > ../xen-rootfs.cpio.gz
> -cd ../..
> +EOF
> +cd ..
I admit I am not a fan of this as it makes the script harder to read.
Given that almost everything on this script and similar scripts is
better run as root because it is all about repackaging cpio archivies,
instead I would do this:
diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh
index 2660403ab2..7c0ec01e05 100755
--- a/automation/scripts/qemu-alpine-x86_64.sh
+++ b/automation/scripts/qemu-alpine-x86_64.sh
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/fakeroot
set -ex -o pipefail
Keeping in mind that anyone could push a branch without fakeroot to
their personal tree triggering a gitlab-ci pipeline, the advantage of
using fakeroot would be if we force the container execution envinronment
(gitlab runner) to run containers as user. This is not currently the
configuration we have. As of now, it doesn't bring an advantage.
Given that the gitlab runners are in flux at the moment, and that this
patch cannot work with the xilinx runners, I would ask you to please
hold on on this patch until the gitlab runners are settled (~1 month).
> cat >> binaries/pxelinux.0 << EOF
> #!ipxe
> diff --git a/automation/scripts/qemu-smoke-dom0-arm64.sh b/automation/scripts/qemu-smoke-dom0-arm64.sh
> index 4d22a124df11..8f9bfd620ee3 100755
> --- a/automation/scripts/qemu-smoke-dom0-arm64.sh
> +++ b/automation/scripts/qemu-smoke-dom0-arm64.sh
> @@ -27,8 +27,10 @@ cd initrd
> find . | cpio --create --format='newc' | gzip > ../initrd.cpio.gz
> cd ..
>
> -mkdir -p rootfs
> +fakeroot <<EOF
> +mkdir rootfs
> cd rootfs
> +
> tar xvzf ../initrd.tar.gz
> mkdir proc
> mkdir run
> @@ -58,7 +60,8 @@ xl create -c /root/test.cfg
> chmod +x etc/local.d/xen.start
> echo "rc_verbose=yes" >> etc/rc.conf
> find . |cpio -H newc -o|gzip > ../xen-rootfs.cpio.gz
> -cd ../..
> +EOF
> +cd ..
>
> # XXX QEMU looks for "efi-virtio.rom" even if it is unneeded
> curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
> diff --git a/automation/scripts/qemu-smoke-dom0less-arm64.sh b/automation/scripts/qemu-smoke-dom0less-arm64.sh
> index 83e1866ca6c5..2d60f22d68b7 100755
> --- a/automation/scripts/qemu-smoke-dom0less-arm64.sh
> +++ b/automation/scripts/qemu-smoke-dom0less-arm64.sh
> @@ -112,7 +112,8 @@ find . | cpio --create --format='newc' | gzip > ../binaries/initrd
> cd ..
>
> # DOM0 rootfs
> -mkdir -p rootfs
> +fakeroot <<EOF
> +mkdir rootfs
> cd rootfs
> tar xzf ../binaries/initrd.tar.gz
> mkdir proc
> @@ -141,7 +142,7 @@ ${dom0_check}
> chmod +x etc/local.d/xen.start
> echo "rc_verbose=yes" >> etc/rc.conf
> find . | cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz
> -cd ..
> +EOF
>
> # ImageBuilder
> echo 'MEMORY_START="0x40000000"
> --
> 2.45.2
>
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [XEN PATCH v1 6/6] CI: Run the builds and tests that use the Debian 12 containers as a normal user
2024-10-24 23:47 ` Stefano Stabellini
@ 2024-10-29 17:18 ` Javi Merino
0 siblings, 0 replies; 19+ messages in thread
From: Javi Merino @ 2024-10-29 17:18 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: xen-devel, Doug Goldstein
Hi Stefano,
On Thu, Oct 24, 2024 at 04:47:28PM -0700, Stefano Stabellini wrote:
> On Thu, 24 Oct 2024, Javi Merino wrote:
> > Use FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR so that GitLab CI clones xen
> > using the user in the image, instead of as root[0].
> >
> > In qemu-smoke-dom0*.sh and qemu-alpine-x86_64.sh, use fakeroot to
> > create the rootfs images that untar a tarball that create character
> > devices. cpio replicates the block and character devices, as well as
> > preserving the uid and gid it sees in the current directory. fakeroot
> > lets tar think that it is creating block and character devices, and
> > all files are owned by root, but it is all smokes and mirrors for
> > cpio.
> >
> > [0] https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1736
> >
> > Signed-off-by: Javi Merino <javi.merino@cloud.com>
> > ---
> >
> > Regarding building the rootfs, I have chosen to use a fakeroot
> > subshell for the entire process. automation/scripts/qubes-x86-64.sh
> > takes a different approach, it just uses fakeroot for the tar/cpio
> > commands. I prefer to do it this way but I am happy to be overridden
> > if `fakeroot -s ../save tar` and `fakeroot -i ../save cpio` are
> > preferred.
> >
> > automation/build/debian/12-arm64v8.dockerfile | 5 ++++-
> > automation/build/debian/12-x86_64.dockerfile | 5 ++++-
> > automation/gitlab-ci/test.yaml | 4 ++++
> > automation/scripts/qemu-alpine-x86_64.sh | 4 +++-
> > automation/scripts/qemu-smoke-dom0-arm64.sh | 7 +++++--
> > automation/scripts/qemu-smoke-dom0less-arm64.sh | 5 +++--
> > 6 files changed, 23 insertions(+), 7 deletions(-)
> >
> > diff --git a/automation/build/debian/12-arm64v8.dockerfile b/automation/build/debian/12-arm64v8.dockerfile
> > index 4da1b074aedb..c2617956ed77 100644
> > --- a/automation/build/debian/12-arm64v8.dockerfile
> > +++ b/automation/build/debian/12-arm64v8.dockerfile
> > @@ -10,6 +10,8 @@ RUN <<EOF
> > #!/bin/bash
> > set -eu
> >
> > + useradd --create-home user
> > +
> > apt-get update
> > DEPS=(
> > # Xen
> > @@ -53,6 +55,7 @@ RUN <<EOF
> > curl
> > device-tree-compiler
> > expect
> > + fakeroot
> > u-boot-qemu
> > # for imagebuilder
> > file
> > @@ -64,5 +67,5 @@ RUN <<EOF
> > rm -rf /var/lib/apt/lists*
> > EOF
> >
> > -USER root
> > +USER user
> > WORKDIR /build
> > diff --git a/automation/build/debian/12-x86_64.dockerfile b/automation/build/debian/12-x86_64.dockerfile
> > index e0ca8b7e9c91..98b23ea3eaa4 100644
> > --- a/automation/build/debian/12-x86_64.dockerfile
> > +++ b/automation/build/debian/12-x86_64.dockerfile
> > @@ -10,6 +10,8 @@ RUN <<EOF
> > #!/bin/bash
> > set -eu
> >
> > + useradd --create-home user
> > +
> > apt-get update
> > DEPS=(
> > # Xen
> > @@ -54,6 +56,7 @@ RUN <<EOF
> > # for qemu-alpine-x86_64-gcc
> > busybox-static
> > cpio
> > + fakeroot
> >
> > # For *-efi jobs
> > ovmf
> > @@ -64,5 +67,5 @@ RUN <<EOF
> > rm -rf /var/lib/apt/lists*
> > EOF
> >
> > -USER root
> > +USER user
> > WORKDIR /build
>
> This breaks the xilinx hardware jobs both arm and x86 as they
> require root inside the container at the moment
>
>
> > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> > index 42baa82fe36f..71f2beb68c4f 100644
> > --- a/automation/gitlab-ci/test.yaml
> > +++ b/automation/gitlab-ci/test.yaml
> > @@ -1,6 +1,10 @@
> > .test-jobs-common:
> > stage: test
> > image: registry.gitlab.com/xen-project/xen/${CONTAINER}
> > + variables:
> > + # Clone xen as the user in the docker images, not root
> > + # See https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1736
> > + FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR: true
> >
> > .arm64-test-needs: &arm64-test-needs
> > - alpine-3.18-arm64-rootfs-export
> > diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh
> > index 1ff689b577e3..2660403ab2b8 100755
> > --- a/automation/scripts/qemu-alpine-x86_64.sh
> > +++ b/automation/scripts/qemu-alpine-x86_64.sh
> > @@ -29,6 +29,7 @@ find . | cpio --create --format='newc' | gzip > ../initrd.cpio.gz
> > cd ..
> >
> > # initrd.tar.gz is Dom0 rootfs
> > +fakeroot <<EOF
> > mkdir -p rootfs
> > cd rootfs
> > tar xvzf ../initrd.tar.gz
> > @@ -63,7 +64,8 @@ chmod +x etc/local.d/xen.start
> > echo "rc_verbose=yes" >> etc/rc.conf
> > # rebuild Dom0 rootfs
> > find . |cpio -H newc -o|gzip > ../xen-rootfs.cpio.gz
> > -cd ../..
> > +EOF
> > +cd ..
>
> I admit I am not a fan of this as it makes the script harder to read.
> Given that almost everything on this script and similar scripts is
> better run as root because it is all about repackaging cpio archivies,
> instead I would do this:
>
> diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh
> index 2660403ab2..7c0ec01e05 100755
> --- a/automation/scripts/qemu-alpine-x86_64.sh
> +++ b/automation/scripts/qemu-alpine-x86_64.sh
> @@ -1,4 +1,4 @@
> -#!/bin/bash
> +#!/usr/bin/fakeroot
>
> set -ex -o pipefail
Running the entire script as a fakeroot subshell is ugly and not
necessary.
A better fix is what I suggested under the commit message, which is
also what the qubes containers do:
--- a/automation/scripts/qemu-alpine-x86_64.sh
+++ b/automation/scripts/qemu-alpine-x86_64.sh
@@ -31,7 +31,7 @@ cd ..
# initrd.tar.gz is Dom0 rootfs
mkdir -p rootfs
cd rootfs
-tar xvzf ../initrd.tar.gz
+fakeroot -s ../fakeroot-save tar xvzf ../initrd.tar.gz
mkdir proc
mkdir run
mkdir srv
@@ -62,7 +62,7 @@ xl create -c /root/test.cfg
chmod +x etc/local.d/xen.start
echo "rc_verbose=yes" >> etc/rc.conf
# rebuild Dom0 rootfs
-find . |cpio -H newc -o|gzip > ../xen-rootfs.cpio.gz
+find . | fakeroot -i ../fakeroot-save cpio -H newc -o|gzip > ../xen-rootfs.cpio.gz
cd ../..
cat >> binaries/pxelinux.0 << EOF
Similar for the dom0 and dom0less scripts and for the xilinx scripts.
> Keeping in mind that anyone could push a branch without fakeroot to
> their personal tree triggering a gitlab-ci pipeline, the advantage of
> using fakeroot would be if we force the container execution envinronment
> (gitlab runner) to run containers as user. This is not currently the
> configuration we have. As of now, it doesn't bring an advantage.
>
> Given that the gitlab runners are in flux at the moment, and that this
> patch cannot work with the xilinx runners, I would ask you to please
> hold on on this patch until the gitlab runners are settled (~1 month).
Ok, I will hold the patch. I will also fix the xilinx scripts.
Cheers,
Javi
^ permalink raw reply [flat|nested] 19+ messages in thread