[XEN PATCH v3 0/4] automation: Update containers to allow HTTPS access to xenbits

[XEN PATCH v3 0/4] automation: Update containers to allow HTTPS access to xenbits

[Anthony PERARD]

Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-containers-update-v3

v3:
- new patch which remove non-debug x86_32 builds
- don't fix root certificates in jessie containers as those won't be used
  anymore on the main branch.

v2:
- Remove CentOS 7.2
- Remove Debian Jessie test, but update container recipe for the benefit of
  older branches.
- Fix CentOS 7 containner recipe to update all packages. (Fix missing update of
  HTTPS root certificates)

There is work in progress [1] to update urls in our repo to use https, but
those https urls to xenbits don't work in our containers, due to an expired
root certificate. So we need to update those containers.

This series update the dockerfile where just rebuilding the container isn't enough.

Cheers,

Anthony PERARD (4):
  automation: Use EOL tag for Jessie container
  automation: Remove testing on Debian Jessie
  automation: Remove expired root certificates used to be used by let's
    encrypt
  automation: Remove non-debug x86_32 build jobs

 .../build/debian/jessie-i386.dockerfile       |  2 +-
 automation/build/debian/jessie.dockerfile     |  2 +-
 automation/build/ubuntu/trusty.dockerfile     |  5 ++
 automation/gitlab-ci/build.yaml               | 60 -------------------
 4 files changed, 7 insertions(+), 62 deletions(-)

-- 
Anthony PERARD

[XEN PATCH v3 1/4] automation: Use EOL tag for Jessie container

[Anthony PERARD]

As Jessie is EOL, the official tag isn't supported anymore. Also, the
GPG key for the packages on the repository on the official image are
expired and it isn't possible to update or install packages.

But we can use the image from "debian/eol" tag which use repositories
from archive.debian.org and have workaround to ignore the validity
date of the keys.

There isn't a dedicated i386 tag for jessie, but we can ask docker to
pull the i386 image of the "debial/eol:jessie" tag.

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---

Notes:
    v2:
    - new patch, this replace "automation: Ignore package authentification issue in Jessie container"
    
    workaround I've seen in the debian/eol:jessie:
        'Acquire::Check-Valid-Until "false";' in /etc/apt/apt.conf.d/
    And a script to replace the "gpgv" binary used by apt, which check that
    the only issue with a signature is that the key has expired.

 automation/build/debian/jessie-i386.dockerfile | 2 +-
 automation/build/debian/jessie.dockerfile      | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/automation/build/debian/jessie-i386.dockerfile b/automation/build/debian/jessie-i386.dockerfile
index b152d658d3..c9ac15a3ff 100644
--- a/automation/build/debian/jessie-i386.dockerfile
+++ b/automation/build/debian/jessie-i386.dockerfile
@@ -1,4 +1,4 @@
-FROM i386/debian:jessie
+FROM --platform=linux/i386 debian/eol:jessie
 LABEL maintainer.name="The Xen Project" \
       maintainer.email="xen-devel@lists.xenproject.org"
 
diff --git a/automation/build/debian/jessie.dockerfile b/automation/build/debian/jessie.dockerfile
index 1e33e635d2..63b2c1e5b7 100644
--- a/automation/build/debian/jessie.dockerfile
+++ b/automation/build/debian/jessie.dockerfile
@@ -1,4 +1,4 @@
-FROM debian:jessie
+FROM debian/eol:jessie
 LABEL maintainer.name="The Xen Project" \
       maintainer.email="xen-devel@lists.xenproject.org"
 
-- 
Anthony PERARD

[XEN PATCH v3 2/4] automation: Remove testing on Debian Jessie

[Anthony PERARD]

Jessie as rearch EOL in 2020.

Even if we update the containers, we would still not be able to reach
HTTPS webside with Let's Encrypt certificates and thus would need more
change to the container.

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---

Notes:
    While we remove some x86_32 testing, there's still
    debian-stretch-32-* and debian-unstable-32-*.

 automation/gitlab-ci/build.yaml | 40 ---------------------------------
 1 file changed, 40 deletions(-)

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 22ce1c45e7..2be1b05d5c 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -221,46 +221,6 @@ centos-7-gcc-debug:
   variables:
     CONTAINER: centos:7
 
-debian-jessie-clang:
-  extends: .clang-x86-64-build
-  variables:
-    CONTAINER: debian:jessie
-
-debian-jessie-clang-debug:
-  extends: .clang-x86-64-build-debug
-  variables:
-    CONTAINER: debian:jessie
-
-debian-jessie-gcc:
-  extends: .gcc-x86-64-build
-  variables:
-    CONTAINER: debian:jessie
-
-debian-jessie-gcc-debug:
-  extends: .gcc-x86-64-build-debug
-  variables:
-    CONTAINER: debian:jessie
-
-debian-jessie-32-clang:
-  extends: .clang-x86-32-build
-  variables:
-    CONTAINER: debian:jessie-i386
-
-debian-jessie-32-clang-debug:
-  extends: .clang-x86-32-build-debug
-  variables:
-    CONTAINER: debian:jessie-i386
-
-debian-jessie-32-gcc:
-  extends: .gcc-x86-32-build
-  variables:
-    CONTAINER: debian:jessie-i386
-
-debian-jessie-32-gcc-debug:
-  extends: .gcc-x86-32-build-debug
-  variables:
-    CONTAINER: debian:jessie-i386
-
 debian-stretch-clang:
   extends: .clang-x86-64-build
   variables:
-- 
Anthony PERARD

[XEN PATCH v3 3/4] automation: Remove expired root certificates used to be used by let's encrypt

[Anthony PERARD]

While the Let's Encrypt root certificate ISRG_Root_X1.crt is already
present, openssl seems to still check for the root certificate
DST_Root_CA_X3.crt which has expired. This prevent https connections.

Removing DST_Root_CA_X3 fix the issue.

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---

Notes:
    v3:
    - remove change to Debian Jessie containers, as we won't use them with
      HTTPS urls.
    
    v2:
    - remove unneeded changes to CentOS containers

 automation/build/ubuntu/trusty.dockerfile | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/automation/build/ubuntu/trusty.dockerfile b/automation/build/ubuntu/trusty.dockerfile
index b298a515c6..22e294c20c 100644
--- a/automation/build/ubuntu/trusty.dockerfile
+++ b/automation/build/ubuntu/trusty.dockerfile
@@ -47,3 +47,8 @@ RUN apt-get update && \
         apt-get autoremove -y && \
         apt-get clean && \
         rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
+
+# Remove expired certificate that Let's Encrypt certificates used to relie on.
+# (Not needed anymore)
+RUN sed -i 's#mozilla/DST_Root_CA_X3\.crt#!\0#' /etc/ca-certificates.conf && \
+    update-ca-certificates
-- 
Anthony PERARD

[XEN PATCH v3 4/4] automation: Remove non-debug x86_32 build jobs

[Anthony PERARD]

In the interest of having less jobs, we remove the x86_32 build jobs
that do release build. Debug build is very likely to be enough to find
32bit build issues.

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---

Notes:
    v3:
    - new patch
    
    Proposed here:
        https://lore.kernel.org/r/7ec61260-0fec-25e2-6fa7-ec1d86e8067d@citrix.com

 automation/gitlab-ci/build.yaml | 20 --------------------
 1 file changed, 20 deletions(-)

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 2be1b05d5c..a2891c5937 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -251,21 +251,11 @@ debian-stretch-gcc-debug:
   variables:
     CONTAINER: debian:stretch
 
-debian-stretch-32-clang:
-  extends: .clang-x86-32-build
-  variables:
-    CONTAINER: debian:stretch-i386
-
 debian-stretch-32-clang-debug:
   extends: .clang-x86-32-build-debug
   variables:
     CONTAINER: debian:stretch-i386
 
-debian-stretch-32-gcc:
-  extends: .gcc-x86-32-build
-  variables:
-    CONTAINER: debian:stretch-i386
-
 debian-stretch-32-gcc-debug:
   extends: .gcc-x86-32-build-debug
   variables:
@@ -311,21 +301,11 @@ debian-unstable-gcc-debug-randconfig:
     CONTAINER: debian:unstable
     RANDCONFIG: y
 
-debian-unstable-32-clang:
-  extends: .clang-x86-32-build
-  variables:
-    CONTAINER: debian:unstable-i386
-
 debian-unstable-32-clang-debug:
   extends: .clang-x86-32-build-debug
   variables:
     CONTAINER: debian:unstable-i386
 
-debian-unstable-32-gcc:
-  extends: .gcc-x86-32-build
-  variables:
-    CONTAINER: debian:unstable-i386
-
 debian-unstable-32-gcc-debug:
   extends: .gcc-x86-32-build-debug
   variables:
-- 
Anthony PERARD

Re: [XEN PATCH v3 0/4] automation: Update containers to allow HTTPS access to xenbits

[Andrew Cooper]

On 24/02/2023 5:29 pm, Anthony PERARD wrote:
> Patch series available in this git branch:
> https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-containers-update-v3
>
> v3:
> - new patch which remove non-debug x86_32 builds
> - don't fix root certificates in jessie containers as those won't be used
>   anymore on the main branch.
>
> v2:
> - Remove CentOS 7.2
> - Remove Debian Jessie test, but update container recipe for the benefit of
>   older branches.
> - Fix CentOS 7 containner recipe to update all packages. (Fix missing update of
>   HTTPS root certificates)
>
> There is work in progress [1] to update urls in our repo to use https, but
> those https urls to xenbits don't work in our containers, due to an expired
> root certificate. So we need to update those containers.
>
> This series update the dockerfile where just rebuilding the container isn't enough.

LGTM.

Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>

I'll add this to my commit sweep, and rebuild the remaining containers.

But on that note, I noticed that the debian unstable container was 2.3G
when I last rebuilt it.  Which I think is obscenely large for what we're
doing.

Can we see about switching to slim/tiny container bases?

~Andrew

Re: [XEN PATCH v3 0/4] automation: Update containers to allow HTTPS access to xenbits

[Anthony PERARD]

On Fri, Feb 24, 2023 at 05:37:44PM +0000, Andrew Cooper wrote:
> On 24/02/2023 5:29 pm, Anthony PERARD wrote:
> > Patch series available in this git branch:
> > https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-containers-update-v3
> >
> > v3:
> > - new patch which remove non-debug x86_32 builds
> > - don't fix root certificates in jessie containers as those won't be used
> >   anymore on the main branch.
> >
> > v2:
> > - Remove CentOS 7.2
> > - Remove Debian Jessie test, but update container recipe for the benefit of
> >   older branches.
> > - Fix CentOS 7 containner recipe to update all packages. (Fix missing update of
> >   HTTPS root certificates)
> >
> > There is work in progress [1] to update urls in our repo to use https, but
> > those https urls to xenbits don't work in our containers, due to an expired
> > root certificate. So we need to update those containers.
> >
> > This series update the dockerfile where just rebuilding the container isn't enough.
> 
> LGTM.
> 
> Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
> 
> I'll add this to my commit sweep, and rebuild the remaining containers.
> 
> But on that note, I noticed that the debian unstable container was 2.3G
> when I last rebuilt it.  Which I think is obscenely large for what we're
> doing.
> 
> Can we see about switching to slim/tiny container bases?

I don't think that would help much, the non-slim container is only 116MB
vs 74.6MB for the slim (amd64 containers). But maybe we could try to use
"--no-install-recommends", that might save a few bytes in our containers.

Cheers,

-- 
Anthony PERARD

Re: [XEN PATCH v3 0/4] automation: Update containers to allow HTTPS access to xenbits

[Andrew Cooper]

On 24/02/2023 6:12 pm, Anthony PERARD wrote:
> On Fri, Feb 24, 2023 at 05:37:44PM +0000, Andrew Cooper wrote:
>> On 24/02/2023 5:29 pm, Anthony PERARD wrote:
>>> Patch series available in this git branch:
>>> https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-containers-update-v3
>>>
>>> v3:
>>> - new patch which remove non-debug x86_32 builds
>>> - don't fix root certificates in jessie containers as those won't be used
>>>   anymore on the main branch.
>>>
>>> v2:
>>> - Remove CentOS 7.2
>>> - Remove Debian Jessie test, but update container recipe for the benefit of
>>>   older branches.
>>> - Fix CentOS 7 containner recipe to update all packages. (Fix missing update of
>>>   HTTPS root certificates)
>>>
>>> There is work in progress [1] to update urls in our repo to use https, but
>>> those https urls to xenbits don't work in our containers, due to an expired
>>> root certificate. So we need to update those containers.
>>>
>>> This series update the dockerfile where just rebuilding the container isn't enough.
>> LGTM.
>>
>> Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
>>
>> I'll add this to my commit sweep, and rebuild the remaining containers.
>>
>> But on that note, I noticed that the debian unstable container was 2.3G
>> when I last rebuilt it.  Which I think is obscenely large for what we're
>> doing.
>>
>> Can we see about switching to slim/tiny container bases?
> I don't think that would help much, the non-slim container is only 116MB
> vs 74.6MB

That's 64% smaller to start with...

>  for the slim (amd64 containers). But maybe we could try to use
> "--no-install-recommends", that might save a few bytes in our containers.

Oh wow - yeah.  For buster-gcc-ibt (which is already slim), we're getting:

Suggested packages:
  binutils-doc bison-doc bzip2-doc cpp-doc gcc-8-locales dbus-user-session
  libpam-systemd pinentry-gnome3 tor debian-keyring flex-doc g++-multilib
  g++-8-multilib gcc-8-doc libstdc++6-8-dbg autoconf automake libtool gdb
  gcc-doc libgcc1-dbg libgomp1-dbg libitm1-dbg libatomic1-dbg libasan5-dbg
  liblsan0-dbg libtsan0-dbg libubsan1-dbg libmpx2-dbg libquadmath0-dbg
  parcimonie xloadimage scdaemon glibc-doc sensible-utils git bzr gdbm-l10n
  gmp-doc libgmp10-doc libmpfr-doc libsasl2-modules-gssapi-mit
  | libsasl2-modules-gssapi-heimdal libsasl2-modules-ldap
libsasl2-modules-otp
  libsasl2-modules-sql libstdc++-8-doc m4-doc make-doc man-browser ed
  diffutils-doc perl-doc libterm-readline-gnu-perl
  | libterm-readline-perl-perl libb-debug-perl liblocale-codes-perl
  pinentry-doc readline-doc


which is a whole bunch of junk we absolutely do not need.

~Andrew