[PATCH 0/2] gitlab: expose installed package info in build logs

[PATCH 0/2] gitlab: expose installed package info in build logs

[Daniel P. Berrangé]

Many times we see a build job start failing, we wonder if the installed
packages have changed since the last passing build. We can rarely
diagnose this, however, since we only have the new container image, not
the old one.

The lcitool generated containers create a /packages.txt file whose
content is intended to be output in the build phase, so record the
packages associated with the build.

This adds packages.txt to the manually written containers, and modifies
the build jobs to display this content. This will improve our future
debuggability of CI problems.

Daniel P. Berrangé (2):
  gitlab: record installed packages in /packages.txt in containers
  gitlab: display /packages.txt in build jobs

 .gitlab-ci.d/buildtest-template.yml                      | 1 +
 .gitlab-ci.d/crossbuild-template.yml                     | 2 ++
 tests/docker/dockerfiles/debian-all-test-cross.docker    | 3 ++-
 tests/docker/dockerfiles/debian-hexagon-cross.docker     | 3 ++-
 tests/docker/dockerfiles/debian-legacy-test-cross.docker | 3 ++-
 tests/docker/dockerfiles/debian-loongarch-cross.docker   | 3 ++-
 tests/docker/dockerfiles/debian-tricore-cross.docker     | 3 ++-
 tests/docker/dockerfiles/debian-xtensa-cross.docker      | 3 ++-
 8 files changed, 15 insertions(+), 6 deletions(-)

-- 
2.45.2

[PATCH 1/2] gitlab: record installed packages in /packages.txt in containers

[Daniel P. Berrangé]

The lcitool created containers save the full distro package list
details into /packages.txt. The idea is that build jobs will 'cat'
this file, so that the build log has a record of what packages
were used. This is important info, because when it comes to debug
failures, the original container is often lost.

This extends the manually written dockerfiles to also create the
/packages.txt file.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 tests/docker/dockerfiles/debian-all-test-cross.docker    | 3 ++-
 tests/docker/dockerfiles/debian-hexagon-cross.docker     | 3 ++-
 tests/docker/dockerfiles/debian-legacy-test-cross.docker | 3 ++-
 tests/docker/dockerfiles/debian-loongarch-cross.docker   | 3 ++-
 tests/docker/dockerfiles/debian-tricore-cross.docker     | 3 ++-
 tests/docker/dockerfiles/debian-xtensa-cross.docker      | 3 ++-
 6 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/tests/docker/dockerfiles/debian-all-test-cross.docker b/tests/docker/dockerfiles/debian-all-test-cross.docker
index 6cc38a3633..8ab244e018 100644
--- a/tests/docker/dockerfiles/debian-all-test-cross.docker
+++ b/tests/docker/dockerfiles/debian-all-test-cross.docker
@@ -62,7 +62,8 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
         gcc-s390x-linux-gnu \
         libc6-dev-s390x-cross \
         gcc-sparc64-linux-gnu \
-        libc6-dev-sparc64-cross
+        libc6-dev-sparc64-cross && \
+        dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt
 
 
 ENV QEMU_CONFIGURE_OPTS --disable-system --disable-docs --disable-tools
diff --git a/tests/docker/dockerfiles/debian-hexagon-cross.docker b/tests/docker/dockerfiles/debian-hexagon-cross.docker
index f2d40f2dee..23152b4918 100644
--- a/tests/docker/dockerfiles/debian-hexagon-cross.docker
+++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker
@@ -33,7 +33,8 @@ RUN apt-get update && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/c++ && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/g++ && \
-    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
+    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc && \
+    dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt
 
 RUN /usr/bin/pip3 install tomli
 
diff --git a/tests/docker/dockerfiles/debian-legacy-test-cross.docker b/tests/docker/dockerfiles/debian-legacy-test-cross.docker
index d75e0b85e2..5a6616b7d3 100644
--- a/tests/docker/dockerfiles/debian-legacy-test-cross.docker
+++ b/tests/docker/dockerfiles/debian-legacy-test-cross.docker
@@ -36,7 +36,8 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
         python3-pip \
         python3-setuptools \
         python3-venv \
-        python3-wheel
+        python3-wheel && \
+        dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt
 
 RUN /usr/bin/pip3 install tomli
 
diff --git a/tests/docker/dockerfiles/debian-loongarch-cross.docker b/tests/docker/dockerfiles/debian-loongarch-cross.docker
index 6a9197528b..79eab5621e 100644
--- a/tests/docker/dockerfiles/debian-loongarch-cross.docker
+++ b/tests/docker/dockerfiles/debian-loongarch-cross.docker
@@ -32,7 +32,8 @@ RUN apt-get update && \
         python3-pip \
         python3-setuptools \
         python3-venv \
-        python3-wheel
+        python3-wheel && \
+        dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt
 
 RUN /usr/bin/pip3 install tomli
 
diff --git a/tests/docker/dockerfiles/debian-tricore-cross.docker b/tests/docker/dockerfiles/debian-tricore-cross.docker
index 16276aa21d..479b4d6eba 100644
--- a/tests/docker/dockerfiles/debian-tricore-cross.docker
+++ b/tests/docker/dockerfiles/debian-tricore-cross.docker
@@ -34,7 +34,8 @@ RUN apt update && \
        python3-pip \
        python3-setuptools \
        python3-wheel \
-       python3-venv
+       python3-venv && \
+       dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt
 
 RUN /usr/bin/pip3 install tomli
 
diff --git a/tests/docker/dockerfiles/debian-xtensa-cross.docker b/tests/docker/dockerfiles/debian-xtensa-cross.docker
index 413881899b..d011eee2ad 100644
--- a/tests/docker/dockerfiles/debian-xtensa-cross.docker
+++ b/tests/docker/dockerfiles/debian-xtensa-cross.docker
@@ -16,7 +16,8 @@ RUN apt-get update && \
         curl \
         gettext \
         git \
-        python3-minimal
+        python3-minimal && \
+    dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt
 
 ENV CPU_LIST dc232b dc233c de233_fpu dsp3400
 ENV TOOLCHAIN_RELEASE 2020.07
-- 
2.45.2

[PATCH 2/2] gitlab: display /packages.txt in build jobs

[Daniel P. Berrangé]

The lcitool created containers save the full distro package list
details into /packages.txt. The idea is that build jobs will 'cat'
this file, so that the build log has a record of what packages
were used. This is important info, because when it comes to debug
failures, the original container is often lost.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/buildtest-template.yml  | 1 +
 .gitlab-ci.d/crossbuild-template.yml | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index 8f7ebfaed8..844c26623d 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -9,6 +9,7 @@
     when: always
   before_script:
     - JOBS=$(expr $(nproc) + 1)
+    - cat /packages.txt
   script:
     - export CCACHE_BASEDIR="$(pwd)"
     - export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
index d9f81b7061..53051ec793 100644
--- a/.gitlab-ci.d/crossbuild-template.yml
+++ b/.gitlab-ci.d/crossbuild-template.yml
@@ -8,6 +8,8 @@
     key: "$CI_JOB_NAME"
     when: always
   timeout: 80m
+  before_script:
+    - cat /packages.txt
   script:
     - export CCACHE_BASEDIR="$(pwd)"
     - export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
-- 
2.45.2

Re: [PATCH 0/2] gitlab: expose installed package info in build logs

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

> Many times we see a build job start failing, we wonder if the installed
> packages have changed since the last passing build. We can rarely
> diagnose this, however, since we only have the new container image, not
> the old one.
>
> The lcitool generated containers create a /packages.txt file whose
> content is intended to be output in the build phase, so record the
> packages associated with the build.
>
> This adds packages.txt to the manually written containers, and modifies
> the build jobs to display this content. This will improve our future
> debuggability of CI problems.

Queued to maintainer/for-9.1, thanks.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH 0/2] gitlab: expose installed package info in build logs

[Philippe Mathieu-Daudé]

On 24/7/24 11:55, Daniel P. Berrangé wrote:

> Daniel P. Berrangé (2):
>    gitlab: record installed packages in /packages.txt in containers
>    gitlab: display /packages.txt in build jobs

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

Re: [PATCH 0/2] gitlab: expose installed package info in build logs

[Manos Pitsidianakis]

Hello Daniel,

On Wed, 24 Jul 2024 12:55, "Daniel P. Berrangé" <berrange@redhat.com> wrote:
>Many times we see a build job start failing, we wonder if the installed
>packages have changed since the last passing build. We can rarely
>diagnose this, however, since we only have the new container image, not
>the old one.
>

APT allows you to specify to pin package versions when installing; 
wouldn't that help ensure our tests are deterministic?

Furthermore, a gitlab cron job pipeline can be set up to run every e.g. 
few months and inform of any updates so that we can manually bump them.

Manos


>The lcitool generated containers create a /packages.txt file whose
>content is intended to be output in the build phase, so record the
>packages associated with the build.
>
>This adds packages.txt to the manually written containers, and modifies
>the build jobs to display this content. This will improve our future
>debuggability of CI problems.
>
>Daniel P. Berrangé (2):
>  gitlab: record installed packages in /packages.txt in containers
>  gitlab: display /packages.txt in build jobs
>
> .gitlab-ci.d/buildtest-template.yml                      | 1 +
> .gitlab-ci.d/crossbuild-template.yml                     | 2 ++
> tests/docker/dockerfiles/debian-all-test-cross.docker    | 3 ++-
> tests/docker/dockerfiles/debian-hexagon-cross.docker     | 3 ++-
> tests/docker/dockerfiles/debian-legacy-test-cross.docker | 3 ++-
> tests/docker/dockerfiles/debian-loongarch-cross.docker   | 3 ++-
> tests/docker/dockerfiles/debian-tricore-cross.docker     | 3 ++-
> tests/docker/dockerfiles/debian-xtensa-cross.docker      | 3 ++-
> 8 files changed, 15 insertions(+), 6 deletions(-)
>
>-- 
>2.45.2
>
>

Re: [PATCH 0/2] gitlab: expose installed package info in build logs

[Thomas Huth]

On 25/07/2024 11.42, Manos Pitsidianakis wrote:
> Hello Daniel,
> 
> On Wed, 24 Jul 2024 12:55, "Daniel P. Berrangé" <berrange@redhat.com> wrote:
>> Many times we see a build job start failing, we wonder if the installed
>> packages have changed since the last passing build. We can rarely
>> diagnose this, however, since we only have the new container image, not
>> the old one.
>>
> 
> APT allows you to specify to pin package versions when installing; wouldn't 
> that help ensure our tests are deterministic?

Even if tests would be more deterministic that way, I also don't think 
that's what we want: We would test with backlevel packages that users are 
not using (since they'll normally use the latest version of a package from a 
certain version of a distro). So in our CI, we would not be able to 
reproduce the issues that the users are seeing on their systems.

  Thomas

Re: [PATCH 0/2] gitlab: expose installed package info in build logs

[Daniel P. Berrangé]

On Thu, Jul 25, 2024 at 12:42:18PM +0300, Manos Pitsidianakis wrote:
> Hello Daniel,
> 
> On Wed, 24 Jul 2024 12:55, "Daniel P. Berrangé" <berrange@redhat.com> wrote:
> > Many times we see a build job start failing, we wonder if the installed
> > packages have changed since the last passing build. We can rarely
> > diagnose this, however, since we only have the new container image, not
> > the old one.
> > 
> 
> APT allows you to specify to pin package versions when installing; wouldn't
> that help ensure our tests are deterministic?

We want to be testing against latest packages because that reflects
what our users are likely to see on their own systems. IOW, having
failures after newly updated packages is a good thing - provided we
can easily identify what caused the breakage, to enable us to fix
it promptly - by promptly I mean same-day.

> Furthermore, a gitlab cron job pipeline can be set up to run every e.g. few
> months and inform of any updates so that we can manually bump them.

Only seeing breakage from new packages as long as a few months after it
happens is a bad thing. We benefit from fast detection and fast fixing.
In cases where the new package is broken, rather than QEMU, it is also
better to be able to report that back to the distro within days of them
rolling out the problem as it'll be fresh in their minds to fix.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 0/2] gitlab: expose installed package info in build logs

[Alex Bennée]

Manos Pitsidianakis <manos.pitsidianakis@linaro.org> writes:

> Hello Daniel,
>
> On Wed, 24 Jul 2024 12:55, "Daniel P. Berrangé" <berrange@redhat.com> wrote:
>>Many times we see a build job start failing, we wonder if the installed
>>packages have changed since the last passing build. We can rarely
>>diagnose this, however, since we only have the new container image, not
>>the old one.
>>
>
> APT allows you to specify to pin package versions when installing;
> wouldn't that help ensure our tests are deterministic?

Generally we want track the latest LTS version and usually the distros
are pretty good about being cautious about updates. I think the recent
case was because BSD's have a different policy about updating python.

IOW this is a nice to have for debugging but I don't think we want to
freeze the acceptable package set for any given install.

> Furthermore, a gitlab cron job pipeline can be set up to run every
> e.g. few months and inform of any updates so that we can manually bump
> them.
>
> Manos
>
>
>>The lcitool generated containers create a /packages.txt file whose
>>content is intended to be output in the build phase, so record the
>>packages associated with the build.
>>
>>This adds packages.txt to the manually written containers, and modifies
>>the build jobs to display this content. This will improve our future
>>debuggability of CI problems.
>>
>>Daniel P. Berrangé (2):
>>  gitlab: record installed packages in /packages.txt in containers
>>  gitlab: display /packages.txt in build jobs
>>
>> .gitlab-ci.d/buildtest-template.yml                      | 1 +
>> .gitlab-ci.d/crossbuild-template.yml                     | 2 ++
>> tests/docker/dockerfiles/debian-all-test-cross.docker    | 3 ++-
>> tests/docker/dockerfiles/debian-hexagon-cross.docker     | 3 ++-
>> tests/docker/dockerfiles/debian-legacy-test-cross.docker | 3 ++-
>> tests/docker/dockerfiles/debian-loongarch-cross.docker   | 3 ++-
>> tests/docker/dockerfiles/debian-tricore-cross.docker     | 3 ++-
>> tests/docker/dockerfiles/debian-xtensa-cross.docker      | 3 ++-
>> 8 files changed, 15 insertions(+), 6 deletions(-)
>>
>> -- 2.45.2
>>
>>

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH 0/2] gitlab: expose installed package info in build logs

[Manos Pitsidianakis]

On Thu, 25 Jul 2024 12:56, Thomas Huth <thuth@redhat.com> wrote:
>On 25/07/2024 11.42, Manos Pitsidianakis wrote:
>> Hello Daniel,
>> 
>> On Wed, 24 Jul 2024 12:55, "Daniel P. Berrangé" <berrange@redhat.com> wrote:
>>> Many times we see a build job start failing, we wonder if the installed
>>> packages have changed since the last passing build. We can rarely
>>> diagnose this, however, since we only have the new container image, not
>>> the old one.
>>>
>> 
>> APT allows you to specify to pin package versions when installing; wouldn't 
>> that help ensure our tests are deterministic?
>
>Even if tests would be more deterministic that way, I also don't think 
>that's what we want: We would test with backlevel packages that users are 
>not using (since they'll normally use the latest version of a package from a 
>certain version of a distro). So in our CI, we would not be able to 
>reproduce the issues that the users are seeing on their systems.
>

Definitely, but the current testing makes it harder to discern the lines 
between QEMU regression and QEMU not working with latest updates. If we 
had e.g. a pinned job and a "latest" job disagreeing we'd know the 
cause. Of course that means extra CI minutes, maintainer upkeep etc etc, 
I know, but this was my reasoning, I don't dismiss testing for what 
users experience. :)

Manos