[RFC PATCH 0/2] ci: allow running Coverity Scan uploads via GitLab

[RFC PATCH 0/2] ci: allow running Coverity Scan uploads via GitLab

[Paolo Bonzini]

The machine that is used to upload QEMU builds to Coverity is used daily
as a development machine by Emanuele and myself, and as a result sometimes
its podman/docker setup gets messed up.  When this happens, Coverity
uploads might stop for extended periods of time.

In the interest of tightening this and of depending less on infrastructure
maintained by specific people, replace the manually-managed crontab
entry with a new job in GitLab's CI; this is also what Libvirt does.
The rules to trigger it are a bit different compared to other jobs:

* on mainline, it only runs for the default (master) branch and only
  as part of scheduled pipeline runs.  A rule is added to remove all
  other jobs when running from a scheduled pipeline.

* on forks, it is always manual (and only appears if QEMU_CI=1 or 2,
  like other build jobs)

For now I implemented these rules directly in the buildtest.yml file,
but it is also possible to add a QEMU_JOB_SCHEDULE variable.

Example of a working run: https://gitlab.com/bonzini/qemu/-/jobs/6312777493

Example of a run that is over quota: https://gitlab.com/bonzini/qemu/-/jobs/6315895373
This run shows an attempt to cancel the job if the quota is exceeded, but
it does not work (it worked on GitLab <17.0) so I removed it from the patch.

Paolo

Paolo Bonzini (2):
  run-coverity-scan: add --check-upload-only option
  gitlab-ci: add manual job to run Coverity

 .gitlab-ci.d/base.yml                   |  4 ++
 .gitlab-ci.d/buildtest.yml              | 36 +++++++++++++++++
 scripts/coverity-scan/run-coverity-scan | 51 ++++++++++++++++++-------
 3 files changed, 78 insertions(+), 13 deletions(-)

-- 
2.39.1

[RFC PATCH 1/2] run-coverity-scan: add --check-upload-only option

[Paolo Bonzini]

Add an option to check if upload is permitted without actually
attempting a build.  This can be useful to add a third outcome
beyond success and failure---namely, a CI job can self-cancel
if the uploading quota has been reached.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 scripts/coverity-scan/run-coverity-scan | 51 ++++++++++++++++++-------
 1 file changed, 38 insertions(+), 13 deletions(-)

diff --git a/scripts/coverity-scan/run-coverity-scan b/scripts/coverity-scan/run-coverity-scan
index 871826b29e..c199d57a5f 100755
--- a/scripts/coverity-scan/run-coverity-scan
+++ b/scripts/coverity-scan/run-coverity-scan
@@ -28,6 +28,7 @@
 # project settings, if you have maintainer access there.
 
 # Command line options:
+#   --check-upload-only : return success if upload is possible
 #   --dry-run : run the tools, but don't actually do the upload
 #   --docker : create and work inside a container
 #   --docker-engine : specify the container engine to use (docker/podman/auto);
@@ -57,18 +58,18 @@
 # putting it in a file and using --tokenfile. Everything else has
 # a reasonable default if this is run from a git tree.
 
-check_upload_permissions() {
+upload_permitted() {
     # Check whether we can do an upload to the server; will exit the script
     # with status 1 if the check failed (usually a bad token);
     # will exit the script with status 0 if the check indicated that we
     # can't upload yet (ie we are at quota)
-    # Assumes that COVERITY_TOKEN, PROJNAME and DRYRUN have been initialized.
+    # Assumes that COVERITY_TOKEN and PROJNAME have been initialized.
 
     echo "Checking upload permissions..."
 
     if ! up_perm="$(wget https://scan.coverity.com/api/upload_permitted --post-data "token=$COVERITY_TOKEN&project=$PROJNAME" -q -O -)"; then
         echo "Coverity Scan API access denied: bad token?"
-        exit 1
+        exit 99
     fi
 
     # Really up_perm is a JSON response with either
@@ -76,25 +77,40 @@ check_upload_permissions() {
     # We do some hacky string parsing instead of properly parsing it.
     case "$up_perm" in
         *upload_permitted*true*)
-            echo "Coverity Scan: upload permitted"
+            return 0
             ;;
         *next_upload_permitted_at*)
-            if [ "$DRYRUN" = yes ]; then
-                echo "Coverity Scan: upload quota reached, continuing dry run"
-            else
-                echo "Coverity Scan: upload quota reached; stopping here"
-                # Exit success as this isn't a build error.
-                exit 0
-            fi
+            return 1
             ;;
         *)
             echo "Coverity Scan upload check: unexpected result $up_perm"
-            exit 1
+            exit 99
             ;;
     esac
 }
 
 
+check_upload_permissions() {
+    # Check whether we can do an upload to the server; will exit the script
+    # with status 1 if the check failed (usually a bad token);
+    # will exit the script with status 0 if the check indicated that we
+    # can't upload yet (ie we are at quota)
+    # Assumes that COVERITY_TOKEN, PROJNAME and DRYRUN have been initialized.
+
+    if upload_permitted; then
+        echo "Coverity Scan: upload permitted"
+    else
+        if [ "$DRYRUN" = yes ]; then
+            echo "Coverity Scan: upload quota reached, continuing dry run"
+        else
+            echo "Coverity Scan: upload quota reached; stopping here"
+            # Exit success as this isn't a build error.
+            exit 0
+        fi
+    fi
+}
+
+
 build_docker_image() {
     # build docker container including the coverity-scan tools
     echo "Building docker container..."
@@ -152,9 +168,14 @@ update_coverity_tools () {
 DRYRUN=no
 UPDATE=yes
 DOCKER=no
+PROJNAME=QEMU
 
 while [ "$#" -ge 1 ]; do
     case "$1" in
+        --check-upload-only)
+            shift
+            DRYRUN=check
+            ;;
         --dry-run)
             shift
             DRYRUN=yes
@@ -251,6 +272,11 @@ if [ -z "$COVERITY_TOKEN" ]; then
     exit 1
 fi
 
+if [ "$DRYRUN" = check ]; then
+    upload_permitted
+    exit $?
+fi
+
 if [ -z "$COVERITY_BUILD_CMD" ]; then
     NPROC=$(nproc)
     COVERITY_BUILD_CMD="make -j$NPROC"
@@ -266,7 +292,6 @@ if [ -z "$SRCDIR" ]; then
     SRCDIR="$PWD"
 fi
 
-PROJNAME=QEMU
 TARBALL=cov-int.tar.xz
 
 if [ "$UPDATE" = only ]; then
-- 
2.39.1

[RFC PATCH 2/2] gitlab-ci: add manual job to run Coverity

[Paolo Bonzini]

Add a job that can be run, either manually or on a schedule, to upload
a build to Coverity Scan.  The job uses the run-coverity-scan script
in multiple phases of check, download tools and upload, in order to
avoid both wasting time (skip everything if you are above the upload
quota) and avoid filling the log with the progress of downloading
the tools.

The job is intended to run on a scheduled pipeline run, and scheduled
runs will not get any other job.  It requires two variables to be in
GitLab CI, COVERITY_TOKEN and COVERITY_EMAIL.  Those are already set up
in qemu-project's configuration as protected and masked variables.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 .gitlab-ci.d/base.yml      |  4 ++++
 .gitlab-ci.d/buildtest.yml | 36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index ef173a34e6..2dd8a9b57c 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -41,6 +41,10 @@ variables:
     - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_TAG'
       when: never
 
+    # Scheduled runs on mainline don't get pipelines except for the special Coverity job
+    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_PIPELINE_SOURCE == "schedule"'
+      when: never
+
     # Cirrus jobs can't run unless the creds / target repo are set
     - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == null || $CIRRUS_API_TOKEN == null)'
       when: never
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index a1c030337b..378dee055b 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -729,3 +729,38 @@ pages:
       - public
   variables:
     QEMU_JOB_PUBLISH: 1
+
+coverity:
+  image: $CI_REGISTRY_IMAGE/qemu/fedora:$QEMU_CI_CONTAINER_TAG
+  stage: build
+  allow_failure: true
+  timeout: 3h
+  needs:
+    - job: amd64-fedora-container
+      optional: true
+  before_script:
+    - dnf install -y curl wget
+  script:
+    # would be nice to cancel the job if over quota (https://gitlab.com/gitlab-org/gitlab/-/issues/256089)
+    - 'scripts/coverity-scan/run-coverity-scan --check-upload-only || (exitcode=$?; if test $exitcode = 1; then
+        exit 0;
+      else
+        exit $exitcode;
+      fi)'
+    - 'scripts/coverity-scan/run-coverity-scan --update-tools-only > update-tools.log 2>&1 || cat update-tools.log'
+    - 'scripts/coverity-scan/run-coverity-scan --no-update-tools'
+  rules:
+    - if: '$COVERITY_TOKEN == null'
+      when: never
+    - if: '$COVERITY_EMAIL == null'
+      when: never
+    # Never included on upstream pipelines, except for schedules
+    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_PIPELINE_SOURCE == "schedule" && $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH'
+      when: on_success
+    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM'
+      when: never
+    # Forks don't get any pipeline unless QEMU_CI=1 or QEMU_CI=2 is set
+    - if: '$QEMU_CI != "1" && $QEMU_CI != "2"'
+      when: never
+    # Always manual on forks even if $QEMU_CI == "2"
+    - when: manual
-- 
2.39.1

Re: [RFC PATCH 2/2] gitlab-ci: add manual job to run Coverity

[Daniel P. Berrangé]

On Mon, Mar 04, 2024 at 05:06:31PM -0500, Paolo Bonzini wrote:
> Add a job that can be run, either manually or on a schedule, to upload
> a build to Coverity Scan.  The job uses the run-coverity-scan script
> in multiple phases of check, download tools and upload, in order to
> avoid both wasting time (skip everything if you are above the upload
> quota) and avoid filling the log with the progress of downloading
> the tools.
> 
> The job is intended to run on a scheduled pipeline run, and scheduled
> runs will not get any other job.  It requires two variables to be in
> GitLab CI, COVERITY_TOKEN and COVERITY_EMAIL.  Those are already set up
> in qemu-project's configuration as protected and masked variables.
> 
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>  .gitlab-ci.d/base.yml      |  4 ++++
>  .gitlab-ci.d/buildtest.yml | 36 ++++++++++++++++++++++++++++++++++++
>  2 files changed, 40 insertions(+)
> 
> diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
> index ef173a34e6..2dd8a9b57c 100644
> --- a/.gitlab-ci.d/base.yml
> +++ b/.gitlab-ci.d/base.yml
> @@ -41,6 +41,10 @@ variables:
>      - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_TAG'
>        when: never
>  
> +    # Scheduled runs on mainline don't get pipelines except for the special Coverity job
> +    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_PIPELINE_SOURCE == "schedule"'
> +      when: never
> +
>      # Cirrus jobs can't run unless the creds / target repo are set
>      - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == null || $CIRRUS_API_TOKEN == null)'
>        when: never
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index a1c030337b..378dee055b 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -729,3 +729,38 @@ pages:
>        - public
>    variables:
>      QEMU_JOB_PUBLISH: 1
> +
> +coverity:
> +  image: $CI_REGISTRY_IMAGE/qemu/fedora:$QEMU_CI_CONTAINER_TAG
> +  stage: build
> +  allow_failure: true
> +  timeout: 3h
> +  needs:
> +    - job: amd64-fedora-container
> +      optional: true
> +  before_script:
> +    - dnf install -y curl wget
> +  script:
> +    # would be nice to cancel the job if over quota (https://gitlab.com/gitlab-org/gitlab/-/issues/256089)
> +    - 'scripts/coverity-scan/run-coverity-scan --check-upload-only || (exitcode=$?; if test $exitcode = 1; then
> +        exit 0;
> +      else
> +        exit $exitcode;
> +      fi)'
> +    - 'scripts/coverity-scan/run-coverity-scan --update-tools-only > update-tools.log 2>&1 || cat update-tools.log'

Slightly shorter as:

  .....   2>&1 | tee update-tools.log

> +    - 'scripts/coverity-scan/run-coverity-scan --no-update-tools'
> +  rules:
> +    - if: '$COVERITY_TOKEN == null'
> +      when: never
> +    - if: '$COVERITY_EMAIL == null'
> +      when: never
> +    # Never included on upstream pipelines, except for schedules
> +    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_PIPELINE_SOURCE == "schedule" && $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH'
> +      when: on_success
> +    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM'
> +      when: never
> +    # Forks don't get any pipeline unless QEMU_CI=1 or QEMU_CI=2 is set
> +    - if: '$QEMU_CI != "1" && $QEMU_CI != "2"'
> +      when: never
> +    # Always manual on forks even if $QEMU_CI == "2"
> +    - when: manual
> -- 

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH 2/2] gitlab-ci: add manual job to run Coverity

[Paolo Bonzini]

On Tue, Mar 5, 2024 at 9:52 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > +    - 'scripts/coverity-scan/run-coverity-scan --update-tools-only > update-tools.log 2>&1 || cat update-tools.log'
>
> Slightly shorter as:
>
>   .....   2>&1 | tee update-tools.log

Note that stdout/stderr are only dumped if the download fails, so tee
cannot be used.

> > +    - 'scripts/coverity-scan/run-coverity-scan --no-update-tools'
> > +  rules:
> > +    - if: '$COVERITY_TOKEN == null'
> > +      when: never
> > +    - if: '$COVERITY_EMAIL == null'
> > +      when: never
> > +    # Never included on upstream pipelines, except for schedules
> > +    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_PIPELINE_SOURCE == "schedule" && $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH'
> > +      when: on_success
> > +    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM'
> > +      when: never
> > +    # Forks don't get any pipeline unless QEMU_CI=1 or QEMU_CI=2 is set
> > +    - if: '$QEMU_CI != "1" && $QEMU_CI != "2"'
> > +      when: never
> > +    # Always manual on forks even if $QEMU_CI == "2"
> > +    - when: manual
> > --
>
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

Thanks! There will be a few small changes after I actually tested this
on a schedule, but nothing major.

Paolo