[PATCH v2 0/2] ci: allow running Coverity Scan uploads via GitLab

[PATCH v2 0/2] ci: allow running Coverity Scan uploads via GitLab

[Paolo Bonzini]

The machine that is used to upload QEMU builds to Coverity is used daily
as a development machine by Emanuele and myself, and as a result sometimes
its podman/docker setup gets messed up.  When this happens, Coverity
uploads might stop for extended periods of time.

In the interest of tightening this and of depending less on infrastructure
maintained by specific people, replace the manually-managed crontab
entry with a new job in GitLab's CI; this is also what Libvirt does.

The only change from v1 is better documentation of the changes in the
script exit code when the upload check fails, for example due to a bad
token.

Paolo

Paolo Bonzini (2):
  run-coverity-scan: add --check-upload-only option
  gitlab-ci: add manual job to run Coverity

 .gitlab-ci.d/base.yml                   |  4 ++
 .gitlab-ci.d/buildtest.yml              | 37 ++++++++++++++++
 .gitlab-ci.d/opensbi.yml                |  4 ++
 scripts/coverity-scan/run-coverity-scan | 59 ++++++++++++++++++-------
 4 files changed, 87 insertions(+), 17 deletions(-)

-- 
2.43.2

[PATCH v2 1/2] run-coverity-scan: add --check-upload-only option

[Paolo Bonzini]

Add an option to check if upload is permitted without actually
attempting a build.  This can be useful to add a third outcome
beyond success and failure---namely, a CI job can self-cancel
if the uploading quota has been reached.

There is a small change here in that a failure to do the upload
check changes the exit code from 1 to 99.  99 was chosen because
it is what Autotools and Meson use to represent a problem in the
setup (as opposed to a failure in the test).

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 scripts/coverity-scan/run-coverity-scan | 59 ++++++++++++++++++-------
 1 file changed, 42 insertions(+), 17 deletions(-)

diff --git a/scripts/coverity-scan/run-coverity-scan b/scripts/coverity-scan/run-coverity-scan
index d56c9b66776..96d852aa362 100755
--- a/scripts/coverity-scan/run-coverity-scan
+++ b/scripts/coverity-scan/run-coverity-scan
@@ -28,6 +28,7 @@
 # project settings, if you have maintainer access there.
 
 # Command line options:
+#   --check-upload-only : return success if upload is possible
 #   --dry-run : run the tools, but don't actually do the upload
 #   --docker : create and work inside a container
 #   --docker-engine : specify the container engine to use (docker/podman/auto);
@@ -57,18 +58,18 @@
 # putting it in a file and using --tokenfile. Everything else has
 # a reasonable default if this is run from a git tree.
 
-check_upload_permissions() {
-    # Check whether we can do an upload to the server; will exit the script
-    # with status 1 if the check failed (usually a bad token);
-    # will exit the script with status 0 if the check indicated that we
-    # can't upload yet (ie we are at quota)
-    # Assumes that COVERITY_TOKEN, PROJNAME and DRYRUN have been initialized.
+upload_permitted() {
+    # Check whether we can do an upload to the server; will exit *the script*
+    # with status 99 if the check failed (usually a bad token);
+    # will return from the function with status 1 if the check indicated
+    # that we can't upload yet (ie we are at quota)
+    # Assumes that COVERITY_TOKEN and PROJNAME have been initialized.
 
     echo "Checking upload permissions..."
 
     if ! up_perm="$(wget https://scan.coverity.com/api/upload_permitted --post-data "token=$COVERITY_TOKEN&project=$PROJNAME" -q -O -)"; then
         echo "Coverity Scan API access denied: bad token?"
-        exit 1
+        exit 99
     fi
 
     # Really up_perm is a JSON response with either
@@ -76,25 +77,40 @@ check_upload_permissions() {
     # We do some hacky string parsing instead of properly parsing it.
     case "$up_perm" in
         *upload_permitted*true*)
-            echo "Coverity Scan: upload permitted"
+            return 0
             ;;
         *next_upload_permitted_at*)
-            if [ "$DRYRUN" = yes ]; then
-                echo "Coverity Scan: upload quota reached, continuing dry run"
-            else
-                echo "Coverity Scan: upload quota reached; stopping here"
-                # Exit success as this isn't a build error.
-                exit 0
-            fi
+            return 1
             ;;
         *)
             echo "Coverity Scan upload check: unexpected result $up_perm"
-            exit 1
+            exit 99
             ;;
     esac
 }
 
 
+check_upload_permissions() {
+    # Check whether we can do an upload to the server; will exit the script
+    # with status 1 if the check failed (usually a bad token);
+    # will exit the script with status 0 if the check indicated that we
+    # can't upload yet (ie we are at quota)
+    # Assumes that COVERITY_TOKEN, PROJNAME and DRYRUN have been initialized.
+
+    if upload_permitted; then
+        echo "Coverity Scan: upload permitted"
+    else
+        if [ "$DRYRUN" = yes ]; then
+            echo "Coverity Scan: upload quota reached, continuing dry run"
+        else
+            echo "Coverity Scan: upload quota reached; stopping here"
+            # Exit success as this isn't a build error.
+            exit 0
+        fi
+    fi
+}
+
+
 build_docker_image() {
     # build docker container including the coverity-scan tools
     echo "Building docker container..."
@@ -152,9 +168,14 @@ update_coverity_tools () {
 DRYRUN=no
 UPDATE=yes
 DOCKER=no
+PROJNAME=QEMU
 
 while [ "$#" -ge 1 ]; do
     case "$1" in
+        --check-upload-only)
+            shift
+            DRYRUN=check
+            ;;
         --dry-run)
             shift
             DRYRUN=yes
@@ -251,6 +272,11 @@ if [ -z "$COVERITY_TOKEN" ]; then
     exit 1
 fi
 
+if [ "$DRYRUN" = check ]; then
+    upload_permitted
+    exit $?
+fi
+
 if [ -z "$COVERITY_BUILD_CMD" ]; then
     NPROC=$(nproc)
     COVERITY_BUILD_CMD="make -j$NPROC"
@@ -266,7 +292,6 @@ if [ -z "$SRCDIR" ]; then
     SRCDIR="$PWD"
 fi
 
-PROJNAME=QEMU
 TARBALL=cov-int.tar.xz
 
 if [ "$UPDATE" = only ]; then
-- 
2.43.2

[PATCH v2 2/2] gitlab-ci: add manual job to run Coverity

[Paolo Bonzini]

Add a job that can be run, either manually or on a schedule, to upload
a build to Coverity Scan.  The job uses the run-coverity-scan script
in multiple phases of check, download tools and upload, in order to
avoid both wasting time (skip everything if you are above the upload
quota) and avoid filling the log with the progress of downloading
the tools.

The job is intended to run on a scheduled pipeline run, and scheduled
runs will not get any other job.  It requires two variables to be in
GitLab CI, COVERITY_TOKEN and COVERITY_EMAIL.  Those are already set up
in qemu-project's configuration as protected and masked variables.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 .gitlab-ci.d/base.yml      |  4 ++++
 .gitlab-ci.d/buildtest.yml | 37 +++++++++++++++++++++++++++++++++++++
 .gitlab-ci.d/opensbi.yml   |  4 ++++
 3 files changed, 45 insertions(+)

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index ef173a34e63..2dd8a9b57cb 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -41,6 +41,10 @@ variables:
     - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_TAG'
       when: never
 
+    # Scheduled runs on mainline don't get pipelines except for the special Coverity job
+    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_PIPELINE_SOURCE == "schedule"'
+      when: never
+
     # Cirrus jobs can't run unless the creds / target repo are set
     - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == null || $CIRRUS_API_TOKEN == null)'
       when: never
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 901265af95d..c7d92fc3018 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -729,3 +729,40 @@ pages:
       - public
   variables:
     QEMU_JOB_PUBLISH: 1
+
+coverity:
+  image: $CI_REGISTRY_IMAGE/qemu/fedora:$QEMU_CI_CONTAINER_TAG
+  stage: build
+  allow_failure: true
+  timeout: 3h
+  needs:
+    - job: amd64-fedora-container
+      optional: true
+  before_script:
+    - dnf install -y curl wget
+  script:
+    # would be nice to cancel the job if over quota (https://gitlab.com/gitlab-org/gitlab/-/issues/256089)
+    # for example:
+    #   curl --request POST --header "PRIVATE-TOKEN: $CI_JOB_TOKEN" "${CI_SERVER_URL}/api/v4/projects/${CI_PROJECT_ID}/jobs/${CI_JOB_ID}/cancel
+    - 'scripts/coverity-scan/run-coverity-scan --check-upload-only || { exitcode=$?; if test $exitcode = 1; then
+        exit 0;
+      else
+        exit $exitcode;
+      fi; };
+      scripts/coverity-scan/run-coverity-scan --update-tools-only > update-tools.log 2>&1 || { cat update-tools.log; exit 1; };
+      scripts/coverity-scan/run-coverity-scan --no-update-tools'
+  rules:
+    - if: '$COVERITY_TOKEN == null'
+      when: never
+    - if: '$COVERITY_EMAIL == null'
+      when: never
+    # Never included on upstream pipelines, except for schedules
+    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_PIPELINE_SOURCE == "schedule"'
+      when: on_success
+    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM'
+      when: never
+    # Forks don't get any pipeline unless QEMU_CI=1 or QEMU_CI=2 is set
+    - if: '$QEMU_CI != "1" && $QEMU_CI != "2"'
+      when: never
+    # Always manual on forks even if $QEMU_CI == "2"
+    - when: manual
diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml
index fd293e6c317..42f137d624e 100644
--- a/.gitlab-ci.d/opensbi.yml
+++ b/.gitlab-ci.d/opensbi.yml
@@ -24,6 +24,10 @@
     - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE != "qemu-project" && $CI_COMMIT_MESSAGE =~ /opensbi/i'
       when: manual
 
+    # Scheduled runs on mainline don't get pipelines except for the special Coverity job
+    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_PIPELINE_SOURCE == "schedule"'
+      when: never
+
     # Run if any files affecting the build output are touched
     - changes:
         - .gitlab-ci.d/opensbi.yml
-- 
2.43.2

Re: [PATCH v2 1/2] run-coverity-scan: add --check-upload-only option

[Peter Maydell]

On Fri, 8 Mar 2024 at 13:05, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> Add an option to check if upload is permitted without actually
> attempting a build.  This can be useful to add a third outcome
> beyond success and failure---namely, a CI job can self-cancel
> if the uploading quota has been reached.
>
> There is a small change here in that a failure to do the upload
> check changes the exit code from 1 to 99.  99 was chosen because
> it is what Autotools and Meson use to represent a problem in the
> setup (as opposed to a failure in the test).
>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

> -check_upload_permissions() {
> -    # Check whether we can do an upload to the server; will exit the script
> -    # with status 1 if the check failed (usually a bad token);
> -    # will exit the script with status 0 if the check indicated that we
> -    # can't upload yet (ie we are at quota)
> -    # Assumes that COVERITY_TOKEN, PROJNAME and DRYRUN have been initialized.
> +upload_permitted() {
> +    # Check whether we can do an upload to the server; will exit *the script*
> +    # with status 99 if the check failed (usually a bad token);
> +    # will return from the function with status 1 if the check indicated
> +    # that we can't upload yet (ie we are at quota)
> +    # Assumes that COVERITY_TOKEN and PROJNAME have been initialized.
>
>      echo "Checking upload permissions..."
>
>      if ! up_perm="$(wget https://scan.coverity.com/api/upload_permitted --post-data "token=$COVERITY_TOKEN&project=$PROJNAME" -q -O -)"; then
>          echo "Coverity Scan API access denied: bad token?"
> -        exit 1
> +        exit 99
>      fi
>
>      # Really up_perm is a JSON response with either
> @@ -76,25 +77,40 @@ check_upload_permissions() {
>      # We do some hacky string parsing instead of properly parsing it.
>      case "$up_perm" in
>          *upload_permitted*true*)
> -            echo "Coverity Scan: upload permitted"
> +            return 0
>              ;;
>          *next_upload_permitted_at*)
> -            if [ "$DRYRUN" = yes ]; then
> -                echo "Coverity Scan: upload quota reached, continuing dry run"
> -            else
> -                echo "Coverity Scan: upload quota reached; stopping here"
> -                # Exit success as this isn't a build error.
> -                exit 0
> -            fi
> +            return 1
>              ;;
>          *)
>              echo "Coverity Scan upload check: unexpected result $up_perm"
> -            exit 1
> +            exit 99
>              ;;
>      esac
>  }
>
>
> +check_upload_permissions() {
> +    # Check whether we can do an upload to the server; will exit the script
> +    # with status 1 if the check failed (usually a bad token);

This should also be "status 99", I think.

> +    # will exit the script with status 0 if the check indicated that we
> +    # can't upload yet (ie we are at quota)
> +    # Assumes that COVERITY_TOKEN, PROJNAME and DRYRUN have been initialized.
> +
> +    if upload_permitted; then
> +        echo "Coverity Scan: upload permitted"
> +    else
> +        if [ "$DRYRUN" = yes ]; then
> +            echo "Coverity Scan: upload quota reached, continuing dry run"
> +        else
> +            echo "Coverity Scan: upload quota reached; stopping here"
> +            # Exit success as this isn't a build error.
> +            exit 0
> +        fi
> +    fi
> +}

Otherwise
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>

thanks
-- PMM

Re: [PATCH v2 2/2] gitlab-ci: add manual job to run Coverity

[Daniel P. Berrangé]

On Fri, Mar 08, 2024 at 02:05:07PM +0100, Paolo Bonzini wrote:
> Add a job that can be run, either manually or on a schedule, to upload
> a build to Coverity Scan.  The job uses the run-coverity-scan script
> in multiple phases of check, download tools and upload, in order to
> avoid both wasting time (skip everything if you are above the upload
> quota) and avoid filling the log with the progress of downloading
> the tools.
> 
> The job is intended to run on a scheduled pipeline run, and scheduled
> runs will not get any other job.  It requires two variables to be in
> GitLab CI, COVERITY_TOKEN and COVERITY_EMAIL.  Those are already set up
> in qemu-project's configuration as protected and masked variables.
> 
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>  .gitlab-ci.d/base.yml      |  4 ++++
>  .gitlab-ci.d/buildtest.yml | 37 +++++++++++++++++++++++++++++++++++++
>  .gitlab-ci.d/opensbi.yml   |  4 ++++
>  3 files changed, 45 insertions(+)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|