[isar-cip-core][PATCH 0/4] ci: add LAVA live output, fix and clean up submission script

[isar-cip-core][PATCH 0/4] ci: add LAVA live output, fix and clean up submission script

[Jan Kiszka]

See patches for details, look at the output here:

https://gitlab.com/cip-project/cip-core/isar-cip-core/-/pipelines/1353479619

Jan

Jan Kiszka (4):
  ci: submit_lava: Fix reporting of submission errors
  ci: submit_lava: Fix access to LAVA API
  ci: Show live output of LAVA jobs
  ci: submit_lava: Reduce validate_jobs to handle only a single one

 scripts/submit_lava.sh | 196 ++++++-----------------------------------
 1 file changed, 27 insertions(+), 169 deletions(-)

-- 
2.43.0

[isar-cip-core][PATCH 1/4] ci: submit_lava: Fix reporting of submission errors

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Errors are collected in the global ERROR variable. Lower-case error is
just a local variable in check_status.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/submit_lava.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/submit_lava.sh b/scripts/submit_lava.sh
index 4b1b6711..2464c291 100755
--- a/scripts/submit_lava.sh
+++ b/scripts/submit_lava.sh
@@ -146,7 +146,7 @@ submit_job() {
 		echo "Submitting $1 to LAVA master..."
 		# Catch error that occurs if invalid yaml file is submitted
 		# shellcheck disable=2086
-		ret=$(lavacli $LAVACLI_ARGS jobs submit "$1") || error=true
+		ret=$(lavacli $LAVACLI_ARGS jobs submit "$1") || ERROR=true
 
 		if [[ $ret != [0-9]* ]]
 		then
-- 
2.43.0

[isar-cip-core][PATCH 2/4] ci: submit_lava: Fix access to LAVA API

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

We need to provide the token in order to retrieve job results.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/submit_lava.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/submit_lava.sh b/scripts/submit_lava.sh
index 2464c291..48c5d3c1 100755
--- a/scripts/submit_lava.sh
+++ b/scripts/submit_lava.sh
@@ -9,7 +9,7 @@ set -e
 ################################################################################
 LAVA_TEMPLATES="tests/templates"
 LAVA_JOBS_URL="https://${CIP_LAVA_LAB_SERVER:-lava.ciplatform.org}/scheduler/job"
-LAVA_API_URL="https://${CIP_LAVA_LAB_SERVER:-lava.ciplatform.org}/api/v0.2"
+LAVA_API_URL="https://$CIP_LAVA_LAB_USER:$CIP_LAVA_LAB_TOKEN@${CIP_LAVA_LAB_SERVER:-lava.ciplatform.org}/api/v0.2"
 LAVACLI_ARGS="--uri https://$CIP_LAVA_LAB_USER:$CIP_LAVA_LAB_TOKEN@${CIP_LAVA_LAB_SERVER:-lava.ciplatform.org}/RPC2"
 SQUAD_GROUP="cip-core"
 SQUAD_WATCH_JOBS_URL="${CIP_SQUAD_URL}/api/watchjob"
-- 
2.43.0

[isar-cip-core][PATCH 3/4] ci: Show live output of LAVA jobs

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This both simplifies the code massively and provides direct access to
the test progress in gitlab.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/submit_lava.sh | 164 ++++-------------------------------------
 1 file changed, 13 insertions(+), 151 deletions(-)

diff --git a/scripts/submit_lava.sh b/scripts/submit_lava.sh
index 48c5d3c1..6bf844bf 100755
--- a/scripts/submit_lava.sh
+++ b/scripts/submit_lava.sh
@@ -154,32 +154,22 @@ submit_job() {
 			return 1
 		else
 			echo "Job submitted successfully as #${ret}."
+			echo "URL: ${LAVA_JOBS_URL}/${ret}"
 
 			local lavacli_output=${job_dir}/lavacli_output
 			# shellcheck disable=2086
 			lavacli $LAVACLI_ARGS jobs show "${ret}" \
 				> "$lavacli_output"
 
-			STATUS=$(grep "state" "$lavacli_output" \
-				| cut -d ":" -f 2 \
-				| awk '{$1=$1};1')
-
-			HEALTH=$(grep "Health" "$lavacli_output" \
-				| cut -d ":" -f 2 \
-				| awk '{$1=$1};1')
-
 			DEVICE=$(grep "device      :" "$lavacli_output" \
 				| cut -d ":" -f 2 \
 				| awk '{$1=$1};1')
 
-			TESTING=$(grep "description" "$lavacli_output" \
-				| rev | cut -d "_" -f 1 | rev)
-
 			submit_squad_watch_job "${ret}" "${DEVICE}"
 
-			if ! check_status "$ret"; then
-				ERROR=true
-			fi
+			lavacli $LAVACLI_ARGS jobs logs "${ret}"
+			lavacli $LAVACLI_ARGS results "${ret}"
+
 			get_junit_test_results "$ret"
 		fi
 	else
@@ -227,151 +217,23 @@ validate_jobs () {
 	return $ret
 }
 
-check_if_finished () {
-	if [ "${STATUS}" != "Finished" ]; then
-		return 1
-	else
-		return 0
-	fi
-}
-
-check_for_test_error () {
-	if [ "${HEALTH}" != "Complete" ]; then
-		return 0
-	else
-		return 1
-	fi
-}
+get_first_xml_attr_value() {
+	file=${1}
+	tag=${2}
 
-# $1: LAVA job ID to show results for
-get_test_result () {
-	if [ -n "${1}" ]; then
-		# shellcheck disable=2086
-		lavacli "$LAVACLI_ARGS" results "${1}"
-	fi
+	grep -m 1 -o "${tag}=\".*\"" "${file}" | cut -d\" -f2
 }
 
 get_junit_test_results () {
 	mkdir -p "${RESULTS_DIR}"
 	curl -s -o "${RESULTS_DIR}"/results_"$1".xml "${LAVA_API_URL}"/jobs/"$1"/junit/
-}
 
-# $1: Test to print before job summary
-# $2: Set to true to print results for the job
-print_status () {
-	if [ -z "${1}" ]; then
-	# Set default text
-		local message="Current job status:"
-	else
-		local message="${1}"
+	# change return code to generate a error in gitlab-ci if a test is failed
+	errors=$(get_first_xml_attr_value "${RESULTS_DIR}"/results_"$1".xml errors)
+	failures=$(get_first_xml_attr_value "${RESULTS_DIR}"/results_"$1".xml failures)
+	if [ "${errors}" -gt "0" ] || [ "${failures}" -gt "0" ]; then
+		ERROR=true
 	fi
-
-	echo "------------------------------"
-	echo "${message}"
-	echo "------------------------------"
-	echo "Job #$2: ${STATUS}"
-	echo "Health: ${HEALTH}"
-	echo "Device: ${DEVICE}"
-	echo "Test: ${TESTING}"
-	echo "URL: ${LAVA_JOBS_URL}/$2"
-	if [ -n "${2}" ]; then
-		get_test_result "$2"
-	fi
-	echo " "
-}
-
-print_summary () {
-	echo "------------------------------"
-	echo "Job Summary"
-	echo "------------------------------"
-	echo "Job #${1} ${STATUS}. Job health: ${HEALTH}. URL: ${LAVA_JOBS_URL}/${1}"
-}
-
-check_status () {
-	local end_time status now
-	if [ -n "$TEST_TIMEOUT" ]; then
-		# Current time + timeout time
-		end_time=$(date +%s -d "+ $TEST_TIMEOUT min")
-	fi
-
-	local error=false
-
-	if [ -n "$1" ]; then
-		print_status "Current job status:"
-		while true
-		do
-			# Get latest status
-			if [ "${STATUS}" != "Finished" ]
-			then
-				local lavacli_output=${job_dir}/lavacli_output
-				# shellcheck disable=2086
-				lavacli $LAVACLI_ARGS jobs show "$1" \
-					> "$lavacli_output"
-
-				status=$(grep "state" "$lavacli_output" \
-					| cut -d ":" -f 2 \
-					| awk '{$1=$1};1')
-
-				HEALTH=$(grep "Health" "$lavacli_output" \
-					| cut -d ":" -f 2 \
-					| awk '{$1=$1};1')
-
-				DEVICE=$(grep "device      :" "$lavacli_output" \
-					| cut -d ":" -f 2 \
-					| awk '{$1=$1};1')
-
-				if [ "${STATUS}" != "$status" ]; then
-					STATUS=$status
-
-					# Something has changed
-					print_status "Current job status:" "$1"
-				else
-					STATUS=$status
-				fi
-			fi
-
-			if check_if_finished; then
-				break
-			fi
-
-			if [ -n "$TEST_TIMEOUT" ]; then
-				# Check timeout
-				now=$(date +%s)
-				if [ "$now" -ge "$end_time" ]; then
-					echo "Timed out waiting for test jobs to complete"
-					error=true
-					break
-				fi
-			fi
-
-			# Wait to avoid spamming the server too hard
-			sleep 60
-		done
-
-		if check_if_finished; then
-			# Print job outcome
-			print_status "Final job status:" "$1"
-
-			if check_for_test_error; then
-				error=true
-			fi
-		fi
-	fi
-
-	if $error; then
-		echo "---------------------"
-		echo "Errors during testing"
-		echo "---------------------"
-		print_summary "$1"
-		clean_up
-		return 1
-	fi
-
-	echo "-----------------------------------"
-	echo "Submitted test is successful"
-	echo "-----------------------------------"
-	print_summary "$1"
-	return 0
 }
 
 set_up
-- 
2.43.0

[isar-cip-core][PATCH 4/4] ci: submit_lava: Reduce validate_jobs to handle only a single one

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

We switched to one job per invocation already in 6bc117569001.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/submit_lava.sh | 32 ++++++++++++++------------------
 1 file changed, 14 insertions(+), 18 deletions(-)

diff --git a/scripts/submit_lava.sh b/scripts/submit_lava.sh
index 6bf844bf..092819cd 100755
--- a/scripts/submit_lava.sh
+++ b/scripts/submit_lava.sh
@@ -196,25 +196,20 @@ is_device_online () {
 	return 1
 }
 
-# This method is added with the intention to check if all the jobs are valid before submit
-# If even a single definition is found to be invalid, then no job shall be submitted until
-# it is fixed by the maintainer
-validate_jobs () {
-	local ret=0
-	for JOB in "${job_dir}"/*.yml; do
-		# shellcheck disable=2086
-		if lavacli $LAVACLI_ARGS jobs validate "$JOB"; then
-			echo "$JOB is a valid definition"
-			if ! submit_job $JOB; then
-				clean_up
-				exit 1
-			fi
-		else
-			echo "$JOB is not a valid definition"
-			ret=1
+# This method checks if the job is valid before submitting it later on.
+validate_job () {
+	# shellcheck disable=2086
+	if lavacli $LAVACLI_ARGS jobs validate "${job_dir}"/*.yml; then
+		echo "$JOB is a valid definition"
+		if ! submit_job $JOB; then
+			clean_up
+			exit 1
 		fi
-	done
-	return $ret
+	else
+		echo "$JOB is not a valid definition"
+		return 1
+	fi
+	return 0
 }
 
 get_first_xml_attr_value() {
@@ -237,6 +232,7 @@ get_junit_test_results () {
 }
 
 set_up
+
 create_job "$TEST" "$TARGET"
 
 if ! validate_jobs; then
-- 
2.43.0

[isar-cip-core][PATCH v2 4/4] ci: submit_lava: Reduce validate_jobs to handle only a single one

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

We switched to one job per invocation already in 6bc117569001.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---

This one now actually passed, except for the preexisting failures of IEC tests:
https://gitlab.com/cip-project/cip-core/isar-cip-core/-/pipelines

 scripts/submit_lava.sh | 35 ++++++++++++++++-------------------
 1 file changed, 16 insertions(+), 19 deletions(-)

diff --git a/scripts/submit_lava.sh b/scripts/submit_lava.sh
index 6bf844bf..7f151697 100755
--- a/scripts/submit_lava.sh
+++ b/scripts/submit_lava.sh
@@ -196,25 +196,21 @@ is_device_online () {
 	return 1
 }
 
-# This method is added with the intention to check if all the jobs are valid before submit
-# If even a single definition is found to be invalid, then no job shall be submitted until
-# it is fixed by the maintainer
-validate_jobs () {
-	local ret=0
-	for JOB in "${job_dir}"/*.yml; do
-		# shellcheck disable=2086
-		if lavacli $LAVACLI_ARGS jobs validate "$JOB"; then
-			echo "$JOB is a valid definition"
-			if ! submit_job $JOB; then
-				clean_up
-				exit 1
-			fi
-		else
-			echo "$JOB is not a valid definition"
-			ret=1
+# This method checks if the job is valid before submitting it later on.
+validate_job () {
+	# shellcheck disable=2086
+	local job=$(find "${job_dir}"/*.yml)
+	if lavacli $LAVACLI_ARGS jobs validate "${job}"; then
+		echo "$job is a valid definition"
+		if ! submit_job $job; then
+			clean_up
+			exit 1
 		fi
-	done
-	return $ret
+	else
+		echo "$job is not a valid definition"
+		return 1
+	fi
+	return 0
 }
 
 get_first_xml_attr_value() {
@@ -237,9 +233,10 @@ get_junit_test_results () {
 }
 
 set_up
+
 create_job "$TEST" "$TARGET"
 
-if ! validate_jobs; then
+if ! validate_job; then
 	clean_up
 	exit 1
 fi
-- 
2.43.0