[PATCH v5 0/2] ci: link failure and leak annotations to the test script

["Harald Nordgren via GitGitGadget" <gitgitgadget@gmail.com>]

Link failure and leak annotations in CI to the test script, so both can be
found from the job summary.

V5 CI Job where failures and leaks are reported:
https://github.com/git/git/actions/runs/36979818141/job/110752027863?pr=2426

Changes in v5:

 * Removed % escaping entirely, verified on CI that it isn't needed. Every
   existing test description that uses % renders correctly unescaped.
 * Rewrote the file/line commit message with a concrete example (failed:
   t1060.17 partial clone of corrupted repository).

Changes in v4:

 * Clarify commit messages and simplify escaping logic.

Changes in v3:

 * Fixed bug in the --immediate exit ordering: the --immediate &&
   --invert-exit-code path called exit 0 before the test's annotation was
   written, now a single unconditional call covers both exit paths.
 * github_escape_message_ no longer relies on \r being a portable sed escape
   sequence (not POSIX-guaranteed and BSD sed implementations can differ),
   it splices in the literal carriage-return byte via printf instead.
 * Reverted unrelated test-tool line back to its original form.

Changes in v2:

 * Split into two commits, each explaining its own reasoning.
 * Leak output is no longer capped or embedded in the message, it's now an
   uncapped fold, so multiple leaks in the same test both show in full. A
   second leak in a different test still won't show in the same run,
   --immediate stops the script at the first failure, but it no longer gets
   buried under every later test falsely reporting "not ok" either.
 * Drops the giant unfolded message that annotations used to carry, which is
   what probably caused the scrolling behavior.

Harald Nordgren (2):
  ci: annotate leaks and stop a leak-sanitizer script at its first
    failure
  ci: point test failures and fixed known breakages at their file and
    line

 ci/lib.sh                            |  1 +
 t/test-lib-github-workflow-markup.sh | 46 ++++++++++++++++++++++++----
 t/test-lib.sh                        |  6 +++-
 3 files changed, 46 insertions(+), 7 deletions(-)


base-commit: c46c1e37724f0478939de636ab8ea5a89086d532
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2419%2FHaraldNordgren%2Fci-annotation-file-line-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2419/HaraldNordgren/ci-annotation-file-line-v5
Pull-Request: https://github.com/git/git/pull/2419

Range-diff vs v4:

 1:  138394b48b = 1:  851efeec8b ci: annotate leaks and stop a leak-sanitizer script at its first failure
 2:  8ec2b53d82 ! 2:  46f93a9e16 ci: point test failures and fixed known breakages at their file and line
     @@ Metadata
       ## Commit message ##
          ci: point test failures and fixed known breakages at their file and line
      
     -    A test failure or a fixed known breakage gets an annotation that names
     -    the test but says nothing about where it's defined, so a reviewer has
     -    to search the script by hand to find it.
     +    When a test fails, GitHub shows an annotation naming it, for example:
      
     -    Find the line a test is defined on by searching the script for its
     -    description as a fixed string, using the first match. Fall back to
     +        failed: t1060.17 partial clone of corrupted repository
     +
     +    but the location GitHub attaches to that annotation is the CI
     +    workflow file itself, not the test script, so there is nothing
     +    pointing at where the test actually lives.
     +
     +    Find the line a test is defined on by searching its script for the
     +    test's own description as a fixed string, using the first match, and
     +    attach that file and line to the annotation instead. Fall back to
          line 1 when the description is not found verbatim, which happens when
          a test builds its description at runtime instead of writing it out
          literally.
      
     -    A GitHub annotation is a single line, so a `%` in a test description
     -    has to be percent-encoded as `%25`, or GitHub misreads it as its own
     -    escape sequence. for-each-ref's format atoms use plenty of them, e.g.
     -    `%(raw)`.
     -
          Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
      
       ## t/test-lib-github-workflow-markup.sh ##
     @@ t/test-lib-github-workflow-markup.sh: start_test_output () {
       	github_markup_script_name=${0##*/}
       }
       
     -+github_escape_message_ () {
     -+	# % has to be escaped or GitHub misreads it as the start of its own
     -+	# percent-encoding (e.g. a literal %(raw) in a for-each-ref test
     -+	# description).
     -+	sed -e 's/%/%25/g'
     -+}
     -+
      +find_test_case_line_ () {
      +	# A description can contain characters like [ or * that would
      +	# corrupt a regex search, so match it literally and take the first
     @@ t/test-lib-github-workflow-markup.sh: github_annotation_ () {
      +	esac
      +
      +	test_case_line=$(find_test_case_line_ "$1")
     -+	test_case_description=$(printf '%s' "$1" | github_escape_message_)
      +
       	case "$test_case_result" in
       	failure)
      -		echo >>$github_markup_output "::error::failed: $this_test.$test_count $1"
      +		github_annotation_ error "t/$github_markup_script_name" "${test_case_line:-1}" \
     -+			"failed: $this_test.$test_count $test_case_description"
     ++			"failed: $this_test.$test_count $1"
       		;;
       	fixed)
      -		echo >>$github_markup_output "::notice::fixed: $this_test.$test_count $1"
     @@ t/test-lib-github-workflow-markup.sh: github_annotation_ () {
      -		# Exit without printing the "ok" or ""broken" tests
      -		return
      +		github_annotation_ notice "t/$github_markup_script_name" "${test_case_line:-1}" \
     -+			"fixed: $this_test.$test_count $test_case_description"
     ++			"fixed: $this_test.$test_count $1"
       		;;
       	esac
      +

-- 
gitgitgadget