From: Matthieu Baerts <matthieu.baerts@tessares.net>
To: mptcp@lists.linux.dev, Mat Martineau <martineau@kernel.org>,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@google.com>,
Jakub Kicinski <kuba@kernel.org>,
Paolo Abeni <pabeni@redhat.com>, Shuah Khan <shuah@kernel.org>
Cc: netdev@vger.kernel.org, linux-kselftest@vger.kernel.org,
linux-kernel@vger.kernel.org,
Matthieu Baerts <matthieu.baerts@tessares.net>
Subject: [PATCH net-next 13/13] selftests: mptcp: userspace_pm: format subtests results in TAP
Date: Mon, 17 Jul 2023 15:21:33 +0200 [thread overview]
Message-ID: <20230717-upstream-net-next-20230712-selftests-mptcp-subtests-v1-13-695127e0ad83@tessares.net> (raw)
In-Reply-To: <20230717-upstream-net-next-20230712-selftests-mptcp-subtests-v1-0-695127e0ad83@tessares.net>
The current selftests infrastructure formats the results in TAP 13. This
version doesn't support subtests and only the end result of each
selftest is taken into account. It means that a single issue in a
subtest of a selftest containing multiple subtests forces the whole
selftest to be marked as failed. It also means that subtests results are
not tracked by CIs executing selftests.
MPTCP selftests run hundreds of various subtests. It is then important
to track each of them and not one result per selftest.
It is particularly interesting to do that when validating stable kernels
with the last version of the test suite: tests might fail because a
feature is not supported but the test didn't skip that part. In this
case, if subtests are not tracked, the whole selftest will be marked as
failed making the other subtests useless because their results are
ignored.
This patch formats subtests results in TAP in userspace_pm.sh selftest.
Link: https://github.com/multipath-tcp/mptcp_net-next/issues/368
Acked-by: Paolo Abeni <pabeni@redhat.com>
Signed-off-by: Matthieu Baerts <matthieu.baerts@tessares.net>
---
tools/testing/selftests/net/mptcp/userspace_pm.sh | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/tools/testing/selftests/net/mptcp/userspace_pm.sh b/tools/testing/selftests/net/mptcp/userspace_pm.sh
index 568ddee1d102..23f8959a8ea8 100755
--- a/tools/testing/selftests/net/mptcp/userspace_pm.sh
+++ b/tools/testing/selftests/net/mptcp/userspace_pm.sh
@@ -59,6 +59,8 @@ rndh=$(printf %x "$sec")-$(mktemp -u XXXXXX)
ns1="ns1-$rndh"
ns2="ns2-$rndh"
ret=0
+test_name=""
+
_printf() {
stdbuf -o0 -e0 printf "${@}"
}
@@ -71,7 +73,9 @@ print_title()
# $1: test name
print_test()
{
- _printf "%-63s" "${1}"
+ test_name="${1}"
+
+ _printf "%-63s" "${test_name}"
}
print_results()
@@ -82,11 +86,13 @@ print_results()
test_pass()
{
print_results " OK "
+ mptcp_lib_result_pass "${test_name}"
}
test_skip()
{
print_results "SKIP"
+ mptcp_lib_result_skip "${test_name}"
}
# $1: msg
@@ -98,6 +104,8 @@ test_fail()
if [ -n "${1}" ]; then
_printf "\t%s\n" "${1}"
fi
+
+ mptcp_lib_result_fail "${test_name}"
}
kill_wait()
@@ -255,6 +263,7 @@ make_connection()
test_pass
else
test_fail "Expected tokens (c:${client_token} - s:${server_token}) and server (c:${client_serverside} - s:${server_serverside})"
+ mptcp_lib_result_print_all_tap
exit 1
fi
@@ -990,4 +999,5 @@ test_subflows_v4_v6_mix
test_prio
test_listener
+mptcp_lib_result_print_all_tap
exit ${ret}
--
2.40.1
next prev parent reply other threads:[~2023-07-17 13:21 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-17 13:21 [PATCH net-next 00/13] selftests: mptcp: format subtests results in TAP Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 01/13] selftests: mptcp: connect: don't stop if error Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 02/13] selftests: mptcp: userspace pm: " Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 03/13] selftests: mptcp: userspace_pm: fix shellcheck warnings Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 04/13] selftests: mptcp: userspace_pm: uniform results printing Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 05/13] selftests: mptcp: userspace_pm: reduce dup code around printf Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 06/13] selftests: mptcp: lib: format subtests results in TAP Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 07/13] selftests: mptcp: connect: " Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 08/13] selftests: mptcp: pm_netlink: " Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 09/13] selftests: mptcp: join: " Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 10/13] selftests: mptcp: diag: " Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 11/13] selftests: mptcp: simult flows: " Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 12/13] selftests: mptcp: sockopt: " Matthieu Baerts
2023-07-17 13:21 ` Matthieu Baerts [this message]
2023-07-19 10:20 ` [PATCH net-next 00/13] selftests: mptcp: " patchwork-bot+netdevbpf
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230717-upstream-net-next-20230712-selftests-mptcp-subtests-v1-13-695127e0ad83@tessares.net \
--to=matthieu.baerts@tessares.net \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=martineau@kernel.org \
--cc=mptcp@lists.linux.dev \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=shuah@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).