From: Matthieu Baerts <matthieu.baerts@tessares.net>
To: mptcp@lists.linux.dev, Mat Martineau <martineau@kernel.org>,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@google.com>,
Jakub Kicinski <kuba@kernel.org>,
Paolo Abeni <pabeni@redhat.com>, Shuah Khan <shuah@kernel.org>
Cc: netdev@vger.kernel.org, linux-kselftest@vger.kernel.org,
linux-kernel@vger.kernel.org,
Matthieu Baerts <matthieu.baerts@tessares.net>
Subject: [PATCH net-next 10/13] selftests: mptcp: diag: format subtests results in TAP
Date: Mon, 17 Jul 2023 15:21:30 +0200 [thread overview]
Message-ID: <20230717-upstream-net-next-20230712-selftests-mptcp-subtests-v1-10-695127e0ad83@tessares.net> (raw)
In-Reply-To: <20230717-upstream-net-next-20230712-selftests-mptcp-subtests-v1-0-695127e0ad83@tessares.net>
The current selftests infrastructure formats the results in TAP 13. This
version doesn't support subtests and only the end result of each
selftest is taken into account. It means that a single issue in a
subtest of a selftest containing multiple subtests forces the whole
selftest to be marked as failed. It also means that subtests results are
not tracked by CIs executing selftests.
MPTCP selftests run hundreds of various subtests. It is then important
to track each of them and not one result per selftest.
It is particularly interesting to do that when validating stable kernels
with the last version of the test suite: tests might fail because a
feature is not supported but the test didn't skip that part. In this
case, if subtests are not tracked, the whole selftest will be marked as
failed making the other subtests useless because their results are
ignored.
This patch formats subtests results in TAP in diag.sh selftest.
Link: https://github.com/multipath-tcp/mptcp_net-next/issues/368
Acked-by: Paolo Abeni <pabeni@redhat.com>
Signed-off-by: Matthieu Baerts <matthieu.baerts@tessares.net>
---
tools/testing/selftests/net/mptcp/diag.sh | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/tools/testing/selftests/net/mptcp/diag.sh b/tools/testing/selftests/net/mptcp/diag.sh
index fa9e09ad97d9..85a8ee9395b3 100755
--- a/tools/testing/selftests/net/mptcp/diag.sh
+++ b/tools/testing/selftests/net/mptcp/diag.sh
@@ -65,12 +65,15 @@ __chk_nr()
if [ $nr != $expected ]; then
if [ $nr = "$skip" ] && ! mptcp_lib_expect_all_features; then
echo "[ skip ] Feature probably not supported"
+ mptcp_lib_result_skip "${msg}"
else
echo "[ fail ] expected $expected found $nr"
+ mptcp_lib_result_fail "${msg}"
ret=$test_cnt
fi
else
echo "[ ok ]"
+ mptcp_lib_result_pass "${msg}"
fi
test_cnt=$((test_cnt+1))
}
@@ -111,12 +114,15 @@ wait_msk_nr()
printf "%-50s" "$msg"
if [ $i -ge $timeout ]; then
echo "[ fail ] timeout while expecting $expected max $max last $nr"
+ mptcp_lib_result_fail "${msg} # timeout"
ret=$test_cnt
elif [ $nr != $expected ]; then
echo "[ fail ] expected $expected found $nr"
+ mptcp_lib_result_fail "${msg} # unexpected result"
ret=$test_cnt
else
echo "[ ok ]"
+ mptcp_lib_result_pass "${msg}"
fi
test_cnt=$((test_cnt+1))
}
@@ -276,4 +282,5 @@ flush_pids
chk_msk_inuse 0 "....chk 0 msk in use after flush"
+mptcp_lib_result_print_all_tap
exit $ret
--
2.40.1
next prev parent reply other threads:[~2023-07-17 13:21 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-17 13:21 [PATCH net-next 00/13] selftests: mptcp: format subtests results in TAP Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 01/13] selftests: mptcp: connect: don't stop if error Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 02/13] selftests: mptcp: userspace pm: " Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 03/13] selftests: mptcp: userspace_pm: fix shellcheck warnings Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 04/13] selftests: mptcp: userspace_pm: uniform results printing Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 05/13] selftests: mptcp: userspace_pm: reduce dup code around printf Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 06/13] selftests: mptcp: lib: format subtests results in TAP Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 07/13] selftests: mptcp: connect: " Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 08/13] selftests: mptcp: pm_netlink: " Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 09/13] selftests: mptcp: join: " Matthieu Baerts
2023-07-17 13:21 ` Matthieu Baerts [this message]
2023-07-17 13:21 ` [PATCH net-next 11/13] selftests: mptcp: simult flows: " Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 12/13] selftests: mptcp: sockopt: " Matthieu Baerts
2023-07-17 13:21 ` [PATCH net-next 13/13] selftests: mptcp: userspace_pm: " Matthieu Baerts
2023-07-19 10:20 ` [PATCH net-next 00/13] selftests: mptcp: " patchwork-bot+netdevbpf
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230717-upstream-net-next-20230712-selftests-mptcp-subtests-v1-10-695127e0ad83@tessares.net \
--to=matthieu.baerts@tessares.net \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=martineau@kernel.org \
--cc=mptcp@lists.linux.dev \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=shuah@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).