* [LTP] [PATCH 1/3] lib/tst_net.sh: print all netstress results in tst_netload()
2021-02-02 13:34 [LTP] [PATCH 0/3] lib/tst_net.sh: calc median instead of mean in tst_netload() Alexey Kodanev
@ 2021-02-02 13:34 ` Alexey Kodanev
2021-02-05 6:59 ` Petr Vorel
2021-02-02 13:34 ` [LTP] [PATCH 2/3] lib: add tst_get_median helper binary for use in script tests Alexey Kodanev
` (2 subsequent siblings)
3 siblings, 1 reply; 9+ messages in thread
From: Alexey Kodanev @ 2021-02-02 13:34 UTC (permalink / raw)
To: ltp
This helps during debugging when you need to see from which
dataset the final result was obtained (mean).
Signed-off-by: Alexey Kodanev <alexey.kodanev@oracle.com>
---
testcases/lib/tst_net.sh | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/testcases/lib/tst_net.sh b/testcases/lib/tst_net.sh
index ef9354903..f1a498306 100644
--- a/testcases/lib/tst_net.sh
+++ b/testcases/lib/tst_net.sh
@@ -710,7 +710,7 @@ tst_netload()
tst_rhost_run -c "pkill -9 netstress\$"
rm -f tst_netload.log
- local res=0
+ local results
local passed=0
for i in $(seq 1 $run_cnt); do
@@ -751,7 +751,7 @@ tst_netload()
[ ! -f $rfile ] && \
tst_netload_brk TFAIL "can't read $rfile"
- res="$((res + $(cat $rfile)))"
+ results="$results $(cat $rfile)"
passed=$((passed + 1))
done
@@ -761,10 +761,14 @@ tst_netload()
tst_netload_brk TFAIL "expected '$expect_res' but ret: '$ret'"
fi
- res=$((res / $passed))
- echo "$res" > $rfile
+ local mean res_sum
+ for r in $results; do
+ res_sum="$((res_sum + r))"
+ done
+ mean=$((res_sum / passed))
+ echo "$mean" > $rfile
- tst_res_ TPASS "netstress passed, mean time '$res' ms"
+ tst_res_ TPASS "netstress passed, mean time $mean ms, data:$results"
return $ret
}
--
2.25.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* [LTP] [PATCH 2/3] lib: add tst_get_median helper binary for use in script tests
2021-02-02 13:34 [LTP] [PATCH 0/3] lib/tst_net.sh: calc median instead of mean in tst_netload() Alexey Kodanev
2021-02-02 13:34 ` [LTP] [PATCH 1/3] lib/tst_net.sh: print all netstress results " Alexey Kodanev
@ 2021-02-02 13:34 ` Alexey Kodanev
2021-02-05 10:15 ` Petr Vorel
2021-02-02 13:34 ` [LTP] [PATCH 3/3] lib/tst_net.sh: calc median instead of mean in tst_netload() Alexey Kodanev
2021-02-05 10:21 ` [LTP] [PATCH 0/3] " Petr Vorel
3 siblings, 1 reply; 9+ messages in thread
From: Alexey Kodanev @ 2021-02-02 13:34 UTC (permalink / raw)
To: ltp
/tst_get_median 10 11 300 8 9 14
output: 10
Signed-off-by: Alexey Kodanev <alexey.kodanev@oracle.com>
---
testcases/lib/.gitignore | 1 +
testcases/lib/Makefile | 3 ++-
testcases/lib/tst_get_median.c | 37 ++++++++++++++++++++++++++++++++++
3 files changed, 40 insertions(+), 1 deletion(-)
create mode 100644 testcases/lib/tst_get_median.c
diff --git a/testcases/lib/.gitignore b/testcases/lib/.gitignore
index 52f99dc45..bc299b6ee 100644
--- a/testcases/lib/.gitignore
+++ b/testcases/lib/.gitignore
@@ -2,6 +2,7 @@
/tst_checkpoint
/tst_device
/tst_getconf
+/tst_get_median
/tst_get_unused_port
/tst_kvcmp
/tst_net_iface_prefix
diff --git a/testcases/lib/Makefile b/testcases/lib/Makefile
index 4616e24c0..f77da0d56 100644
--- a/testcases/lib/Makefile
+++ b/testcases/lib/Makefile
@@ -28,6 +28,7 @@ INSTALL_TARGETS := *.sh
MAKE_TARGETS := tst_sleep tst_random tst_checkpoint tst_rod tst_kvcmp\
tst_device tst_net_iface_prefix tst_net_ip_prefix tst_net_vars\
- tst_getconf tst_supported_fs tst_check_drivers tst_get_unused_port
+ tst_getconf tst_supported_fs tst_check_drivers tst_get_unused_port\
+ tst_get_median
include $(top_srcdir)/include/mk/generic_leaf_target.mk
diff --git a/testcases/lib/tst_get_median.c b/testcases/lib/tst_get_median.c
new file mode 100644
index 000000000..5246f12e0
--- /dev/null
+++ b/testcases/lib/tst_get_median.c
@@ -0,0 +1,37 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/* Copyright (c) 2021 Oracle and/or its affiliates. All Rights Reserved. */
+
+#include <stdio.h>
+#include <stdlib.h>
+
+static int cmp(const void *a, const void *b)
+{
+ return (*(int *)a - *(int *)b);
+}
+
+int main(int argc, const char *argv[])
+{
+ const size_t size = argc - 1;
+
+ if (!size) {
+ fprintf(stderr, "Please provide a numeric list\n");
+ return 1;
+ }
+ if (size == 1) {
+ printf("%d", atoi(argv[1]));
+ return 0;
+ }
+
+ int arr[size];
+ size_t i;
+
+ for (i = 0; i < size; ++i)
+ arr[i] = atoi(argv[i + 1]);
+
+ qsort(arr, size, sizeof(arr[0]), cmp);
+
+ const size_t size2 = size / 2;
+ printf("%d", (size & 1) ? arr[size2] : ((arr[size2 - 1] + arr[size2]) / 2));
+
+ return 0;
+}
--
2.25.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* [LTP] [PATCH 3/3] lib/tst_net.sh: calc median instead of mean in tst_netload()
2021-02-02 13:34 [LTP] [PATCH 0/3] lib/tst_net.sh: calc median instead of mean in tst_netload() Alexey Kodanev
2021-02-02 13:34 ` [LTP] [PATCH 1/3] lib/tst_net.sh: print all netstress results " Alexey Kodanev
2021-02-02 13:34 ` [LTP] [PATCH 2/3] lib: add tst_get_median helper binary for use in script tests Alexey Kodanev
@ 2021-02-02 13:34 ` Alexey Kodanev
2021-02-05 10:17 ` Petr Vorel
2021-02-05 10:21 ` [LTP] [PATCH 0/3] " Petr Vorel
3 siblings, 1 reply; 9+ messages in thread
From: Alexey Kodanev @ 2021-02-02 13:34 UTC (permalink / raw)
To: ltp
Sometimes the tests can get the following results on a test network:
gre01 1 TINFO: run server 'netstress -D ltp_v0...
gre01 1 TINFO: run client 'netstress -l -D ltp_v0... 5 times
gre01 1 TPASS: netstress passed, mean time 4633 ms, data: 128 22627 134 142 137
...
vxlan03 1 TINFO: run server 'netstress -D ltp_v0...
vxlan03 1 TINFO: run client 'netstress -l -D ltp_v0... 5 times
vxlan03 1 TPASS: netstress passed, mean time 4584 ms, data: 142 140 146 145 22350
One unsuccessful run can have a huge impact on the final result,
when using the mean time with such data.
A more suitable solution for short runs would be to obtain a median
time that can remove all outliers. This will lead to more consistent
performance test results. For example, instead of the above runs, we
would get this:
gre01 1 TPASS: netstress passed, median time 137 ms, data: 128 22627 134 142 137
vxlan03 1 TPASS: netstress passed, median time 145 ms, data: 142 140 146 145 22350
Signed-off-by: Alexey Kodanev <alexey.kodanev@oracle.com>
---
testcases/lib/tst_net.sh | 10 +++-------
1 file changed, 3 insertions(+), 7 deletions(-)
diff --git a/testcases/lib/tst_net.sh b/testcases/lib/tst_net.sh
index f1a498306..ca21fe326 100644
--- a/testcases/lib/tst_net.sh
+++ b/testcases/lib/tst_net.sh
@@ -761,14 +761,10 @@ tst_netload()
tst_netload_brk TFAIL "expected '$expect_res' but ret: '$ret'"
fi
- local mean res_sum
- for r in $results; do
- res_sum="$((res_sum + r))"
- done
- mean=$((res_sum / passed))
- echo "$mean" > $rfile
+ local median=$(tst_get_median $results)
+ echo "$median" > $rfile
- tst_res_ TPASS "netstress passed, mean time $mean ms, data:$results"
+ tst_res_ TPASS "netstress passed, median time $median ms, data:$results"
return $ret
}
--
2.25.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [LTP] [PATCH 3/3] lib/tst_net.sh: calc median instead of mean in tst_netload()
2021-02-02 13:34 ` [LTP] [PATCH 3/3] lib/tst_net.sh: calc median instead of mean in tst_netload() Alexey Kodanev
@ 2021-02-05 10:17 ` Petr Vorel
0 siblings, 0 replies; 9+ messages in thread
From: Petr Vorel @ 2021-02-05 10:17 UTC (permalink / raw)
To: ltp
Hi Alexey,
> Sometimes the tests can get the following results on a test network:
> gre01 1 TINFO: run server 'netstress -D ltp_v0...
> gre01 1 TINFO: run client 'netstress -l -D ltp_v0... 5 times
> gre01 1 TPASS: netstress passed, mean time 4633 ms, data: 128 22627 134 142 137
> ...
> vxlan03 1 TINFO: run server 'netstress -D ltp_v0...
> vxlan03 1 TINFO: run client 'netstress -l -D ltp_v0... 5 times
> vxlan03 1 TPASS: netstress passed, mean time 4584 ms, data: 142 140 146 145 22350
> One unsuccessful run can have a huge impact on the final result,
> when using the mean time with such data.
> A more suitable solution for short runs would be to obtain a median
> time that can remove all outliers. This will lead to more consistent
> performance test results. For example, instead of the above runs, we
> would get this:
> gre01 1 TPASS: netstress passed, median time 137 ms, data: 128 22627 134 142 137
> vxlan03 1 TPASS: netstress passed, median time 145 ms, data: 142 140 146 145 22350
Reviewed-by: Petr Vorel <pvorel@suse.cz>
Kind regards,
Petr
^ permalink raw reply [flat|nested] 9+ messages in thread
* [LTP] [PATCH 0/3] lib/tst_net.sh: calc median instead of mean in tst_netload()
2021-02-02 13:34 [LTP] [PATCH 0/3] lib/tst_net.sh: calc median instead of mean in tst_netload() Alexey Kodanev
` (2 preceding siblings ...)
2021-02-02 13:34 ` [LTP] [PATCH 3/3] lib/tst_net.sh: calc median instead of mean in tst_netload() Alexey Kodanev
@ 2021-02-05 10:21 ` Petr Vorel
2021-02-08 15:24 ` Alexey Kodanev
3 siblings, 1 reply; 9+ messages in thread
From: Petr Vorel @ 2021-02-05 10:21 UTC (permalink / raw)
To: ltp
Hi Alexey,
> This series of patches is intended to improve the reliability
> of results in network tests that using tst_netload().
> After TST_NETLOAD_RUN_COUNT was added, we get several results (5 by
> default). The first patch allows all of them to be displayed along
> with the final result. It helps to debug test performance failures.
> Further patches replace the mean calculation of such data set with
> the median one. It allows to exclude extreme points from the final
> result.
Thanks for improving stability of these tests!
I've also noticed often failures in net.features and some of ipsec tests,
this should help.
FYI as TODO I'd like some of net.features have also as functional tests
(ignoring the performance).
Kind regards,
Petr
^ permalink raw reply [flat|nested] 9+ messages in thread* [LTP] [PATCH 0/3] lib/tst_net.sh: calc median instead of mean in tst_netload()
2021-02-05 10:21 ` [LTP] [PATCH 0/3] " Petr Vorel
@ 2021-02-08 15:24 ` Alexey Kodanev
0 siblings, 0 replies; 9+ messages in thread
From: Alexey Kodanev @ 2021-02-08 15:24 UTC (permalink / raw)
To: ltp
On 05.02.2021 13:21, Petr Vorel wrote:
> Hi Alexey,
>
>> This series of patches is intended to improve the reliability
>> of results in network tests that using tst_netload().
>
>> After TST_NETLOAD_RUN_COUNT was added, we get several results (5 by
>> default). The first patch allows all of them to be displayed along
>> with the final result. It helps to debug test performance failures.
>
>> Further patches replace the mean calculation of such data set with
>> the median one. It allows to exclude extreme points from the final
>> result.
>
> Thanks for improving stability of these tests!
> I've also noticed often failures in net.features and some of ipsec tests,
> this should help.
>
> FYI as TODO I'd like some of net.features have also as functional tests
> (ignoring the performance).
>
Merged, thanks for review Petr!
^ permalink raw reply [flat|nested] 9+ messages in thread