[PATCH 0/3] selftests: sync test conversion to TAP13

[PATCH 0/3] selftests: sync test conversion to TAP13

[Shuah Khan]

This patch series includes patches to convert sync test to use TAP13
ksft framework. In addition, fix to sync test to differentiate between
unsupported feature and access error when a non-root user runs it.

Updated kfst framework to return counters for sync test to use to print
the final pass or fail summary message. Updated ksft+print_cnts() to
print counters to summarize the test results.

Shuah Khan (3):
  selftests: sync: differentiate between sync unsupported and access
    errors
  selftests: kselftest framework: add API to return pass/fail/* counts
  selftests: sync: convert to use TAP13 ksft framework

 tools/testing/selftests/kselftest.h      | 10 +++++
 tools/testing/selftests/sync/sync_test.c | 71 +++++++++++++++++++++-----------
 tools/testing/selftests/sync/synctest.h  |  3 +-
 3 files changed, 58 insertions(+), 26 deletions(-)

-- 
2.11.0

[PATCH 1/3] selftests: sync: differentiate between sync unsupported and access errors

[Shuah Khan]

Sync test doesn't differentiate between sync unsupported and test run
by non-root user and treats both as unsupported cases.

Fix it to add handling for these two different scenarios.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/sync/sync_test.c | 23 +++++++++++++++++++----
 1 file changed, 19 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/sync/sync_test.c b/tools/testing/selftests/sync/sync_test.c
index 62fa666e501a..86ae45ad0347 100644
--- a/tools/testing/selftests/sync/sync_test.c
+++ b/tools/testing/selftests/sync/sync_test.c
@@ -31,6 +31,7 @@
 #include <sys/types.h>
 #include <sys/stat.h>
 #include <sys/wait.h>
+#include <errno.h>
 
 #include "synctest.h"
 
@@ -56,18 +57,32 @@ static int run_test(int (*test)(void), char *name)
 static int sync_api_supported(void)
 {
 	struct stat sbuf;
+	int ret;
+
+	ret = stat("/sys/kernel/debug/sync/sw_sync", &sbuf);
+	if (!ret)
+		return 0;
+
+	if (errno == ENOENT) {
+		printf("SKIP: Sync framework not supported by kernel\n");
+		exit(0);
+	}
+	if (errno == EACCES) {
+		printf("SKIP: Run Sync test as root.\n");
+		exit(0);
+	}
+
+	perror("stat");
+	exit(ret);
 
-	return 0 == stat("/sys/kernel/debug/sync/sw_sync", &sbuf);
 }
 
 int main(void)
 {
 	int err = 0;
 
-	if (!sync_api_supported()) {
-		printf("SKIP: Sync framework not supported by kernel\n");
+	if (!sync_api_supported())
 		return 0;
-	}
 
 	printf("[RUN]\tTesting sync framework\n");
 
-- 
2.11.0

[PATCH 2/3] selftests: kselftest framework: add API to return pass/fail/* counts

[Shuah Khan]

Some tests print final pass/fail message based on fail count. Add
ksft_get_*_cnt() API to kselftest framework to return counts.

Update ksft_print_cnts() to print the test results summary message with
individual pass, fail, ... counters.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/kselftest.h | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/tools/testing/selftests/kselftest.h b/tools/testing/selftests/kselftest.h
index 08e90c2cc5cb..45bf25905279 100644
--- a/tools/testing/selftests/kselftest.h
+++ b/tools/testing/selftests/kselftest.h
@@ -45,6 +45,12 @@ static inline void ksft_inc_xfail_cnt(void) { ksft_cnt.ksft_xfail++; }
 static inline void ksft_inc_xpass_cnt(void) { ksft_cnt.ksft_xpass++; }
 static inline void ksft_inc_xskip_cnt(void) { ksft_cnt.ksft_xskip++; }
 
+static inline int ksft_get_pass_cnt(void) { return ksft_cnt.ksft_pass; }
+static inline int ksft_get_fail_cnt(void) { return ksft_cnt.ksft_fail; }
+static inline int ksft_get_xfail_cnt(void) { return ksft_cnt.ksft_xfail; }
+static inline int ksft_get_xpass_cnt(void) { return ksft_cnt.ksft_xpass; }
+static inline int ksft_get_xskip_cnt(void) { return ksft_cnt.ksft_xskip; }
+
 static inline void ksft_print_header(void)
 {
 	printf("TAP version 13\n");
@@ -52,6 +58,10 @@ static inline void ksft_print_header(void)
 
 static inline void ksft_print_cnts(void)
 {
+	printf("Pass %d Fail %d Xfail %d Xpass %d Skip %d\n",
+		ksft_cnt.ksft_pass, ksft_cnt.ksft_fail,
+		ksft_cnt.ksft_xfail, ksft_cnt.ksft_xpass,
+		ksft_cnt.ksft_xskip);
 	printf("1..%d\n", ksft_test_num());
 }
 
-- 
2.11.0

[PATCH 3/3] selftests: sync: convert to use TAP13 ksft framework

[Shuah Khan]

Convert test to use TAP13 ksft framework. Output after conversion:

TAP version 13
# [RUN]	Testing sync framework
ok 1 [RUN]	test_alloc_timeline
ok 2 [RUN]	test_alloc_fence
ok 3 [RUN]	test_alloc_fence_negative
ok 4 [RUN]	test_fence_one_timeline_wait
ok 5 [RUN]	test_fence_one_timeline_merge
ok 6 [RUN]	test_fence_merge_same_fence
ok 7 [RUN]	test_fence_multi_timeline_wait
ok 8 [RUN]	test_stress_two_threads_shared_timeline
ok 9 [RUN]	test_consumer_stress_multi_producer_single_consumer
ok 10 [RUN]	test_merge_stress_random_merge
Pass 10 Fail 0 Xfail 0 Xpass 0 Skip 0
1..10

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/sync/sync_test.c | 72 +++++++++++++++++---------------
 tools/testing/selftests/sync/synctest.h  |  3 +-
 2 files changed, 41 insertions(+), 34 deletions(-)

diff --git a/tools/testing/selftests/sync/sync_test.c b/tools/testing/selftests/sync/sync_test.c
index 86ae45ad0347..7f7938263c5c 100644
--- a/tools/testing/selftests/sync/sync_test.c
+++ b/tools/testing/selftests/sync/sync_test.c
@@ -32,76 +32,82 @@
 #include <sys/stat.h>
 #include <sys/wait.h>
 #include <errno.h>
+#include <string.h>
 
+#include "../kselftest.h"
 #include "synctest.h"
 
 static int run_test(int (*test)(void), char *name)
 {
 	int result;
 	pid_t childpid;
+	int ret;
 
 	fflush(stdout);
 	childpid = fork();
 
 	if (childpid) {
 		waitpid(childpid, &result, 0);
-		if (WIFEXITED(result))
-			return WEXITSTATUS(result);
+		if (WIFEXITED(result)) {
+			ret = WEXITSTATUS(result);
+			if (!ret)
+				ksft_test_result_pass("[RUN]\t%s\n", name);
+			else
+				ksft_test_result_fail("[RUN]\t%s\n", name);
+			return ret;
+		}
 		return 1;
 	}
 
-	printf("[RUN]\tExecuting %s\n", name);
 	exit(test());
 }
 
-static int sync_api_supported(void)
+static void sync_api_supported(void)
 {
 	struct stat sbuf;
 	int ret;
 
 	ret = stat("/sys/kernel/debug/sync/sw_sync", &sbuf);
 	if (!ret)
-		return 0;
+		return;
 
-	if (errno == ENOENT) {
-		printf("SKIP: Sync framework not supported by kernel\n");
-		exit(0);
-	}
-	if (errno == EACCES) {
-		printf("SKIP: Run Sync test as root.\n");
-		exit(0);
-	}
+	if (errno == ENOENT)
+		ksft_exit_skip("Sync framework not supported by kernel\n");
 
-	perror("stat");
-	exit(ret);
+	if (errno == EACCES)
+		ksft_exit_skip("Run Sync test as root.\n");
 
+	ksft_exit_fail_msg("stat failed on /sys/kernel/debug/sync/sw_sync: %s",
+				strerror(errno));
 }
 
 int main(void)
 {
-	int err = 0;
+	int err;
+
+	ksft_print_header();
 
-	if (!sync_api_supported())
-		return 0;
+	sync_api_supported();
 
-	printf("[RUN]\tTesting sync framework\n");
+	ksft_print_msg("[RUN]\tTesting sync framework\n");
 
-	err += RUN_TEST(test_alloc_timeline);
-	err += RUN_TEST(test_alloc_fence);
-	err += RUN_TEST(test_alloc_fence_negative);
+	RUN_TEST(test_alloc_timeline);
+	RUN_TEST(test_alloc_fence);
+	RUN_TEST(test_alloc_fence_negative);
 
-	err += RUN_TEST(test_fence_one_timeline_wait);
-	err += RUN_TEST(test_fence_one_timeline_merge);
-	err += RUN_TEST(test_fence_merge_same_fence);
-	err += RUN_TEST(test_fence_multi_timeline_wait);
-	err += RUN_TEST(test_stress_two_threads_shared_timeline);
-	err += RUN_TEST(test_consumer_stress_multi_producer_single_consumer);
-	err += RUN_TEST(test_merge_stress_random_merge);
+	RUN_TEST(test_fence_one_timeline_wait);
+	RUN_TEST(test_fence_one_timeline_merge);
+	RUN_TEST(test_fence_merge_same_fence);
+	RUN_TEST(test_fence_multi_timeline_wait);
+	RUN_TEST(test_stress_two_threads_shared_timeline);
+	RUN_TEST(test_consumer_stress_multi_producer_single_consumer);
+	RUN_TEST(test_merge_stress_random_merge);
 
+	err = ksft_get_fail_cnt();
 	if (err)
-		printf("[FAIL]\tsync errors: %d\n", err);
-	else
-		printf("[OK]\tsync\n");
+		ksft_exit_fail_msg("%d out of %d sync tests failed\n",
+					err, ksft_test_num());
 
-	return !!err;
+	/* need this return to keep gcc happy */
+	return ksft_exit_pass();
 }
diff --git a/tools/testing/selftests/sync/synctest.h b/tools/testing/selftests/sync/synctest.h
index e7d1d57dba7a..90a8e5369914 100644
--- a/tools/testing/selftests/sync/synctest.h
+++ b/tools/testing/selftests/sync/synctest.h
@@ -29,10 +29,11 @@
 #define SELFTESTS_SYNCTEST_H
 
 #include <stdio.h>
+#include "../kselftest.h"
 
 #define ASSERT(cond, msg) do { \
 	if (!(cond)) { \
-		printf("[ERROR]\t%s", (msg)); \
+		ksft_print_msg("[ERROR]\t%s", (msg)); \
 		return 1; \
 	} \
 } while (0)
-- 
2.11.0

Re: [PATCH 0/3] selftests: sync test conversion to TAP13

[Gustavo Padovan]

Hi Shuah,

Thank you for your patches.

On Mon, 2017-07-24 at 15:07 -0600, Shuah Khan wrote:
> This patch series includes patches to convert sync test to use TAP13
> ksft framework. In addition, fix to sync test to differentiate
> between
> unsupported feature and access error when a non-root user runs it.
> 
> Updated kfst framework to return counters for sync test to use to
> print
> the final pass or fail summary message. Updated ksft+print_cnts() to
> print counters to summarize the test results.
> 
> Shuah Khan (3):
>   selftests: sync: differentiate between sync unsupported and access
>     errors
>   selftests: kselftest framework: add API to return pass/fail/*
> counts
>   selftests: sync: convert to use TAP13 ksft framework
> 
>  tools/testing/selftests/kselftest.h      | 10 +++++
>  tools/testing/selftests/sync/sync_test.c | 71 +++++++++++++++++++++-
> ----------
>  tools/testing/selftests/sync/synctest.h  |  3 +-
>  3 files changed, 58 insertions(+), 26 deletions(-)

All looks good to me, so for the 3 patches:
Reviewed-by: Gustavo Padovan <gustavo.padovan@collabora.com>

	Gustavo

-- 
Gustavo Padovan
Collabora Ltd.

Re: [PATCH 0/3] selftests: sync test conversion to TAP13

[Shuah Khan]

On 07/28/2017 12:25 PM, Gustavo Padovan wrote:
> Hi Shuah,
> 
> Thank you for your patches.
> 
> On Mon, 2017-07-24 at 15:07 -0600, Shuah Khan wrote:
>> This patch series includes patches to convert sync test to use TAP13
>> ksft framework. In addition, fix to sync test to differentiate
>> between
>> unsupported feature and access error when a non-root user runs it.
>>
>> Updated kfst framework to return counters for sync test to use to
>> print
>> the final pass or fail summary message. Updated ksft+print_cnts() to
>> print counters to summarize the test results.
>>
>> Shuah Khan (3):
>>   selftests: sync: differentiate between sync unsupported and access
>>     errors
>>   selftests: kselftest framework: add API to return pass/fail/*
>> counts
>>   selftests: sync: convert to use TAP13 ksft framework
>>
>>  tools/testing/selftests/kselftest.h      | 10 +++++
>>  tools/testing/selftests/sync/sync_test.c | 71 +++++++++++++++++++++-
>> ----------
>>  tools/testing/selftests/sync/synctest.h  |  3 +-
>>  3 files changed, 58 insertions(+), 26 deletions(-)
> 
> All looks good to me, so for the 3 patches:
> Reviewed-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> 	Gustavo
> 

Thanks Gustavo. I will get them in to linux-ksefltest next for 4.14-rc1
with your Reviwed-by.

-- Shuah