[PATCH] tools: selftests - create a separate hotplug target

[PATCH] tools: selftests - create a separate hotplug target

[Shuah Khan]

On some systems, hotplug tests could hang forever waiting for cpu and
memory to be ready to be offlined. A special hotplug target is created,
which will help run non-hotplug tests and run hotplug tests as a special
case. Individual hotplug tests can still be run as a special target
targeted for a single subsystem.

Signed-off-by: Shuah Khan <shuah.kh@samsung.com>
---
 tools/testing/selftests/Makefile   | 20 ++++++++++++++++++--
 tools/testing/selftests/README.txt | 24 +++++++++++++++++++++---
 2 files changed, 39 insertions(+), 5 deletions(-)

diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index e66e710..8b3f029 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -1,8 +1,6 @@
 TARGETS = breakpoints
-TARGETS += cpu-hotplug
 TARGETS += efivarfs
 TARGETS += kcmp
-TARGETS += memory-hotplug
 TARGETS += mqueue
 TARGETS += net
 TARGETS += ptrace
@@ -12,6 +10,9 @@ TARGETS += powerpc
 TARGETS += user
 TARGETS += sysctl
 
+TARGETS_HOTPLUG = cpu-hotplug
+TARGETS_HOTPLUG += memory-hotplug
+
 all:
 	for TARGET in $(TARGETS); do \
 		make -C $$TARGET; \
@@ -22,6 +23,21 @@ run_tests: all
 		make -C $$TARGET run_tests; \
 	done;
 
+hotplug:
+	for TARGET in $(TARGETS_HOTPLUG); do \
+		make -C $$TARGET; \
+	done;
+
+run_hotplug: hotplug
+	for TARGET in $(TARGETS_HOTPLUG); do \
+		make -C $$TARGET run_tests; \
+	done;
+
+clean_hotplug:
+	for TARGET in $(TARGETS_HOTPLUG); do \
+		make -C $$TARGET clean; \
+	done;
+
 clean:
 	for TARGET in $(TARGETS); do \
 		make -C $$TARGET clean; \
diff --git a/tools/testing/selftests/README.txt b/tools/testing/selftests/README.txt
index 5e2faf9..af050cd 100644
--- a/tools/testing/selftests/README.txt
+++ b/tools/testing/selftests/README.txt
@@ -4,8 +4,13 @@ The kernel contains a set of "self tests" under the tools/testing/selftests/
 directory. These are intended to be small unit tests to exercise individual
 code paths in the kernel.
 
-Running the selftests
-=====================
+On some systems, hotplug tests could hang forever waiting for cpu and
+memory to be ready to be offlined. A special hotplug target is created,
+which will help run non-hotplug tests and run hotplug tests as a special
+case.
+
+Running the selftests (non-hotplug tests)
+=========================================
 
 To build the tests:
 
@@ -19,13 +24,26 @@ To run the tests:
 - note that some tests will require root privileges.
 
 
-To run only tests targetted for a single subsystem:
+To run only tests targeted for a single subsystem: (including
+hotplug targets)
 
   $  make -C tools/testing/selftests TARGETS=cpu-hotplug run_tests
 
 See the top-level tools/testing/selftests/Makefile for the list of all possible
 targets.
 
+Running the hotplug selftests
+==============================
+
+To build the tests:
+
+  $ make -C tools/testing/selftests hotplug
+
+To run the tests:
+
+  $ make -C tools/testing/selftests run_hotplug
+
+- note that some tests will require root privileges.
 
 Contributing new tests
 ======================
-- 
1.9.1

Re: [PATCH] tools: selftests - create a separate hotplug target

[Andrew Morton]

On Thu, 26 Jun 2014 14:33:56 -0600 Shuah Khan <shuah.kh@samsung.com> wrote:

> On some systems, hotplug tests could hang forever waiting for cpu and
> memory to be ready to be offlined. A special hotplug target is created,
> which will help run non-hotplug tests and run hotplug tests as a special
> case. Individual hotplug tests can still be run as a special target
> targeted for a single subsystem.

This is a bit sad.  The general philosophy with selftests is that they
should run to completion even if the kernel/hardware which they are
testing isn't available - they should work it out for themselves.

But that's obviously a problem with hotplug.  And with networking or
anything else which needs external action.

On the other hand, networking has loopback and the kernel supports cpu
hotplug simulation via procfs.  So perhaps the cpu and memory hotplug
tests should be redone so they do the plug/unplug injection themselves,
so they can run without external intervention?

Failing that, all I can think of is timeouts or some silly "press any
key to continue" operator intervention.

Re: [PATCH] tools: selftests - create a separate hotplug target

[Shuah Khan]

On 06/26/2014 03:51 PM, Andrew Morton wrote:
> On Thu, 26 Jun 2014 14:33:56 -0600 Shuah Khan <shuah.kh@samsung.com> wrote:
>
>> On some systems, hotplug tests could hang forever waiting for cpu and
>> memory to be ready to be offlined. A special hotplug target is created,
>> which will help run non-hotplug tests and run hotplug tests as a special
>> case. Individual hotplug tests can still be run as a special target
>> targeted for a single subsystem.
>
> This is a bit sad.  The general philosophy with selftests is that they
> should run to completion even if the kernel/hardware which they are
> testing isn't available - they should work it out for themselves.
>
> But that's obviously a problem with hotplug.  And with networking or
> anything else which needs external action.
>
> On the other hand, networking has loopback and the kernel supports cpu
> hotplug simulation via procfs.  So perhaps the cpu and memory hotplug
> tests should be redone so they do the plug/unplug injection themselves,
> so they can run without external intervention?

Changing/running the tests in a safe mode (least possibility of hang)
mode is another option. This way the tests are run in normal mode with
reduced scope. Memory hotplug test has the ratio option and when I
specified low ratio 1-5%, it completed in a few seconds.

cpu-hotplug test will require changes. I am working on a change to
offline a user specified # of cpus instead offlining all hotpluggable
cpus and then onlining them again at the end of the test.

When all selftests are run, safe mode hotplug tests will be run.

Does this approach sound reasonable?

-- Shuah

-- 
Shuah Khan
Senior Linux Kernel Developer - Open Source Group
Samsung Research America(Silicon Valley)
shuah.kh@samsung.com | (970) 672-0658

Re: [PATCH] tools: selftests - create a separate hotplug target

[Andrew Morton]

On Fri, 27 Jun 2014 11:10:37 -0600 Shuah Khan <shuah.kh@samsung.com> wrote:

> On 06/26/2014 03:51 PM, Andrew Morton wrote:
> > On Thu, 26 Jun 2014 14:33:56 -0600 Shuah Khan <shuah.kh@samsung.com> wrote:
> >
> >> On some systems, hotplug tests could hang forever waiting for cpu and
> >> memory to be ready to be offlined. A special hotplug target is created,
> >> which will help run non-hotplug tests and run hotplug tests as a special
> >> case. Individual hotplug tests can still be run as a special target
> >> targeted for a single subsystem.
> >
> > This is a bit sad.  The general philosophy with selftests is that they
> > should run to completion even if the kernel/hardware which they are
> > testing isn't available - they should work it out for themselves.
> >
> > But that's obviously a problem with hotplug.  And with networking or
> > anything else which needs external action.
> >
> > On the other hand, networking has loopback and the kernel supports cpu
> > hotplug simulation via procfs.  So perhaps the cpu and memory hotplug
> > tests should be redone so they do the plug/unplug injection themselves,
> > so they can run without external intervention?
> 
> Changing/running the tests in a safe mode (least possibility of hang)
> mode is another option. This way the tests are run in normal mode with
> reduced scope. Memory hotplug test has the ratio option and when I
> specified low ratio 1-5%, it completed in a few seconds.
> 
> cpu-hotplug test will require changes. I am working on a change to
> offline a user specified # of cpus instead offlining all hotpluggable
> cpus and then onlining them again at the end of the test.
> 
> When all selftests are run, safe mode hotplug tests will be run.
> 
> Does this approach sound reasonable?

I don't know really.  You know more about this than I - what advantages
does the separate-make-target approach have over this approach?

Re: [PATCH] tools: selftests - create a separate hotplug target

[Shuah Khan]

On 06/27/2014 01:45 PM, Andrew Morton wrote:
> On Fri, 27 Jun 2014 11:10:37 -0600 Shuah Khan <shuah.kh@samsung.com> wrote:
>
>> On 06/26/2014 03:51 PM, Andrew Morton wrote:
>>> On Thu, 26 Jun 2014 14:33:56 -0600 Shuah Khan <shuah.kh@samsung.com> wrote:
>>>
>>>> On some systems, hotplug tests could hang forever waiting for cpu and
>>>> memory to be ready to be offlined. A special hotplug target is created,
>>>> which will help run non-hotplug tests and run hotplug tests as a special
>>>> case. Individual hotplug tests can still be run as a special target
>>>> targeted for a single subsystem.
>>>
>>> This is a bit sad.  The general philosophy with selftests is that they
>>> should run to completion even if the kernel/hardware which they are
>>> testing isn't available - they should work it out for themselves.
>>>
>>> But that's obviously a problem with hotplug.  And with networking or
>>> anything else which needs external action.
>>>
>>> On the other hand, networking has loopback and the kernel supports cpu
>>> hotplug simulation via procfs.  So perhaps the cpu and memory hotplug
>>> tests should be redone so they do the plug/unplug injection themselves,
>>> so they can run without external intervention?
>>
>> Changing/running the tests in a safe mode (least possibility of hang)
>> mode is another option. This way the tests are run in normal mode with
>> reduced scope. Memory hotplug test has the ratio option and when I
>> specified low ratio 1-5%, it completed in a few seconds.
>>
>> cpu-hotplug test will require changes. I am working on a change to
>> offline a user specified # of cpus instead offlining all hotpluggable
>> cpus and then onlining them again at the end of the test.
>>
>> When all selftests are run, safe mode hotplug tests will be run.
>>
>> Does this approach sound reasonable?
>
> I don't know really.  You know more about this than I - what advantages
> does the separate-make-target approach have over this approach?
>

Currently these tests run with full range - i.e try to offline
all cpus that are hotpluggable and try to offline all memory
that is hotpluggable. This results in hangs.

Creating a separate target the way I did it in this patch excludes these
tests all together. i.e when somebody runs:

make -C tools/testing/selftests run_tests

hotplug tests don't run.

Instead, with a few changes, tests can be run with a reduced scope so
a % of the memory gets offlined as opposed to all of it and the same
thing with cpus. This way hotplug code gets tested as opposed to
being excluded in a default test run case.

However, if limited scope testing isn't useful, separate target is
better until tests can be made safe to run without hangs.

-- Shuah

-- 
Shuah Khan
Senior Linux Kernel Developer - Open Source Group
Samsung Research America(Silicon Valley)
shuah.kh@samsung.com | (970) 672-0658

Re: [PATCH] tools: selftests - create a separate hotplug target

[Andrew Morton]

On Fri, 27 Jun 2014 13:59:55 -0600 Shuah Khan <shuah.kh@samsung.com> wrote:

> > I don't know really.  You know more about this than I - what advantages
> > does the separate-make-target approach have over this approach?
> >
> 
> Currently these tests run with full range - i.e try to offline
> all cpus that are hotpluggable and try to offline all memory
> that is hotpluggable. This results in hangs.
> 
> Creating a separate target the way I did it in this patch excludes these
> tests all together. i.e when somebody runs:
> 
> make -C tools/testing/selftests run_tests
> 
> hotplug tests don't run.
> 
> Instead, with a few changes, tests can be run with a reduced scope so
> a % of the memory gets offlined as opposed to all of it and the same
> thing with cpus. This way hotplug code gets tested as opposed to
> being excluded in a default test run case.
> 
> However, if limited scope testing isn't useful, separate target is
> better until tests can be made safe to run without hangs.

hm, OK.  So it sounds like the best solution would be to run the
limited tests by default and to require special intervention to run the
full tests.

I guess a separate make target is a suitable way of running the full
tests.  We could use an environment variable or various other things.