[LTP] [RFC] ltp test add reboot function

[LTP] [RFC] ltp test add reboot function

[Cixi Geng]

Hi ALL:

I want to add a reboot function in LTP for my team write a memory page
related case?
and need reboot to make the memory cache or some likely it pure clear.

So I want to add a tst_needs_reboot. when it was set, reboot the
machine and continue
run test from the point before.

I think  this requirement is meaningful? So can you give me some advise?

[LTP] [RFC] ltp test add reboot function

[Petr Vorel]

Hi Cixi,

> Hi ALL:

> I want to add a reboot function in LTP for my team write a memory page
> related case?
> and need reboot to make the memory cache or some likely it pure clear.

> So I want to add a tst_needs_reboot. when it was set, reboot the
> machine and continue
> run test from the point before.

> I think  this requirement is meaningful? So can you give me some advise?

IMHO setup for reboot has been usually handled outside of LTP (in frameworks
which use LTP), because without any more capable runner [*] reboot would just
stop testing. I'm not saying it's not possible, but requires more thinking. At
least I'd add some variable which would prevent boot (and thus tst_brk TCONF
would be issued for this test).

I wonder what other think about it.

Kind regards,
Petr

[*] runltp is dead, but even Cyril's not yet upstreamed runltp-ng [1] and other
plans [2] does not handle reboot either. It'd have to be some service which can
pick-up the state.

[1] https://github.com/metan-ucw/runltp-ng
[2] https://people.kernel.org/metan/towards-parallel-kernel-test-runs

[LTP] [RFC] ltp test add reboot function

[Cyril Hrubis]

Hi!
> I want to add a reboot function in LTP for my team write a memory page
> related case???
> and need reboot to make the memory cache or some likely it pure clear.
> 
> So I want to add a tst_needs_reboot. when it was set, reboot the
> machine and continue run test from the point before.
>
> I think  this requirement is meaningful??? So can you give me some advise?

Yes it is, actually you are not the only one who needs that. There are
plenty of other testcases that will have to reboot the machine, for
example the kexec() syscall tests, MCE injection tests, etc.

What I wanted to add is a bit similar to your proposal.

I wanted to add reboots_machine flag to the tst_test structure, which
will inform the test framework that the test will reboot the machine in
the middle of the test.

* The framework will have to inevitably run on a differen machine, but the
  runltp-ng already does that.

* The framework will reconnect to the machine after a reboot and
  re-execute the test with a flag that would say that this is a second
  stage of the test, i.e. after reboot.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [RFC] ltp test add reboot function

[Cixi Geng]

Thanks for you experiences on the reboot?
So far my opinion is add a reboot service by chkconfig or something like that.
the service will start runltp scripts from last break off point.
when the tst_test have .tst_needs_reboot =1 flag, it wil strore the
run status and
reboot the machine before run the test case.
Now I want to know what needs to be preserved?

Cyril Hrubis <chrubis@suse.cz> ?2020?8?24??? ??3:41???
>
> Hi!
> > I want to add a reboot function in LTP for my team write a memory page
> > related case???
> > and need reboot to make the memory cache or some likely it pure clear.
> >
> > So I want to add a tst_needs_reboot. when it was set, reboot the
> > machine and continue run test from the point before.
> >
> > I think  this requirement is meaningful??? So can you give me some advise?
>
> Yes it is, actually you are not the only one who needs that. There are
> plenty of other testcases that will have to reboot the machine, for
> example the kexec() syscall tests, MCE injection tests, etc.
>
> What I wanted to add is a bit similar to your proposal.
>
> I wanted to add reboots_machine flag to the tst_test structure, which
> will inform the test framework that the test will reboot the machine in
> the middle of the test.
>
> * The framework will have to inevitably run on a differen machine, but the
>   runltp-ng already does that.
>
> * The framework will reconnect to the machine after a reboot and
>   re-execute the test with a flag that would say that this is a second
>   stage of the test, i.e. after reboot.
>
> --
> Cyril Hrubis
> chrubis@suse.cz

[LTP] [RFC] ltp test add reboot function

[Cyril Hrubis]

Hi!
> Thanks for you experiences on the reboot???
> So far my opinion is add a reboot service by chkconfig or something like that.
> the service will start runltp scripts from last break off point.
> when the tst_test have .tst_needs_reboot =1 flag, it wil strore the
> run status and
> reboot the machine before run the test case.
> Now I want to know what needs to be preserved???

The runltp script and ltp-pan cannot save and restore a testrun,
although something like that may be possible to implement, I doubt that
it could be easily added to the current infrastructure. Not to mention
that ltp-pan has been in a maintenance mode (which means no new
features) for years now.

Also I do consider the whole concept of running the test exectution
framework on the same machine as the testcases broken by desing for
kernel related tests. The way forward is the patchset send by Richard
that implements simple test executor that runs on the machine under test
and communicates with a execution framework that runs on a central
server.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [RFC] ltp test add reboot function

[Bird, Tim]

> -----Original Message-----
> From:  Cyril Hrubis
> 
> Hi!
> > Thanks for you experiences on the reboot???
> > So far my opinion is add a reboot service by chkconfig or something like that.
> > the service will start runltp scripts from last break off point.
> > when the tst_test have .tst_needs_reboot =1 flag, it wil strore the
> > run status and
> > reboot the machine before run the test case.
> > Now I want to know what needs to be preserved???
> 
> The runltp script and ltp-pan cannot save and restore a testrun,
> although something like that may be possible to implement, I doubt that
> it could be easily added to the current infrastructure. Not to mention
> that ltp-pan has been in a maintenance mode (which means no new
> features) for years now.
> 
> Also I do consider the whole concept of running the test exectution
> framework on the same machine as the testcases broken by desing for
> kernel related tests. The way forward is the patchset send by Richard
> that implements simple test executor that runs on the machine under test
> and communicates with a execution framework that runs on a central
> server.

I don't have anything to contribute to the primary discussion topic,
but I just have to chime in here that it's very nice to hear someone
else express this idea.  Fuego's architecture is built on having the test
execution driven from a machine external to the machine under test.
To me, having the machine under test drive its own test execution seems weird,
in that it requires the machine to potentially perform actions (like reboot,
or saving panic data), that require operational fitness - which is the very thing
being tested.

Sorry - just had to vent a little.  I "preach" this a lot, and it's nice to the same
opinion from someone else.
 -- Tim

[LTP] [RFC] ltp test add reboot function

[Cixi Geng]

>Also I do consider the whole concept of running the test exectution
>framework on the same machine as the testcases broken by desing for
>kernel related tests. The way forward is the patchset send by Richard
>that implements simple test executor that runs on the machine under test
>and communicates with a execution framework that runs on a central
>server.

I agree with some of your views? the test architecture server/slave is most
common test method. but I also want to know what kind of connection
method between test machine and central server,  the ssh , adb or serial?

because my test machine is a cellphone, our purpose is test the kernel before
androidOS , so I build debianOS on our haraware, and we can only used the
serial port communicate with our testmachine.

Bird, Tim <Tim.Bird@sony.com> ?2020?8?28??? ??11:37???
>
>
>
> > -----Original Message-----
> > From:  Cyril Hrubis
> >
> > Hi!
> > > Thanks for you experiences on the reboot???
> > > So far my opinion is add a reboot service by chkconfig or something like that.
> > > the service will start runltp scripts from last break off point.
> > > when the tst_test have .tst_needs_reboot =1 flag, it wil strore the
> > > run status and
> > > reboot the machine before run the test case.
> > > Now I want to know what needs to be preserved???
> >
> > The runltp script and ltp-pan cannot save and restore a testrun,
> > although something like that may be possible to implement, I doubt that
> > it could be easily added to the current infrastructure. Not to mention
> > that ltp-pan has been in a maintenance mode (which means no new
> > features) for years now.
> >
> > Also I do consider the whole concept of running the test exectution
> > framework on the same machine as the testcases broken by desing for
> > kernel related tests. The way forward is the patchset send by Richard
> > that implements simple test executor that runs on the machine under test
> > and communicates with a execution framework that runs on a central
> > server.
>
> I don't have anything to contribute to the primary discussion topic,
> but I just have to chime in here that it's very nice to hear someone
> else express this idea.  Fuego's architecture is built on having the test
> execution driven from a machine external to the machine under test.
> To me, having the machine under test drive its own test execution seems weird,
> in that it requires the machine to potentially perform actions (like reboot,
> or saving panic data), that require operational fitness - which is the very thing
> being tested.
>
> Sorry - just had to vent a little.  I "preach" this a lot, and it's nice to the same
> opinion from someone else.
>  -- Tim
>

[LTP] [RFC] ltp test add reboot function

[Bird, Tim]

> -----Original Message-----
> From: Cixi Geng <gengcixi@gmail.com>
> 
> >Also I do consider the whole concept of running the test exectution
> >framework on the same machine as the testcases broken by desing for
> >kernel related tests. The way forward is the patchset send by Richard
> >that implements simple test executor that runs on the machine under test
> >and communicates with a execution framework that runs on a central
> >server.
> 
> I agree with some of your views? the test architecture server/slave is most
> common test method. but I also want to know what kind of connection
> method between test machine and central server,  the ssh , adb or serial?
> 
> because my test machine is a cellphone, our purpose is test the kernel before
> androidOS , so I build debianOS on our haraware, and we can only used the
> serial port communicate with our testmachine.

Fuego and LAVA both support serial port as a method to communicate
from a testing host to a device under test.  The default is usually ssh, but boards
with only a serial port are also supported.

LAVA uses ser2net and Fuego uses serio and serlogin.  Those are the tools I'm
most familiar with for doing testing operations over the serial port.

See https://github.com/frowand/serio.git
and https://github.com/tbird20d/serlogin.git

> 
> Bird, Tim <Tim.Bird@sony.com> ?2020?8?28??? ??11:37???
> >
> >
> >
> > > -----Original Message-----
> > > From:  Cyril Hrubis
> > >
> > > Hi!
> > > > Thanks for you experiences on the reboot???
> > > > So far my opinion is add a reboot service by chkconfig or something like that.
> > > > the service will start runltp scripts from last break off point.
> > > > when the tst_test have .tst_needs_reboot =1 flag, it wil strore the
> > > > run status and
> > > > reboot the machine before run the test case.
> > > > Now I want to know what needs to be preserved???
> > >
> > > The runltp script and ltp-pan cannot save and restore a testrun,
> > > although something like that may be possible to implement, I doubt that
> > > it could be easily added to the current infrastructure. Not to mention
> > > that ltp-pan has been in a maintenance mode (which means no new
> > > features) for years now.
> > >
> > > Also I do consider the whole concept of running the test exectution
> > > framework on the same machine as the testcases broken by desing for
> > > kernel related tests. The way forward is the patchset send by Richard
> > > that implements simple test executor that runs on the machine under test
> > > and communicates with a execution framework that runs on a central
> > > server.
> >
> > I don't have anything to contribute to the primary discussion topic,
> > but I just have to chime in here that it's very nice to hear someone
> > else express this idea.  Fuego's architecture is built on having the test
> > execution driven from a machine external to the machine under test.
> > To me, having the machine under test drive its own test execution seems weird,
> > in that it requires the machine to potentially perform actions (like reboot,
> > or saving panic data), that require operational fitness - which is the very thing
> > being tested.
> >
> > Sorry - just had to vent a little.  I "preach" this a lot, and it's nice to the same
> > opinion from someone else.
> >  -- Tim
> >

[LTP] [RFC] ltp test add reboot function

[Cyril Hrubis]

Hi!
> >Also I do consider the whole concept of running the test exectution
> >framework on the same machine as the testcases broken by desing for
> >kernel related tests. The way forward is the patchset send by Richard
> >that implements simple test executor that runs on the machine under test
> >and communicates with a execution framework that runs on a central
> >server.
> 
> I agree with some of your views??? the test architecture server/slave is most
> common test method. but I also want to know what kind of connection
> method between test machine and central server,  the ssh , adb or serial?
> 
> because my test machine is a cellphone, our purpose is test the kernel before
> androidOS , so I build debianOS on our haraware, and we can only used the
> serial port communicate with our testmachine.

As for LTP the tool Richard is working on is a daemon that runs on the
SUT[1], will be quite possibly started during the boot and will listen
on pre-defined transport channels, e.g. TCP, serial, bluetooth, etc. The
protocol itself should be transport agnostic and should work over
anything that can transport bytes. But that is something that is under
development at the moment.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [RFC] ltp test add reboot function

[Cixi Geng]

Hi ALL:
I considered  running ltp with reboot function by HOST controller.
And here I found that if case need reboot before or after it run. the
HOST controller
is easy to implement, BUT when the case want reboot between it's verify_func
I hava no idea that how to continue run the rest verify_func
continuing after reboot.
Can you give some guidance?

Cyril Hrubis <chrubis@suse.cz> ?2020?9?3??? ??5:24???
>
> Hi!
> > >Also I do consider the whole concept of running the test exectution
> > >framework on the same machine as the testcases broken by desing for
> > >kernel related tests. The way forward is the patchset send by Richard
> > >that implements simple test executor that runs on the machine under test
> > >and communicates with a execution framework that runs on a central
> > >server.
> >
> > I agree with some of your views??? the test architecture server/slave is most
> > common test method. but I also want to know what kind of connection
> > method between test machine and central server,  the ssh , adb or serial?
> >
> > because my test machine is a cellphone, our purpose is test the kernel before
> > androidOS , so I build debianOS on our haraware, and we can only used the
> > serial port communicate with our testmachine.
>
> As for LTP the tool Richard is working on is a daemon that runs on the
> SUT[1], will be quite possibly started during the boot and will listen
> on pre-defined transport channels, e.g. TCP, serial, bluetooth, etc. The
> protocol itself should be transport agnostic and should work over
> anything that can transport bytes. But that is something that is under
> development at the moment.
>
> --
> Cyril Hrubis
> chrubis@suse.cz

[LTP] [RFC] ltp test add reboot function

[Cyril Hrubis]

Hi!
> I considered  running ltp with reboot function by HOST controller.
> And here I found that if case need reboot before or after it run. the
> HOST controller
> is easy to implement, BUT when the case want reboot between it's verify_func
> I hava no idea that how to continue run the rest verify_func
> continuing after reboot.
> Can you give some guidance?

I guess that we would have to add a command line parameter to the test
library to tell the testcase to continue with the second half of the
test. Then after the reboot the testcase would be executed with that
option so that it knows that we are running it for a second time and
then we have to pass that to the testcases.

And since the uClinux support is dead, we are free to reuse the -C flag
for this purpose. Probably easiest solution would be to set a global
variable (named tst_rebooted or something similar) if -C was passed to
the test on a command line, then we can use the value of this variable
in the test setup/verify/cleanup functions.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [RFC] ltp test add reboot function

[Bird, Tim]

> -----Original Message-----
> From: Cyril Hrubis <chrubis@suse.cz>
> 
> Hi!
> > I considered  running ltp with reboot function by HOST controller.
> > And here I found that if case need reboot before or after it run. the
> > HOST controller
> > is easy to implement, BUT when the case want reboot between it's verify_func
> > I hava no idea that how to continue run the rest verify_func
> > continuing after reboot.
> > Can you give some guidance?
> 
> I guess that we would have to add a command line parameter to the test
> library to tell the testcase to continue with the second half of the
> test. Then after the reboot the testcase would be executed with that
> option so that it knows that we are running it for a second time and
> then we have to pass that to the testcases.
> 
> And since the uClinux support is dead, we are free to reuse the -C flag
> for this purpose. Probably easiest solution would be to set a global
> variable (named tst_rebooted or something similar) if -C was passed to
> the test on a command line, then we can use the value of this variable
> in the test setup/verify/cleanup functions.

Just brainstorming here...

How about an environment variable that contains the location to resume
execution.  It could be a specific test case (within a test), or some value that the verify
function uses to skip down to the place to resume.

If set, the library would skip the setup operation (or modify it appropriately).

Something like this:

LTP_RESUME_POS=reboot_test:testcase_6

(or some better name :-) )

The value for the resume position would have to be sent to the test framework,
so it could set it (in the variable) on machine reboot.  The test framework has to know
to set something (either -C or LTP_RESUME_POS) so that the test can recognize it is
in a resume-after-reboot condition.

OR

Maybe it's sufficient for the test to create a temp file (in a temp directory that is known
to be persistent across a reboot, which not all of them are).  The presence of the temp
file could indicate a resume-after-reboot condition, and it's contents could be used
to indicate the resume position.

Question: does this "resume-after-reboot" condition need to be recognized by ltp-pan?

I'm still not sure what is envisioned for the interface between the test and the
test framework, to detect that it should resume a particular test on DUT reboot.
If the test is initiating the reboot, maybe it needs to communicate some data to 
the test framework (or whatever is performing the reboot), so that things can be
set up during boot to continue where the test left off.

We have been envisioning in Fuego supporting an API like the following, for 
a generic reboot mechanism for rebooting a board:

CLI:
lc board {board_name} reboot

REST API:
wget https://{lab-control-server}/api/devices/{board_name}/power/reboot

These are both intended to support immediate reboot of the board, and don't take
any parameters.  It seems like there's a bit of "knowledge" about the board
bringup that is outside the scope of just a simple board reboot operation, that would have
to be conveyed to the test framework and possibly the on-board boot initialization
code, to accomplish a resume operation for a test.  It's a bit hard to figure out
where the extra information should reside.  Should the data be placed in
the reboot API?  Should there be a separate call to the test framework/board control
software to prepare for a reboot-and-resume-test operation?

Just my 2 cents.
 -- Tim

[LTP] [RFC] ltp test add reboot function

[Cyril Hrubis]

Hi!
> > I guess that we would have to add a command line parameter to the test
> > library to tell the testcase to continue with the second half of the
> > test. Then after the reboot the testcase would be executed with that
> > option so that it knows that we are running it for a second time and
> > then we have to pass that to the testcases.
> > 
> > And since the uClinux support is dead, we are free to reuse the -C flag
> > for this purpose. Probably easiest solution would be to set a global
> > variable (named tst_rebooted or something similar) if -C was passed to
> > the test on a command line, then we can use the value of this variable
> > in the test setup/verify/cleanup functions.
> 
> Just brainstorming here...
> 
> How about an environment variable that contains the location to resume
> execution.  It could be a specific test case (within a test), or some value that the verify
> function uses to skip down to the place to resume.
> 
> If set, the library would skip the setup operation (or modify it appropriately).
> 
> Something like this:
> 
> LTP_RESUME_POS=reboot_test:testcase_6
> 
> (or some better name :-) )
> 
> The value for the resume position would have to be sent to the test framework,
> so it could set it (in the variable) on machine reboot.  The test framework has to know
> to set something (either -C or LTP_RESUME_POS) so that the test can recognize it is
> in a resume-after-reboot condition.
> 
> OR
> 
> Maybe it's sufficient for the test to create a temp file (in a temp directory that is known
> to be persistent across a reboot, which not all of them are).  The presence of the temp
> file could indicate a resume-after-reboot condition, and it's contents could be used
> to indicate the resume position.

That would mean defining a persistent directory and would be prone to
leftovers, i.e. what happends when a test crashes. I think that having a
environment variable would be much better solution.

> Question: does this "resume-after-reboot" condition need to be recognized by ltp-pan?

First of all I think that ltp-pan is something that does not fit into
this picture at all. Hopefully it will be replaced by runltp-ng in a few
years, which will run on a different host and will simply continue to
run during the time the SUT is rebooted.

> I'm still not sure what is envisioned for the interface between the test and the
> test framework, to detect that it should resume a particular test on DUT reboot.
> If the test is initiating the reboot, maybe it needs to communicate some data to 
> the test framework (or whatever is performing the reboot), so that things can be
> set up during boot to continue where the test left off.

I do see it as:

* The test advertizes to the test executing framework that it reboots
  the machine during the testrun

* When the test is executed the test executing framework will expect
  reboot, wait for the machine to boot and finally re-executes the test

* The execution framework will also pass down the evironment
  variable/command line parameters so that the test picks up where it's
  supposed to.

  This, among other things, makes sure that there is no state saved on
  the SUT and when a test fails after third reboot we will know exactly
  where since we are tracking the state in the execution framework.

Does this make sense?

> We have been envisioning in Fuego supporting an API like the following, for 
> a generic reboot mechanism for rebooting a board:
> 
> CLI:
> lc board {board_name} reboot
> 
> REST API:
> wget https://{lab-control-server}/api/devices/{board_name}/power/reboot
> 
> These are both intended to support immediate reboot of the board, and don't take
> any parameters.  It seems like there's a bit of "knowledge" about the board
> bringup that is outside the scope of just a simple board reboot operation, that would have
> to be conveyed to the test framework and possibly the on-board boot initialization
> code, to accomplish a resume operation for a test.  It's a bit hard to figure out
> where the extra information should reside.  Should the data be placed in
> the reboot API?  Should there be a separate call to the test framework/board control
> software to prepare for a reboot-and-resume-test operation?

Well for the parts that are related to the testrun these probably
belongs to the executing framework. The runltp-ng works in a way where
it drives the execution of the tests, i.e. it waits for a machine to
boot, then starts executing binaries. These binaries are really a single
testcases, each with it's own timemout, etc. Which really means that it
can also handle a test that needs reboot just fine since it will execute
the test binary for a first time, waits for a reboot, then execute the
binary for a second time with correspnding parameters.

What exactly do you mean by the board specific info? I suppose that's
about kernel image, rootfs and where to load these from, right? That
should be probably part of the lab-control to remmeber these between
test requested reboots.

-- 
Cyril Hrubis
chrubis@suse.cz