[LTP] the mandatory (?) nature of testscripts

[LTP] the mandatory (?) nature of testscripts

[Jiri Jaburek]

Hello all,
I normally execute the ltp suite using runltp with -f, specifying just
the runtest files I want to execute, possibly even writing my own ones
and using the fact that runltp detects absolute paths in the comma-
separated list and looks for them outside /opt/ltp.

From this viewpoint, and also looking at things like
testscripts/syscalls.sh, this approach seems perfectly valid (there's
at least nothing against it in the documentation, as far as I can tell).
My impression was therefore that testscripts are just convenient
(or legacy?) ways to run big chunks of the suite.

However then I found testscripts/network.sh and all the "mandatory"
variables it exports - not using it resulted in the ping01 test failing
due to recent rewrite in cbab2eed06. And, as it turns out, network.sh
is not alone.

Therefore I would like to ask (since I didn't find this documented);
is running individual tests via runltp -f something "officially"
supported and the network tests should be altered to get their defaults
in another way, or is runltp -f wild west and I'm on my own, with the
only supported way to run the suite being the testscripts?

Thanks,
Jiri

------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] the mandatory (?) nature of testscripts

[Alexey Kodanev]

Hi Jiri!
On 30.01.2015 17:18, Jiri Jaburek wrote:
> Hello all,
> I normally execute the ltp suite using runltp with -f, specifying just
> the runtest files I want to execute, possibly even writing my own ones
> and using the fact that runltp detects absolute paths in the comma-
> separated list and looks for them outside /opt/ltp.
>
> >From this viewpoint, and also looking at things like
> testscripts/syscalls.sh, this approach seems perfectly valid (there's
> at least nothing against it in the documentation, as far as I can tell).
> My impression was therefore that testscripts are just convenient
> (or legacy?) ways to run big chunks of the suite.
>
> However then I found testscripts/network.sh and all the "mandatory"
> variables it exports - not using it resulted in the ping01 test failing
> due to recent rewrite in cbab2eed06. And, as it turns out, network.sh
> is not alone.

network.sh lists network variables which are needed to run all network 
tests (stress group includes additional parameters, not yet moved them 
to network.sh from networkstress.sh). In general, it is essential to 
define remote host name, test interface names, IP addresses before 
running network tests. Not all of them run with localhost, e.g. ping01 
was modified to support various network configuration, you can now run 
it on several interfaces (probably testing various NIC drivers, make 
test traffic run over particular net-link, etc.) or running with 
localhost, for example:

$ echo 'ping601 ping01 -6' > runtest/ping_ipv6
$ RHOST_IPV6_HOST=':1' ./runltp -f ping_ipv6

> Therefore I would like to ask (since I didn't find this documented);
> is running individual tests via runltp -f something "officially"
> supported and the network tests should be altered to get their defaults
> in another way, or is runltp -f wild west and I'm on my own, with the
> only supported way to run the suite being the testscripts?

Test network configuration should be explicitly defined beforehand. You 
could set required parameters in your test environment, then use 
"runltp" as usual. I'm not sure right now if it is ok to add these 
parameters with default values to runltp, then export those variables 
inside it.

Best regards,
Alexey


------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] the mandatory (?) nature of testscripts

[Jiri Jaburek]

On 02/03/2015 11:52 AM, Alexey Kodanev wrote:
> Hi Jiri!
> On 30.01.2015 17:18, Jiri Jaburek wrote:
>> Hello all,
>> I normally execute the ltp suite using runltp with -f, specifying just
>> the runtest files I want to execute, possibly even writing my own ones
>> and using the fact that runltp detects absolute paths in the comma-
>> separated list and looks for them outside /opt/ltp.
>>
>> >From this viewpoint, and also looking at things like
>> testscripts/syscalls.sh, this approach seems perfectly valid (there's
>> at least nothing against it in the documentation, as far as I can tell).
>> My impression was therefore that testscripts are just convenient
>> (or legacy?) ways to run big chunks of the suite.
>>
>> However then I found testscripts/network.sh and all the "mandatory"
>> variables it exports - not using it resulted in the ping01 test failing
>> due to recent rewrite in cbab2eed06. And, as it turns out, network.sh
>> is not alone.
> 
> network.sh lists network variables which are needed to run all network 
> tests (stress group includes additional parameters, not yet moved them 
> to network.sh from networkstress.sh). In general, it is essential to 
> define remote host name, test interface names, IP addresses before 
> running network tests. Not all of them run with localhost, e.g. ping01 
> was modified to support various network configuration, you can now run 
> it on several interfaces (probably testing various NIC drivers, make 
> test traffic run over particular net-link, etc.) or running with 
> localhost, for example:

Thanks, but I can read that from git commits and the code - I understand
what has changed and can guess why it happened and it makes perfect
sense, just asking about the "design" of it.

> 
> $ echo 'ping601 ping01 -6' > runtest/ping_ipv6
> $ RHOST_IPV6_HOST=':1' ./runltp -f ping_ipv6

Unfortunately, from what I see, I need to export the "network" and
"host" part separately, by network classes (therefore networks not
using full class boundaries apparently can't be used) - see tst_ipaddr.

Meaning I can make the ping01 test work like it used to via

ping export LTPROOT; export IPV4_NETWORK=127.0.0; export
RHOST_IPV4_HOST=1; ping01

in my runtest file.

Still, my question was rather about the concept (for the future),
ie. whether it's okay to introduce something in network.sh which makes
it impossible to run tests separately (without patches) or whether
the tests *need* to remain executable separately.

Thanks.

> 
>> Therefore I would like to ask (since I didn't find this documented);
>> is running individual tests via runltp -f something "officially"
>> supported and the network tests should be altered to get their defaults
>> in another way, or is runltp -f wild west and I'm on my own, with the
>> only supported way to run the suite being the testscripts?
> 
> Test network configuration should be explicitly defined beforehand. You 
> could set required parameters in your test environment, then use 
> "runltp" as usual. I'm not sure right now if it is ok to add these 
> parameters with default values to runltp, then export those variables 
> inside it.
> 
> Best regards,
> Alexey
> 
> 
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming. The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media, is your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more. Take a
> look and join the conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Ltp-list mailing list
> Ltp-list@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/ltp-list
> 


------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] the mandatory (?) nature of testscripts

[Alexey Kodanev]

On 02/03/2015 02:00 PM, Jiri Jaburek wrote:
> On 02/03/2015 11:52 AM, Alexey Kodanev wrote:
>> Hi Jiri!
>> On 30.01.2015 17:18, Jiri Jaburek wrote:
>>> Hello all,
>>> I normally execute the ltp suite using runltp with -f, specifying just
>>> the runtest files I want to execute, possibly even writing my own ones
>>> and using the fact that runltp detects absolute paths in the comma-
>>> separated list and looks for them outside /opt/ltp.
>>>
>>> >From this viewpoint, and also looking at things like
>>> testscripts/syscalls.sh, this approach seems perfectly valid (there's
>>> at least nothing against it in the documentation, as far as I can tell).
>>> My impression was therefore that testscripts are just convenient
>>> (or legacy?) ways to run big chunks of the suite.
>>>
>>> However then I found testscripts/network.sh and all the "mandatory"
>>> variables it exports - not using it resulted in the ping01 test failing
>>> due to recent rewrite in cbab2eed06. And, as it turns out, network.sh
>>> is not alone.
>> network.sh lists network variables which are needed to run all network
>> tests (stress group includes additional parameters, not yet moved them
>> to network.sh from networkstress.sh). In general, it is essential to
>> define remote host name, test interface names, IP addresses before
>> running network tests. Not all of them run with localhost, e.g. ping01
>> was modified to support various network configuration, you can now run
>> it on several interfaces (probably testing various NIC drivers, make
>> test traffic run over particular net-link, etc.) or running with
>> localhost, for example:
> Thanks, but I can read that from git commits and the code - I understand
> what has changed and can guess why it happened and it makes perfect
> sense, just asking about the "design" of it.
>
>> $ echo 'ping601 ping01 -6' > runtest/ping_ipv6
>> $ RHOST_IPV6_HOST=':1' ./runltp -f ping_ipv6
> Unfortunately, from what I see, I need to export the "network" and
> "host" part separately, by network classes (therefore networks not
> using full class boundaries apparently can't be used) - see tst_ipaddr.

Hmm, the only tests which failed in that case are the tests from the 
stress group, they are using predefined masks for IPv4 and IPv6. Other 
tests shouldn't fail. What about defining variables like this:
IPV4_NETWORK='192.168' RHOST_IPV4_HOST='1.1' LHOST_IPV4_HOST='2.1'
Will it work for you?

By the way, there are certain limitation in the configuration, it only 
supports common use-case. If you want to add support for specific 
configuration (like different sub-nets), please provide a patch with the 
fixes.

IMHO, I wouldn't advise to run network tests together with other hosts, 
at least on a logical level.

> Meaning I can make the ping01 test work like it used to via
>
> ping export LTPROOT; export IPV4_NETWORK=127.0.0; export
> RHOST_IPV4_HOST=1; ping01
>
> in my runtest file.
>
> Still, my question was rather about the concept (for the future),
> ie. whether it's okay to introduce something in network.sh which makes
> it impossible to run tests separately (without patches) or whether
> the tests *need* to remain executable separately.

It's possible to run network tests separately without patches by setting 
the environment variables, you don't need to modify anything in LTP.

Thanks,
Alexey

> Thanks.
>
>>> Therefore I would like to ask (since I didn't find this documented);
>>> is running individual tests via runltp -f something "officially"
>>> supported and the network tests should be altered to get their defaults
>>> in another way, or is runltp -f wild west and I'm on my own, with the
>>> only supported way to run the suite being the testscripts?
>> Test network configuration should be explicitly defined beforehand. You
>> could set required parameters in your test environment, then use
>> "runltp" as usual. I'm not sure right now if it is ok to add these
>> parameters with default values to runltp, then export those variables
>> inside it.
>>
>> Best regards,
>> Alexey
>>
>>
>> ------------------------------------------------------------------------------
>> Dive into the World of Parallel Programming. The Go Parallel Website,
>> sponsored by Intel and developed in partnership with Slashdot Media, is your
>> hub for all things parallel software development, from weekly thought
>> leadership blogs to news, videos, case studies, tutorials and more. Take a
>> look and join the conversation now. http://goparallel.sourceforge.net/
>> _______________________________________________
>> Ltp-list mailing list
>> Ltp-list@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/ltp-list
>>


------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] the mandatory (?) nature of testscripts

[Cyril Hrubis]

Hi!
> Hello all,
> I normally execute the ltp suite using runltp with -f, specifying just
> the runtest files I want to execute, possibly even writing my own ones
> and using the fact that runltp detects absolute paths in the comma-
> separated list and looks for them outside /opt/ltp.
> 
> >From this viewpoint, and also looking at things like
> testscripts/syscalls.sh, this approach seems perfectly valid (there's
> at least nothing against it in the documentation, as far as I can tell).
> My impression was therefore that testscripts are just convenient
> (or legacy?) ways to run big chunks of the suite.

They are there from a times before current maintainers took over the
project. At that time the CVS repository was a dumping ground for
anything that the previous developers found at least a little bit
useful.

As far as I know, there was no architecture decision behind it, no
documentation either.

I simply didn't touched these because I had enough work fixing the
testcases.

I'm all for removing scripts that have no added value, the math.sh and
syscalls.sh looks like an obvious examples. There are likely more of
these, some may be even broken.

> However then I found testscripts/network.sh and all the "mandatory"
> variables it exports - not using it resulted in the ping01 test failing
> due to recent rewrite in cbab2eed06. And, as it turns out, network.sh
> is not alone.
>
> Therefore I would like to ask (since I didn't find this documented);
> is running individual tests via runltp -f something "officially"
> supported and the network tests should be altered to get their defaults
> in another way, or is runltp -f wild west and I'm on my own, with the
> only supported way to run the suite being the testscripts?

Well at the time the runltp is the supported way to run the testcases.
(Although I'm still dreaming of writing a better test driver instead of
the runltp and ltp-pan combination.) Or a more straight answer is that
this is the script I care about. Ideally the network parameters would be
handled somehow in the script, the same way the loop devices are, but
that haven't been done.

Also the mantra I'm repeating here over and over is to make the
testcases as self contained as possible. For the network testcases we
have to pass the the remove host and a few more parameters somehow, but
apart from that, it should just work fine.

-- 
Cyril Hrubis
chrubis@suse.cz

------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list