Re: [conntrack-tools PATCH 1/3] tests: introduce new python-based framework for running tests

[Pablo Neira Ayuso <pablo@netfilter.org>]

Hi Arturo,

On Sun, Jan 24, 2021 at 01:22:37AM +0100, Arturo Borrero Gonzalez wrote:
> This test suite should help us develop better tests for conntrack-tools in general and conntrackd
> in particular.
> 
> The framework is composed of a runner script, written in python3, and 3 yaml files for
> configuration and testcase definition:
> 
>  - scenarios.yaml: contains information on network scenarios for tests to use
>  - tests.yaml: contains testcase definition
>  - env.yaml: contains default values for environment variables
> 
> The test cases can be anything, from a simple command to an external script call to perform more
> complex operations. See follow-up patches to know more on how this works.
> 
> The plan is to replace or call from this framework the other testsuites in this tree.
> 
> The runner script is rather simple, and it should be more or less straight forward to use it.
> It requires the python3-yaml package to be installed.
> 
> For reference, here are the script options:
> 
> === 8< ===
> $ tests/cttools-testing-framework.py --help
> usage: cttools-testing-framework.py [-h] [--tests-file TESTS_FILE]
> 				[--scenarios-file SCENARIOS_FILE]
> 				[--env-file ENV_FILE]
> 				[--single SINGLE]
> 				[--start-scenario START_SCENARIO]
> 				[--stop-scenario STOP_SCENARIO]
> 				[--debug]
> 
> Utility to run tests for conntrack-tools
> 
> optional arguments:
>   -h, --help            show this help message and exit
>   --tests-file TESTS_FILE
>                         File with testcase definitions. Defaults to 'tests.yaml'
>   --scenarios-file SCENARIOS_FILE
>                         File with configuration scenarios for tests. Defaults to 'scenarios.yaml'
>   --env-file ENV_FILE   File with environment variables for scenarios/tests. Defaults to 'env.yaml'
>   --single SINGLE       Execute a single testcase and exit. Use this for developing testcases
>   --start-scenario START_SCENARIO
>                         Execute scenario start commands and exit. Use this for developing testcases
>   --stop-scenario STOP_SCENARIO
>                         Execute scenario stop commands and exit. Use this for cleanup
>   --debug               debug mode
> === 8< ===
> 
> To run it, simply use:
> 
> === 8< ===
> $ cd tests/ ; sudo ./cttools-testing-framework.py

Automated regression test infrastructure is nice to have!

A few nitpick requests and one suggestion:

* Rename cttools-testing-framework.py to conntrackd-tests.py
* Move it to the tests/conntrackd/ folder
* Missing yaml dependency in python in my test machine

Traceback (most recent call last):
  File "cttools-testing-framework.py", line 36, in <module>
    import yaml
ModuleNotFoundError: No module named 'yaml'

this is installed from pip, right? Just a note in the commit message
is fine.

* Would it be possible to define the scenario in shell script files?
  For example, to define the "simple_stats" scenario, the YAML file
  looks like this:

- name: simple_stats
- script: shell/simple_stats.sh

The shell script takes "start" or "stop" as $1 to set up the scenario.
For developing more test, having the separated shell script might be
convenient.

This also allows to run scenario for development purpose away from the
automated regression infrastructure (although, you already thought
about this with the --start-scenario and --stop-scenario options, I
think those options are fine, I would not remove them).

Thanks !