Re: [conntrack-tools PATCH 1/3] tests: introduce new python-based framework for running tests

[Pablo Neira Ayuso <pablo@netfilter.org>]

On Mon, Feb 01, 2021 at 11:49:02AM +0100, Arturo Borrero Gonzalez wrote:
> On 2/1/21 4:31 AM, Pablo Neira Ayuso wrote:
[...]
> > * Missing yaml dependency in python in my test machine
> > 
> > Traceback (most recent call last):
> >    File "cttools-testing-framework.py", line 36, in <module>
> >      import yaml
> > ModuleNotFoundError: No module named 'yaml'
> > 
> > this is installed from pip, right? Just a note in the commit message
> > is fine.
> 
> It was already present in the commit message.
> 
> I made it more clear:
> 
> === 8< ===
> On Debian machines, it requires the *python3-yaml* package to be installed
> as a dependency
> === 8< ===

Sorry, I overlook this.

> > * Would it be possible to define the scenario in shell script files?
> >    For example, to define the "simple_stats" scenario, the YAML file
> >    looks like this:
> > 
> > - name: simple_stats
> > - script: shell/simple_stats.sh
> > 
> > The shell script takes "start" or "stop" as $1 to set up the scenario.
> > For developing more test, having the separated shell script might be
> > convenient.
> > 
> 
> This is already supported:
> 
> === 8< ===
> - name: myscenario
>   start:
>     - ./script.sh start
>   stop:
>     - ./script.sh stop
> === 8< ===

Ok, I've sent a patch to move the netns network setup to a shell
script:

https://patchwork.ozlabs.org/project/netfilter-devel/patch/20210201170015.28217-1-pablo@netfilter.org/

> > Thanks !
> > 
> 
> Thanks for the review. I made the changes you requested and pushed it to the
> repository.
> 
> I plan to follow up soon with more tests.
>
> Question: I have a few testcases that trigger bugs, segfaults etc. Would it
> be OK to create something like 'failingtestcases.yaml' and register all
> those bugs there until the get fixed? That way we have reproducible bugs
> until we can fix them.

That's fine, but before we add more tests, please let's where to move
more inlined configurations in the yaml files to independent files
that can be reused by new tests.

Thanks.