[RFC nft PATCH] tests: shell: add a basic scapy test

[RFC nft PATCH] tests: shell: add a basic scapy test

[Arturo Borrero Gonzalez]

From: Arturo Borrero Gonzalez <arturo@debian.org>

This test uses scapy to send a packet and test our packet/data path.
We grep the 'nft list ruleset' output for a counter increment.

If we like this approach, then we could easily add more testcases
following the pattern in this patch.

Reference:
 https://unixia.wordpress.com/2016/02/17/testing-with-scapy/

Signed-off-by: Arturo Borrero Gonzalez <arturo@debian.org>
---
 tests/shell/testcases/scapy/0001_ip_ttl_0 |   33 +++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
 create mode 100755 tests/shell/testcases/scapy/0001_ip_ttl_0

diff --git a/tests/shell/testcases/scapy/0001_ip_ttl_0 b/tests/shell/testcases/scapy/0001_ip_ttl_0
new file mode 100755
index 0000000..1e4489e
--- /dev/null
+++ b/tests/shell/testcases/scapy/0001_ip_ttl_0
@@ -0,0 +1,33 @@
+#!/usr/bin/env python
+
+# this testcase launches an IP packet with ttl=2. An input
+# rule should capture the packet and increment the counter which
+# we will then grep to know about the result
+
+import os
+from subprocess import call
+try:
+	from scapy.all import *
+except ImportError:
+	sys.stderr.write("Unable to import scapy")
+	exit(0)
+
+# config
+nft = os.environ['NFT']
+conf.L3socket = L3RawSocket
+
+# capturing ruleset
+call([nft + " add table ip t"], shell=True)
+call([nft + " add chain ip t c {type filter hook input priority 0 \; policy accept \;}"], shell=True)
+call([nft + " add rule ip t c ip ttl 2 counter"], shell = True)
+
+# scapy packet
+pkt = IP()
+pkt.ttl = 2
+sr1(pkt, verbose=False)
+
+# results
+ruleset = call([nft + " list ruleset | grep \"ip ttl 2 counter packets 1\" >/dev/null"], shell=True)
+if ruleset != 0:
+	sys.stderr.write("Packet not captured by nftables?")
+	exit(1)

Re: [RFC nft PATCH] tests: shell: add a basic scapy test

[Pablo Neira Ayuso]

On Wed, Nov 30, 2016 at 10:39:06AM +0100, Arturo Borrero Gonzalez wrote:
> From: Arturo Borrero Gonzalez <arturo@debian.org>
> 
> This test uses scapy to send a packet and test our packet/data path.
> We grep the 'nft list ruleset' output for a counter increment.
> 
> If we like this approach, then we could easily add more testcases
> following the pattern in this patch.

I think it's been several netfilter workshops already talking on this,
but it never happens because nobody pushed this forward.

If you can make this happen, it would great. Testing the datapath is
something that we always wanted to have.

Several ideas:

* Check if you can use the dummy interface, so we make sure no other
  packets interfer with the tests.

* You can probably augment this at some pointer to rely on the new
  nf_tables tracing infrastructure.

Anyway, I agree that starting with something simple is good enough.

Re: [RFC nft PATCH] tests: shell: add a basic scapy test

[Pablo Neira Ayuso]

On Wed, Nov 30, 2016 at 07:27:04PM +0100, Pablo Neira Ayuso wrote:
> On Wed, Nov 30, 2016 at 10:39:06AM +0100, Arturo Borrero Gonzalez wrote:
> > From: Arturo Borrero Gonzalez <arturo@debian.org>
> > 
> > This test uses scapy to send a packet and test our packet/data path.
> > We grep the 'nft list ruleset' output for a counter increment.
> > 
> > If we like this approach, then we could easily add more testcases
> > following the pattern in this patch.
> 
> I think it's been several netfilter workshops already talking on this,
> but it never happens because nobody pushed this forward.
> 
> If you can make this happen, it would great. Testing the datapath is
> something that we always wanted to have.
> 
> Several ideas:
> 
> * Check if you can use the dummy interface, so we make sure no other
>   packets interfer with the tests.
> 
> * You can probably augment this at some pointer to rely on the new
>   nf_tables tracing infrastructure.
> 
> Anyway, I agree that starting with something simple is good enough.

Only one more question left: Do you think you can slightly generalize
this so we decouple test files from the script? Similar to what we
have for nft-tests.py.

Re: [RFC nft PATCH] tests: shell: add a basic scapy test

[Arturo Borrero Gonzalez]

On 30 November 2016 at 19:28, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
>> * You can probably augment this at some pointer to rely on the new
>>   nf_tables tracing infrastructure.
>>

That would be rather complex.

>
> Only one more question left: Do you think you can slightly generalize
> this so we decouple test files from the script? Similar to what we
> have for nft-tests.py.

What do you mean?

This testcase script is decoupled enough that you can even call it
this way (and it works):

 % NFT=%(which nft) testcases/scapy/0001_ip_ttl_0

Do you mean you would like to have a different testsuite (with a
different runner script) for these datapath tests?

Re: [RFC nft PATCH] tests: shell: add a basic scapy test

[Pablo Neira Ayuso]

On Thu, Dec 01, 2016 at 09:10:53AM +0100, Arturo Borrero Gonzalez wrote:
> On 30 November 2016 at 19:28, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> >> * You can probably augment this at some pointer to rely on the new
> >>   nf_tables tracing infrastructure.
> >>
> 
> That would be rather complex.

OK, let's start with this something simple.

> > Only one more question left: Do you think you can slightly generalize
> > this so we decouple test files from the script? Similar to what we
> > have for nft-tests.py.
> 
> What do you mean?
> 
> This testcase script is decoupled enough that you can even call it
> this way (and it works):
> 
>  % NFT=%(which nft) testcases/scapy/0001_ip_ttl_0
> 
> Do you mean you would like to have a different testsuite (with a
> different runner script) for these datapath tests?

I mean, it would be good if you place as much common code as possible
in the runner script, so individual unit tests don't result in too
much copy and paste.

Thanks Arturo.

Re: [RFC nft PATCH] tests: shell: add a basic scapy test

[Vadim Kochan]

On Thu, Dec 1, 2016 at 12:45 PM, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> On Thu, Dec 01, 2016 at 09:10:53AM +0100, Arturo Borrero Gonzalez wrote:
>> On 30 November 2016 at 19:28, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
>> >> * You can probably augment this at some pointer to rely on the new
>> >>   nf_tables tracing infrastructure.
>> >>
>>
>> That would be rather complex.
>
> OK, let's start with this something simple.
>
>> > Only one more question left: Do you think you can slightly generalize
>> > this so we decouple test files from the script? Similar to what we
>> > have for nft-tests.py.
>>
>> What do you mean?
>>
>> This testcase script is decoupled enough that you can even call it
>> this way (and it works):
>>
>>  % NFT=%(which nft) testcases/scapy/0001_ip_ttl_0
>>
>> Do you mean you would like to have a different testsuite (with a
>> different runner script) for these datapath tests?
>
> I mean, it would be good if you place as much common code as possible
> in the runner script, so individual unit tests don't result in too
> much copy and paste.
>
> Thanks Arturo.
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Sorry for the noise, may PTF framework help for you ? Usually it is
used for datapath testing
on switches, but may be it can cover your cases.

Regards,
Vadim Kochan

Re: [RFC nft PATCH] tests: shell: add a basic scapy test

[Arturo Borrero Gonzalez]

On 1 December 2016 at 11:45, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> I mean, it would be good if you place as much common code as possible
> in the runner script, so individual unit tests don't result in too
> much copy and paste.
>

Ok, I understand.

Actually, as you know I'm just experimenting with this.

Anyway the problem I see is that we could end losing a lot of flexibility.
The current py testsuite is only able to perform one kind of tests
because of this approach.
In the other hand, the shell testsuite is able to perform almost any
kind of tests because it only executes arbitrary binaries.

So perhaps we could take an intermediate approach:
 * scapy tests are executed by the shell testsuite runner (they are
standalone scripts)
 * we develop a common lib of functions inside
tests/shell/testcases/scapy/ (for example common.py)
 * then, each scapy test load that common lib which includes most of
the factorised code

Common functions would be something like this:

* configure(): we do the scapy configuration, network config, or whatever
* load_ruleset): we pass a nft ruleset (a string) and load it using nft -f
* check_result(): we grep the ruleset counters, or whatever

I'm thinking of some tests we could have using this approach:

* atomic replacement of ruleset during a network transfer
* conntrack modifications (using the conntrack-tools binaries)
* packet mangling, NAT, etc

In any case, I think we should retain the ability to load nft rules,
send/recv scapy packets and check for nft counters at any time during
the execution.

Re: [RFC nft PATCH] tests: shell: add a basic scapy test

[Pablo Neira Ayuso]

On Thu, Dec 01, 2016 at 04:05:07PM +0100, Arturo Borrero Gonzalez wrote:
> On 1 December 2016 at 11:45, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> > I mean, it would be good if you place as much common code as possible
> > in the runner script, so individual unit tests don't result in too
> > much copy and paste.
> >
> 
> Ok, I understand.
> 
> Actually, as you know I'm just experimenting with this.
>
> Anyway the problem I see is that we could end losing a lot of flexibility.
> The current py testsuite is only able to perform one kind of tests
> because of this approach.
> In the other hand, the shell testsuite is able to perform almost any
> kind of tests because it only executes arbitrary binaries.

Flexibility is good indeed. Regarding the comments on the existing
testsuites, I think both complement each other, so they are both
useful.

> So perhaps we could take an intermediate approach:
>  * scapy tests are executed by the shell testsuite runner (they are
> standalone scripts)
>  * we develop a common lib of functions inside
> tests/shell/testcases/scapy/ (for example common.py)
>  * then, each scapy test load that common lib which includes most of
> the factorised code

That's good, I like this balance. Look, my only concern is
maintainability: If we end up using the copy & paste pattern too much
on these scripts, we will have to patch them all if we find a bug on
them.

But don't worry too much on trying to generalize things upfront given
there's experimentation on this going on. We can introduce
abstractions incrementally, as we start seeing that code duplicates.

> Common functions would be something like this:
> 
> * configure(): we do the scapy configuration, network config, or whatever
> * load_ruleset): we pass a nft ruleset (a string) and load it using nft -f
> * check_result(): we grep the ruleset counters, or whatever
> 
> I'm thinking of some tests we could have using this approach:
> 
> * atomic replacement of ruleset during a network transfer
> * conntrack modifications (using the conntrack-tools binaries)
> * packet mangling, NAT, etc
> 
> In any case, I think we should retain the ability to load nft rules,
> send/recv scapy packets and check for nft counters at any time during
> the execution.

Sure, go ahead.

Thanks Arturo.