* [PATCH net-next 0/3] tc-testing: Add JSON verification and simple traffic generation
@ 2019-07-03 2:41 Lucas Bates
2019-07-03 2:41 ` [PATCH net-next 1/3] tc-testing: Add JSON verification to tdc Lucas Bates
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: Lucas Bates @ 2019-07-03 2:41 UTC (permalink / raw)
To: davem
Cc: netdev, jhs, xiyou.wangcong, jiri, mleitner, vladbu, dcaratti,
kernel, Lucas Bates
This patchset introduces JSON as a verification method in tdc and adds a new
plugin, scapyPlugin, as a way to send traffic to test tc filters and actions.
The first patch adds the JSON verification to the core tdc script.
The second patch makes a change to the TdcPlugin module that will allow tdc
plugins to examine the test case currently being executed, such that plugins
can play a more active role in testing. This feature is needed for the
new plugin.
The third patch adds the scapyPlugin itself, and an example test case file to
demonstrate how the scapy block works.
Lucas Bates (3):
tc-testing: Add JSON verification to tdc
tc-testing: Allow tdc plugins to see test case data
tc-testing: introduce scapyPlugin for basic traffic
tools/testing/selftests/tc-testing/TdcPlugin.py | 5 +-
.../creating-testcases/scapy-example.json | 98 ++++++++++++++
.../selftests/tc-testing/plugin-lib/scapyPlugin.py | 51 +++++++
tools/testing/selftests/tc-testing/tdc.py | 146 ++++++++++++++++++---
4 files changed, 279 insertions(+), 21 deletions(-)
create mode 100644 tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
create mode 100644 tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py
--
2.7.4
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH net-next 1/3] tc-testing: Add JSON verification to tdc
2019-07-03 2:41 [PATCH net-next 0/3] tc-testing: Add JSON verification and simple traffic generation Lucas Bates
@ 2019-07-03 2:41 ` Lucas Bates
2019-07-03 2:41 ` [PATCH net-next 2/3] tc-testing: Allow tdc plugins to see test case data Lucas Bates
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Lucas Bates @ 2019-07-03 2:41 UTC (permalink / raw)
To: davem
Cc: netdev, jhs, xiyou.wangcong, jiri, mleitner, vladbu, dcaratti,
kernel, Lucas Bates
This patch allows tdc to process JSON output to perform secondary
verification of the command under test. If the verifyCmd generates
JSON, one can provide the 'matchJSON' key to process it
instead of a regex.
matchJSON has two elements: 'path' and 'value'. The 'path' key is a
list of integers and strings that provide the key values for tdc to
navigate the JSON information. The value is an integer or string
that tdc will compare against what it finds in the provided path.
If the numerical position of an element can vary, it's possible to
substitute an asterisk as a wildcard. tdc will search all possible
entries in the array.
Multiple matches are possible, but everything specified must
match for the test to pass.
If both matchPattern and matchJSON are present, tdc will only
operate on matchPattern. If neither are present, verification
is skipped.
Example:
"cmdUnderTest": "$TC actions add action pass index 8",
"verifyCmd": "$TC actions list action gact",
"matchJSON": [
{
"path": [
0,
"actions",
0,
"control action",
"type"
],
"value": "gact"
},
{
"path": [
0,
"actions",
0,
"index"
],
"value": 8
}
]
---
tools/testing/selftests/tc-testing/tdc.py | 136 +++++++++++++++++++++++++++---
1 file changed, 123 insertions(+), 13 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/tdc.py b/tools/testing/selftests/tc-testing/tdc.py
index 678182a..1afa803 100755
--- a/tools/testing/selftests/tc-testing/tdc.py
+++ b/tools/testing/selftests/tc-testing/tdc.py
@@ -35,6 +35,10 @@ class PluginMgrTestFail(Exception):
self.output = output
self.message = message
+class ElemNotFound(Exception):
+ def __init__(self, path_element):
+ self.path_element = path_element
+
class PluginMgr:
def __init__(self, argparser):
super().__init__()
@@ -167,6 +171,40 @@ class PluginMgr:
self.argparser = argparse.ArgumentParser(
description='Linux TC unit tests')
+def find_in_json(jsonobj, path):
+ if type(jsonobj) == list:
+ if type(path[0]) == int:
+ if len(jsonobj) > path[0]:
+ return find_in_json(jsonobj[path[0]], path[1:])
+ else:
+ raise ElemNotFound(path[0])
+ elif path[0] == '*':
+ res = []
+ for index in jsonobj:
+ try:
+ res.append(find_in_json(index, path[1:]))
+ except ElemNotFound:
+ continue
+ if len(res) == 0:
+ raise ElemNotFound(path[0])
+ else:
+ return res
+ elif type(jsonobj) == dict:
+ if path[0] in jsonobj:
+ if len(path) > 1:
+ return find_in_json(jsonobj[path[0]], path[1:])
+ return jsonobj[path[0]]
+ else:
+ raise ElemNotFound(path[0])
+ else:
+ # Assume we have found the correct depth in the object
+ if len(path) >= 1:
+ print('The remainder of the specified path cannot be found!')
+ print('Path values: {}'.format(path))
+ raise ElemNotFound(path[0])
+ return jsonobj
+
+
def replace_keywords(cmd):
"""
For a given executable command, substitute any known
@@ -246,6 +284,86 @@ def prepare_env(args, pm, stage, prefix, cmdlist, output = None):
stage, output,
'"{}" did not complete successfully'.format(prefix))
+def verify_by_regex(res, tidx, args, pm):
+ if 'matchCount' not in tidx:
+ res.set_result(ResultState.skip)
+ fmsg = 'matchCount was not provided in the test case. '
+ fmsg += 'Unable to complete pattern match.'
+ res.set_failmsg(fmsg)
+ print(fmsg)
+ return res
+ (p, procout) = exec_cmd(args, pm, 'verify', tidx["verifyCmd"])
+ match_pattern = re.compile(
+ str(tidx["matchPattern"]), re.DOTALL | re.MULTILINE)
+ if procout:
+ match_index = re.findall(match_pattern, procout)
+ if len(match_index) != int(tidx["matchCount"]):
+ res.set_result(ResultState.fail)
+ fmsg = 'Verify stage failed because the output did not match '
+ fmsg += 'the pattern in the test case.\nMatch pattern is:\n'
+ fmsg += '\t{}\n'.format(tidx["matchPattern"])
+ fmsg += 'Output generated by the verify command:\n'
+ fmsg += '{}\n'.format(procout)
+ res.set_failmsg(fmsg)
+ else:
+ res.set_result(ResultState.success)
+ elif int(tidx["matchCount"]) != 0:
+ res.set_result(ResultState.fail)
+ res.set_failmsg('No output generated by verify command.')
+ else:
+ res.set_result(ResultState.success)
+ return res
+
+def verify_by_json(res, tidx, args, pm):
+ # Validate the matchJSON struct
+ for match in tidx['matchJSON']:
+ if 'path' in match and 'value' in match:
+ pass
+ else:
+ res.set_result(ResultState.skip)
+ res.set_failmsg('matchJSON missing required keys for this case.')
+ return res
+ (p, procout) = exec_cmd(args, pm, 'verify', tidx["verifyCmd"])
+ # Run procout through the JSON decoder
+ try:
+ jsonobj = json.loads(procout)
+ except json.JSONDecodeError:
+ if len(tidx['matchJSON']) > 0:
+ res.set_result(ResultState.fail)
+ res.set_failmsg('Cannot decode verify command\'s output. Is it JSON?')
+ return res
+ # Then recurse through the object
+ valuesmatch = True
+ for match in tidx['matchJSON']:
+ try:
+ value = find_in_json(jsonobj, match['path'])
+ except ElemNotFound as ENF:
+ fmsg = 'Could not find the element {} specified in the path.'.format(ENF.path_element)
+ valuesmatch = False
+ break
+ if type(value) == list:
+ if match['value'] not in value:
+ valuesmatch = False
+ fmsg = 'Verify stage failed because the value specified in the path\n'
+ fmsg += '{}\n'.format(match['path'])
+ fmsg += 'Expected value: {}\nReceived value: {}'.format(
+ match['value'], value)
+ break
+ elif match['value'] != value:
+ valuesmatch = False
+ fmsg = 'Verify stage failed because the value specified in the path\n'
+ fmsg += '{}\n'.format(match['path'])
+ fmsg += 'Expected value: {}\nReceived value: {}'.format(
+ match['value'], value)
+ break
+ if valuesmatch:
+ res.set_result(ResultState.success)
+ else:
+ res.set_result(ResultState.fail)
+ res.set_failmsg(fmsg)
+ print(fmsg)
+ return res
+
def run_one_test(pm, args, index, tidx):
global NAMES
result = True
@@ -292,21 +410,13 @@ def run_one_test(pm, args, index, tidx):
else:
if args.verbose > 0:
print('-----> verify stage')
- match_pattern = re.compile(
- str(tidx["matchPattern"]), re.DOTALL | re.MULTILINE)
- (p, procout) = exec_cmd(args, pm, 'verify', tidx["verifyCmd"])
- if procout:
- match_index = re.findall(match_pattern, procout)
- if len(match_index) != int(tidx["matchCount"]):
- res.set_result(ResultState.fail)
- res.set_failmsg('Could not match regex pattern. Verify command output:\n{}'.format(procout))
- else:
- res.set_result(ResultState.success)
- elif int(tidx["matchCount"]) != 0:
- res.set_result(ResultState.fail)
- res.set_failmsg('No output generated by verify command.')
+ if 'matchPattern' in tidx:
+ res = verify_by_regex(res, tidx, args, pm)
+ elif 'matchJSON' in tidx:
+ res = verify_by_json(res, tidx, args, pm)
else:
res.set_result(ResultState.success)
+ print('No match method defined in current test case, skipping verify')
prepare_env(args, pm, 'teardown', '-----> teardown stage', tidx['teardown'], procout)
pm.call_post_case()
--
2.7.4
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH net-next 2/3] tc-testing: Allow tdc plugins to see test case data
2019-07-03 2:41 [PATCH net-next 0/3] tc-testing: Add JSON verification and simple traffic generation Lucas Bates
2019-07-03 2:41 ` [PATCH net-next 1/3] tc-testing: Add JSON verification to tdc Lucas Bates
@ 2019-07-03 2:41 ` Lucas Bates
2019-07-03 2:41 ` [PATCH net-next 3/3] tc-testing: introduce scapyPlugin for basic traffic Lucas Bates
2019-07-03 18:26 ` [PATCH net-next 0/3] tc-testing: Add JSON verification and simple traffic generation David Miller
3 siblings, 0 replies; 5+ messages in thread
From: Lucas Bates @ 2019-07-03 2:41 UTC (permalink / raw)
To: davem
Cc: netdev, jhs, xiyou.wangcong, jiri, mleitner, vladbu, dcaratti,
kernel, Lucas Bates
Instead of only passing the test case name and ID, pass the
entire current test case down to the plugins. This change
allows plugins to start accepting commands and directives
from the test cases themselves, for greater flexibility
in testing.
---
tools/testing/selftests/tc-testing/TdcPlugin.py | 5 ++---
tools/testing/selftests/tc-testing/tdc.py | 10 +++++-----
2 files changed, 7 insertions(+), 8 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/TdcPlugin.py b/tools/testing/selftests/tc-testing/TdcPlugin.py
index b980a56..79f3ca8 100644
--- a/tools/testing/selftests/tc-testing/TdcPlugin.py
+++ b/tools/testing/selftests/tc-testing/TdcPlugin.py
@@ -18,12 +18,11 @@ class TdcPlugin:
if self.args.verbose > 1:
print(' -- {}.post_suite'.format(self.sub_class))
- def pre_case(self, testid, test_name, test_skip):
+ def pre_case(self, caseinfo, test_skip):
'''run commands before test_runner does one test'''
if self.args.verbose > 1:
print(' -- {}.pre_case'.format(self.sub_class))
- self.args.testid = testid
- self.args.test_name = test_name
+ self.args.caseinfo = caseinfo
self.args.test_skip = test_skip
def post_case(self):
diff --git a/tools/testing/selftests/tc-testing/tdc.py b/tools/testing/selftests/tc-testing/tdc.py
index 1afa803..de7da9a 100755
--- a/tools/testing/selftests/tc-testing/tdc.py
+++ b/tools/testing/selftests/tc-testing/tdc.py
@@ -126,15 +126,15 @@ class PluginMgr:
for pgn_inst in reversed(self.plugin_instances):
pgn_inst.post_suite(index)
- def call_pre_case(self, testid, test_name, *, test_skip=False):
+ def call_pre_case(self, caseinfo, *, test_skip=False):
for pgn_inst in self.plugin_instances:
try:
- pgn_inst.pre_case(testid, test_name, test_skip)
+ pgn_inst.pre_case(caseinfo, test_skip)
except Exception as ee:
print('exception {} in call to pre_case for {} plugin'.
format(ee, pgn_inst.__class__))
print('test_ordinal is {}'.format(test_ordinal))
- print('testid is {}'.format(testid))
+ print('testid is {}'.format(caseinfo['id']))
raise
def call_post_case(self):
@@ -379,14 +379,14 @@ def run_one_test(pm, args, index, tidx):
res = TestResult(tidx['id'], tidx['name'])
res.set_result(ResultState.skip)
res.set_errormsg('Test case designated as skipped.')
- pm.call_pre_case(tidx['id'], tidx['name'], test_skip=True)
+ pm.call_pre_case(tidx, test_skip=True)
pm.call_post_execute()
return res
# populate NAMES with TESTID for this test
NAMES['TESTID'] = tidx['id']
- pm.call_pre_case(tidx['id'], tidx['name'])
+ pm.call_pre_case(tidx)
prepare_env(args, pm, 'setup', "-----> prepare stage", tidx["setup"])
if (args.verbose > 0):
--
2.7.4
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH net-next 3/3] tc-testing: introduce scapyPlugin for basic traffic
2019-07-03 2:41 [PATCH net-next 0/3] tc-testing: Add JSON verification and simple traffic generation Lucas Bates
2019-07-03 2:41 ` [PATCH net-next 1/3] tc-testing: Add JSON verification to tdc Lucas Bates
2019-07-03 2:41 ` [PATCH net-next 2/3] tc-testing: Allow tdc plugins to see test case data Lucas Bates
@ 2019-07-03 2:41 ` Lucas Bates
2019-07-03 18:26 ` [PATCH net-next 0/3] tc-testing: Add JSON verification and simple traffic generation David Miller
3 siblings, 0 replies; 5+ messages in thread
From: Lucas Bates @ 2019-07-03 2:41 UTC (permalink / raw)
To: davem
Cc: netdev, jhs, xiyou.wangcong, jiri, mleitner, vladbu, dcaratti,
kernel, Lucas Bates
The scapyPlugin allows for simple traffic generation in tdc to
test various tc features. It was tested with scapy v2.4.2, but
should work with any successive version.
In order to use the plugin's functionality, scapy must be
installed. This can be done with:
pip3 install scapy
or to install 2.4.2:
pip3 install scapy==2.4.2
If the plugin is unable to import the scapy module, it will
terminate the tdc run.
The plugin makes use of a new key in the test case data, 'scapy'.
This block contains three other elements: 'iface', 'count', and
'packet':
"scapy": {
"iface": "$DEV0",
"count": 1,
"packet": "Ether(type=0x800)/IP(src='16.61.16.61')/ICMP()"
},
* iface is the name of the device on the host machine from which
the packet(s) will be sent. Values contained within tdc_config.py's
NAMES dict can be used here - this is useful if paired with
nsPlugin
* count is the number of copies of this packet to be sent
* packet is a string detailing the different layers of the packet
to be sent. If a property isn't explicitly set, scapy will set
default values for you.
Layers in the packet info are separated by slashes. For info about
common TCP and IP properties, see:
https://blogs.sans.org/pen-testing/files/2016/04/ScapyCheatSheet_v0.2.pdf
Caution is advised when running tests using the scapy functionality,
since the plugin blindly sends the packet as defined in the test case
data.
See creating-testcases/scapy-example.json for sample test cases;
the first test is intended to pass while the second is intended to
fail. Consider using the matchJSON functionality for verification
when using scapy.
---
.../creating-testcases/scapy-example.json | 98 ++++++++++++++++++++++
.../selftests/tc-testing/plugin-lib/scapyPlugin.py | 51 +++++++++++
2 files changed, 149 insertions(+)
create mode 100644 tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
create mode 100644 tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py
diff --git a/tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json b/tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
new file mode 100644
index 0000000..5a9377b
--- /dev/null
+++ b/tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
@@ -0,0 +1,98 @@
+[
+ {
+ "id": "b1e9",
+ "name": "Test matching of source IP",
+ "category": [
+ "actions",
+ "scapy"
+ ],
+ "plugins": {
+ "requires": [
+ "nsPlugin",
+ "scapyPlugin"
+ ]
+ },
+ "setup": [
+ [
+ "$TC qdisc del dev $DEV1 ingress",
+ 0,
+ 1,
+ 2,
+ 255
+ ],
+ "$TC qdisc add dev $DEV1 ingress"
+ ],
+ "cmdUnderTest": "$TC filter add dev $DEV1 parent ffff: prio 3 protocol ip flower src_ip 16.61.16.61 flowid 1:1 action ok",
+ "scapy": {
+ "iface": "$DEV0",
+ "count": 1,
+ "packet": "Ether(type=0x800)/IP(src='16.61.16.61')/ICMP()"
+ },
+ "expExitCode": "0",
+ "verifyCmd": "$TC -s -j filter ls dev $DEV1 ingress prio 3",
+ "matchJSON": [
+ {
+ "path": [
+ 1,
+ "options",
+ "actions",
+ 0,
+ "stats",
+ "packets"
+ ],
+ "value": 1
+ }
+ ],
+ "teardown": [
+ "$TC qdisc del dev $DEV1 ingress"
+ ]
+ },
+ {
+ "id": "e9c4",
+ "name": "Test matching of source IP with wrong count",
+ "category": [
+ "actions",
+ "scapy"
+ ],
+ "plugins": {
+ "requires": [
+ "nsPlugin",
+ "scapyPlugin"
+ ]
+ },
+ "setup": [
+ [
+ "$TC qdisc del dev $DEV1 ingress",
+ 0,
+ 1,
+ 2,
+ 255
+ ],
+ "$TC qdisc add dev $DEV1 ingress"
+ ],
+ "cmdUnderTest": "$TC filter add dev $DEV1 parent ffff: prio 3 protocol ip flower src_ip 16.61.16.61 flowid 1:1 action ok",
+ "scapy": {
+ "iface": "$DEV0",
+ "count": 3,
+ "packet": "Ether(type=0x800)/IP(src='16.61.16.61')/ICMP()"
+ },
+ "expExitCode": "0",
+ "verifyCmd": "$TC -s -j filter ls dev $DEV1 parent ffff:",
+ "matchJSON": [
+ {
+ "path": [
+ 1,
+ "options",
+ "actions",
+ 0,
+ "stats",
+ "packets"
+ ],
+ "value": 1
+ }
+ ],
+ "teardown": [
+ "$TC qdisc del dev $DEV1 ingress"
+ ]
+ }
+]
diff --git a/tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py b/tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py
new file mode 100644
index 0000000..db57916
--- /dev/null
+++ b/tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py
@@ -0,0 +1,51 @@
+#!/usr/bin/env python3
+
+import os
+import signal
+from string import Template
+import subprocess
+import time
+from TdcPlugin import TdcPlugin
+
+from tdc_config import *
+
+try:
+ from scapy.all import *
+except ImportError:
+ print("Unable to import the scapy python module.")
+ print("\nIf not already installed, you may do so with:")
+ print("\t\tpip3 install scapy==2.4.2")
+ exit(1)
+
+class SubPlugin(TdcPlugin):
+ def __init__(self):
+ self.sub_class = 'scapy/SubPlugin'
+ super().__init__()
+
+ def post_execute(self):
+ if 'scapy' not in self.args.caseinfo:
+ if self.args.verbose:
+ print('{}.post_execute: no scapy info in test case'.format(self.sub_class))
+ return
+
+ # Check for required fields
+ scapyinfo = self.args.caseinfo['scapy']
+ scapy_keys = ['iface', 'count', 'packet']
+ missing_keys = []
+ keyfail = False
+ for k in scapy_keys:
+ if k not in scapyinfo:
+ keyfail = True
+ missing_keys.add(k)
+ if keyfail:
+ print('{}: Scapy block present in the test, but is missing info:'
+ .format(self.sub_class))
+ print('{}'.format(missing_keys))
+
+ pkt = eval(scapyinfo['packet'])
+ if '$' in scapyinfo['iface']:
+ tpl = Template(scapyinfo['iface'])
+ scapyinfo['iface'] = tpl.safe_substitute(NAMES)
+ for count in range(scapyinfo['count']):
+ sendp(pkt, iface=scapyinfo['iface'])
+
--
2.7.4
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH net-next 0/3] tc-testing: Add JSON verification and simple traffic generation
2019-07-03 2:41 [PATCH net-next 0/3] tc-testing: Add JSON verification and simple traffic generation Lucas Bates
` (2 preceding siblings ...)
2019-07-03 2:41 ` [PATCH net-next 3/3] tc-testing: introduce scapyPlugin for basic traffic Lucas Bates
@ 2019-07-03 18:26 ` David Miller
3 siblings, 0 replies; 5+ messages in thread
From: David Miller @ 2019-07-03 18:26 UTC (permalink / raw)
To: lucasb
Cc: netdev, jhs, xiyou.wangcong, jiri, mleitner, vladbu, dcaratti,
kernel
This entire patch series lacks proper signoffs.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-07-03 18:26 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-07-03 2:41 [PATCH net-next 0/3] tc-testing: Add JSON verification and simple traffic generation Lucas Bates
2019-07-03 2:41 ` [PATCH net-next 1/3] tc-testing: Add JSON verification to tdc Lucas Bates
2019-07-03 2:41 ` [PATCH net-next 2/3] tc-testing: Allow tdc plugins to see test case data Lucas Bates
2019-07-03 2:41 ` [PATCH net-next 3/3] tc-testing: introduce scapyPlugin for basic traffic Lucas Bates
2019-07-03 18:26 ` [PATCH net-next 0/3] tc-testing: Add JSON verification and simple traffic generation David Miller
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).