[PATCH nft 0/12] add support for VLAN header filtering in bridge family

[PATCH nft 0/12] add support for VLAN header filtering in bridge family

[Florian Westphal]

This adds the following features:
- stacked headers: vlan sits on top of ethernet, so when vlan id 42 is requested,
  add a dependency on eth and also adjust payload offsets by the size of ethernet
  header.

- allow matching header fields of lengths and/or offsets that are not byte divisible,
  e.g. vlan id, vlan pcp, ip hdrlength, etc.

Missing/not working:

- concat support for vlan ids, e.f. this will not work:
vlan id . ip saddr { 1 . 1.2.3.4 }

- payload merging of fields that have lengths/sizes not divisible by 8.

IOW when asking for vlan id 42 vlan pcp 0 then we generate 2 payload instructions,
2 binops, 2 cmps.

Oddities:

bridge filter input ip saddr 1.2.3.4

... will never match in some cases.

'Problem' (or feature...?) is that nft adds dependency on 'ether type ip'
so if all traffic is VLAN encapsulated this will not match.

[ can use 'vlan type ip ip saddr 1.2.3.4' to match eth-vlan-ip packets ].

There might be a lot of pitfalls that i've missed.

Unless there are objections I will push the first two patches soon.

The vlan patch set has seen minimal testing inside kvm on x86_64,
using a (pending) kernel patch to nft_payload to transparently insert
the missing/stripped vlan header into the register set.

I'd be happy to receive feedback on these patches, I'm sure that some
things can be implemented in a more straightforward fashion.

Cheers,
Florian

[PATCH 01/12] tests: use the src/nft binary instead of $PATH one

[Florian Westphal]

... so one doesn't need to install new binary into $PATH (or
change PATH... ) during development.

Signed-off-by: Florian Westphal <fw@strlen.de>
---
 tests/regression/nft-test.py | 35 ++++++++++++++++++-----------------
 1 file changed, 18 insertions(+), 17 deletions(-)

diff --git a/tests/regression/nft-test.py b/tests/regression/nft-test.py
index 26fc2ec..c3d2aae 100755
--- a/tests/regression/nft-test.py
+++ b/tests/regression/nft-test.py
@@ -20,6 +20,7 @@ import argparse
 import signal
 
 TERMINAL_PATH = os.getcwd()
+NFT_BIN = TERMINAL_PATH + "/src/nft"
 TESTS_PATH = os.path.dirname(os.path.abspath(__file__))
 TESTS_DIRECTORY = ["any", "arp", "bridge", "inet", "ip", "ip6"]
 LOGFILE = "/tmp/nftables-test.log"
@@ -77,7 +78,7 @@ def table_exist(table, filename, lineno):
     '''
     Exists a table.
     '''
-    cmd = "nft list -nnn table " + table[0] + " " + table[1]
+    cmd = NFT_BIN + " list -nnn table " + table[0] + " " + table[1]
     ret = execute_cmd(cmd, filename, lineno)
 
     return True if (ret == 0) else False
@@ -87,7 +88,7 @@ def table_flush(table, filename, lineno):
     '''
     Flush a table.
     '''
-    cmd = "nft flush table " + str(table[0]) + " " + str(table[1])
+    cmd = NFT_BIN + " flush table " + str(table[0]) + " " + str(table[1])
     ret = execute_cmd(cmd, filename, lineno)
 
     return cmd
@@ -106,7 +107,7 @@ def table_create(table, filename, lineno):
     table_list.append(table)
 
     ## We add a new table
-    cmd = "nft add table " + table[0] + " " + table[1]
+    cmd = NFT_BIN + " add table " + table[0] + " " + table[1]
     ret = execute_cmd(cmd, filename, lineno)
 
     if ret != 0:
@@ -138,7 +139,7 @@ def table_delete(table, filename=None, lineno=None):
         print_error(reason, filename, lineno)
         return -1
 
-    cmd = "nft delete table" + table_info
+    cmd = NFT_BIN + " delete table" + table_info
     ret = execute_cmd(cmd, filename, lineno)
     if ret != 0:
         reason = cmd + ": " \
@@ -161,7 +162,7 @@ def chain_exist(chain, table, filename, lineno):
     '''
 
     table_info = " " + table[0] + " " + table[1] + " "
-    cmd = "nft list -nnn chain" + table_info + chain
+    cmd = NFT_BIN + " list -nnn chain" + table_info + chain
     ret = execute_cmd(cmd, filename, lineno)
 
     return True if (ret == 0) else False
@@ -181,9 +182,9 @@ def chain_create(chain, chain_type, chain_list, table, filename, lineno):
         return -1
 
     if chain_type:
-        cmd = "nft add chain" + table_info + chain + "\{ " + chain_type + "\; \}"
+        cmd = NFT_BIN + " add chain" + table_info + chain + "\{ " + chain_type + "\; \}"
     else:
-        cmd = "nft add chain" + table_info + chain
+        cmd = NFT_BIN + " add chain" + table_info + chain
 
     ret = execute_cmd(cmd, filename, lineno)
     if ret != 0:
@@ -216,14 +217,14 @@ def chain_delete(chain, table,  filename=None, lineno=None):
         print_error(reason, filename, lineno)
         return -1
 
-    cmd = "nft flush chain" + table_info + chain
+    cmd = NFT_BIN + " flush chain" + table_info + chain
     ret = execute_cmd(cmd, filename, lineno)
     if ret != 0:
         reason = "I cannot flush this chain " + chain
         print_error(reason, filename, lineno)
         return -1
 
-    cmd = "nft delete chain" + table_info + chain
+    cmd = NFT_BIN + " delete chain" + table_info + chain
     ret = execute_cmd(cmd, filename, lineno)
     if ret != 0:
         reason = cmd + "I cannot delete this chain. DD"
@@ -258,7 +259,7 @@ def set_add(set_info, table_list, filename, lineno):
 
         table_info = " " + table[0] + " " + table[1] + " "
         set_text = " " + set_info[0] + " { type " + set_info[1] + " \;}"
-        cmd = "nft add set" + table_info + set_text
+        cmd = NFT_BIN + " add set" + table_info + set_text
         ret = execute_cmd(cmd, filename, lineno)
 
         if (ret == 0 and set_info[2].rstrip() == "fail") or \
@@ -306,7 +307,7 @@ def set_add_elements(set_element, set_name, set_all, state, table_list,
                 element = element + ", " + e
 
         set_text = set_name + " { " + element + " }"
-        cmd = "nft add element" + table_info + set_text
+        cmd = NFT_BIN + " add element" + table_info + set_text
         ret = execute_cmd(cmd, filename, lineno)
 
         if (state == "fail" and ret == 0) or (state == "ok" and ret != 0):
@@ -332,7 +333,7 @@ def set_delete_elements(set_element, set_name, table, filename=None,
 
     for element in set_element:
         set_text = set_name + " {" + element + "}"
-        cmd = "nft delete element" + table_info + set_text
+        cmd = NFT_BIN + " delete element" + table_info + set_text
         ret = execute_cmd(cmd, filename, lineno)
         if ret != 0:
             reason = "I cannot delete an element" + element + \
@@ -362,7 +363,7 @@ def set_delete(all_set, table, filename=None, lineno=None):
 
         # We delete the set.
         table_info = " " + table[0] + " " + table[1] + " "
-        cmd = "nft delete set " + table_info + " " + set_name
+        cmd = NFT_BIN + " delete set " + table_info + " " + set_name
         ret = execute_cmd(cmd, filename, lineno)
 
         # Check if the set still exists after I deleted it.
@@ -379,7 +380,7 @@ def set_exist(set_name, table, filename, lineno):
     Check if the set exists.
     '''
     table_info = " " + table[0] + " " + table[1] + " "
-    cmd = "nft list -nnn set" + table_info + set_name
+    cmd = NFT_BIN + " list -nnn set" + table_info + set_name
     ret = execute_cmd(cmd, filename, lineno)
 
     return True if (ret == 0) else False
@@ -487,11 +488,11 @@ def rule_add(rule, table_list, chain_list, filename, lineno,
             unit_tests += 1
             table_flush(table, filename, lineno)
             table_info = " " + table[0] + " " + table[1] + " "
-            cmd = "nft add rule" + table_info + chain + " " + rule[0]
+            cmd = NFT_BIN + " add rule" + table_info + chain + " " + rule[0]
 
             payload_log = os.tmpfile();
 
-            cmd = "nft add rule --debug=netlink" + table_info + chain + " " + rule[0]
+            cmd = NFT_BIN + " add rule --debug=netlink" + table_info + chain + " " + rule[0]
             ret = execute_cmd(cmd, filename, lineno, payload_log)
 
             state = rule[1].rstrip()
@@ -527,7 +528,7 @@ def rule_add(rule, table_list, chain_list, filename, lineno,
                     print_warning("Wrote payload for rule %s" % rule[0], gotf.name, 1)
 
             # Check output of nft
-                process = subprocess.Popen(['nft', '-nnn', 'list', 'table'] + table,
+                process = subprocess.Popen([NFT_BIN, '-nnn', 'list', 'table'] + table,
                                            shell=False, stdout=subprocess.PIPE,
                                            preexec_fn=preexec)
                 pre_output = process.communicate()
-- 
2.0.5

[PATCH 02/12] tests: add 'awkward' prefix match expression

[Florian Westphal]

Its just a more complicated way of saying 'ip saddr 255.255.0.0/16'.

Signed-off-by: Florian Westphal <fw@strlen.de>
---
 tests/regression/ip/ip.t              | 3 +++
 tests/regression/ip/ip.t.payload      | 6 ++++++
 tests/regression/ip/ip.t.payload.inet | 8 ++++++++
 3 files changed, 17 insertions(+)

diff --git a/tests/regression/ip/ip.t b/tests/regression/ip/ip.t
index 11a9598..6e29ffb 100644
--- a/tests/regression/ip/ip.t
+++ b/tests/regression/ip/ip.t
@@ -108,3 +108,6 @@ ip daddr 192.168.0.2 log;ok
 
 ip saddr \& 0xff == 1;ok;ip saddr & 0.0.0.255 == 0.0.0.1
 ip saddr \& 0.0.0.255 \< 0.0.0.127;ok;ip saddr & 0.0.0.255 < 0.0.0.127
+
+ip saddr \& 0xffff0000 == 0xffff0000;ok;ip saddr 255.255.0.0/16
+
diff --git a/tests/regression/ip/ip.t.payload b/tests/regression/ip/ip.t.payload
index ec4897e..7a77dc4 100644
--- a/tests/regression/ip/ip.t.payload
+++ b/tests/regression/ip/ip.t.payload
@@ -347,3 +347,9 @@ ip test-ip4 input
   [ bitwise reg 1 = (reg=1 & 0xff000000 ) ^ 0x00000000 ]
   [ cmp lt reg 1 0x7f000000 ]
 
+# ip saddr \& 0xffff0000 == 0xffff0000
+ip test-ip4 input
+  [ payload load 4b @ network header + 12 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000ffff ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x0000ffff ]
+
diff --git a/tests/regression/ip/ip.t.payload.inet b/tests/regression/ip/ip.t.payload.inet
index 1a2b1b5..dbc7852 100644
--- a/tests/regression/ip/ip.t.payload.inet
+++ b/tests/regression/ip/ip.t.payload.inet
@@ -457,3 +457,11 @@ inet test-inet input
   [ bitwise reg 1 = (reg=1 & 0xff000000 ) ^ 0x00000000 ]
   [ cmp lt reg 1 0x7f000000 ]
 
+# ip saddr \& 0xffff0000 == 0xffff0000
+inet test-inet input
+  [ meta load nfproto => reg 1 ]
+  [ cmp eq reg 1 0x00000002 ]
+  [ payload load 4b @ network header + 12 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000ffff ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x0000ffff ]
+
-- 
2.0.5

[PATCH 03/12] nft: allow stacking vlan header on top of ethernet

[Florian Westphal]

currently 'vlan id 42' or even 'vlan type ip' doesn't work since
we expect ethernet header but get vlan.

So if we want to add another protocol header to the same base, we
attempt to figure out if the new header can fit on top of the existing
one (i.e. proto_find_num gives a protocol number when asking to find
link between the two).

We also annotate protocol description for eth and vlan with the full
header size and track the offset from the current base.

Otherwise, 'vlan type ip' fetches the protocol field from mac header
offset 0, which is some mac address.

Instead, we must consider full size of ethernet header.

Signed-off-by: Florian Westphal <fw@strlen.de>
---
 include/proto.h           |  4 +++
 src/evaluate.c            | 78 ++++++++++++++++++++++++++++++++++++++++++++++-
 src/netlink_delinearize.c | 48 +++++++++++++++++++++++------
 src/payload.c             |  1 +
 src/proto.c               |  2 ++
 5 files changed, 123 insertions(+), 10 deletions(-)

diff --git a/include/proto.h b/include/proto.h
index 0e531b2..a43bf98 100644
--- a/include/proto.h
+++ b/include/proto.h
@@ -69,6 +69,7 @@ struct proto_hdr_template {
  * @name:	protocol name
  * @base:	header base
  * @protocol_key: key of template containing upper layer protocol description
+ * @length:	total size of the header, in bits
  * @protocols:	link to upper layer protocol descriptions indexed by protocol value
  * @templates:	header templates
  */
@@ -76,6 +77,7 @@ struct proto_desc {
 	const char			*name;
 	enum proto_bases		base;
 	unsigned int			protocol_key;
+	unsigned int			length;
 	struct {
 		unsigned int			num;
 		const struct proto_desc		*desc;
@@ -122,6 +124,7 @@ extern const struct proto_desc *proto_dev_desc(uint16_t type);
  * @family:	hook family
  * @location:	location of the relational expression defining the context
  * @desc:	protocol description for this layer
+ * @offset:	offset from the base, for stacked headers (eg 8*14 for vlan on top of ether)
  *
  * The location of the context is the location of the relational expression
  * defining it, either directly through a protocol match or indirectly
@@ -132,6 +135,7 @@ struct proto_ctx {
 	struct {
 		struct location			location;
 		const struct proto_desc		*desc;
+		unsigned int			offset;
 	} protocol[PROTO_BASE_MAX + 1];
 };
 
diff --git a/src/evaluate.c b/src/evaluate.c
index d99b38f..c95d154 100644
--- a/src/evaluate.c
+++ b/src/evaluate.c
@@ -271,6 +271,82 @@ static int expr_evaluate_primary(struct eval_ctx *ctx, struct expr **expr)
 	return 0;
 }
 
+static int
+conflict_resolution_gen_dependency(struct eval_ctx *ctx, int protocol,
+				   const struct expr *expr,
+				   struct stmt **res)
+{
+	enum proto_bases base = expr->payload.base;
+	const struct proto_hdr_template *tmpl;
+	const struct proto_desc *desc = NULL;
+	struct expr *dep, *left, *right;
+	struct stmt *stmt;
+
+	assert(expr->payload.base == PROTO_BASE_LL_HDR);
+
+	desc = ctx->pctx.protocol[base].desc;
+	tmpl = &desc->templates[desc->protocol_key];
+	left = payload_expr_alloc(&expr->location, desc, desc->protocol_key);
+
+	right = constant_expr_alloc(&expr->location, tmpl->dtype,
+				    tmpl->dtype->byteorder, tmpl->len,
+				    constant_data_ptr(protocol, tmpl->len));
+
+	dep = relational_expr_alloc(&expr->location, OP_EQ, left, right);
+	stmt = expr_stmt_alloc(&dep->location, dep);
+	if (stmt_evaluate(ctx, stmt) < 0)
+		return expr_error(ctx->msgs, expr,
+					  "dependency statement is invalid");
+
+	ctx->pctx.protocol[base].desc = expr->payload.desc;
+	assert(ctx->pctx.protocol[base].offset == 0);
+
+	assert(desc->length);
+	ctx->pctx.protocol[base].offset += desc->length;
+
+	*res = stmt;
+	return 0;
+}
+
+static bool resolve_protocol_conflict(struct eval_ctx *ctx,
+				      struct expr *payload)
+{
+	const struct hook_proto_desc *h = &hook_proto_desc[ctx->pctx.family];
+	enum proto_bases base = payload->payload.base;
+	const struct proto_desc *desc;
+	struct stmt *nstmt = NULL;
+	int link;
+
+	desc = ctx->pctx.protocol[base].desc;
+
+	if (desc == payload->payload.desc) {
+		payload->payload.offset += ctx->pctx.protocol[base].offset;
+		return true;
+	}
+
+	if (payload->payload.base != h->base)
+		return false;
+
+	assert(desc->length);
+	if (base < PROTO_BASE_MAX) {
+		const struct proto_desc *next = ctx->pctx.protocol[base + 1].desc;
+
+		if (payload->payload.desc == next) {
+			payload->payload.offset += desc->length;
+			return true;
+		}
+	}
+
+	link = proto_find_num(desc, payload->payload.desc);
+	if (link < 0 || conflict_resolution_gen_dependency(ctx, link, payload, &nstmt) < 0)
+		return false;
+
+	payload->payload.offset += ctx->pctx.protocol[base].offset;
+	list_add_tail(&nstmt->list, &ctx->stmt->list);
+
+	return true;
+}
+
 /*
  * Payload expression: check whether dependencies are fulfilled, otherwise
  * generate the necessary relational expression and prepend it to the current
@@ -286,7 +362,7 @@ static int expr_evaluate_payload(struct eval_ctx *ctx, struct expr **expr)
 		if (payload_gen_dependency(ctx, payload, &nstmt) < 0)
 			return -1;
 		list_add_tail(&nstmt->list, &ctx->stmt->list);
-	} else if (ctx->pctx.protocol[base].desc != payload->payload.desc)
+	} else if (!resolve_protocol_conflict(ctx, payload))
 		return expr_error(ctx->msgs, payload,
 				  "conflicting protocols specified: %s vs. %s",
 				  ctx->pctx.protocol[base].desc->name,
diff --git a/src/netlink_delinearize.c b/src/netlink_delinearize.c
index 4226b82..76c07c5 100644
--- a/src/netlink_delinearize.c
+++ b/src/netlink_delinearize.c
@@ -923,7 +923,9 @@ static void payload_match_expand(struct rule_pp_ctx *ctx, struct expr *expr)
 	struct expr *left = expr->left, *right = expr->right, *tmp;
 	struct list_head list = LIST_HEAD_INIT(list);
 	struct stmt *nstmt;
-	struct expr *nexpr;
+	struct expr *nexpr = NULL;
+	enum proto_bases base = left->payload.base;
+	const struct expr_ops *payload_ops = left->ops;
 
 	payload_expr_expand(&list, left, &ctx->pctx);
 	list_for_each_entry(left, &list, list) {
@@ -940,17 +942,39 @@ static void payload_match_expand(struct rule_pp_ctx *ctx, struct expr *expr)
 		nstmt = expr_stmt_alloc(&ctx->stmt->location, nexpr);
 		list_add_tail(&nstmt->list, &ctx->stmt->list);
 
+		assert(left->ops == payload_ops);
+		assert(left->payload.base);
+		assert(base == left->payload.base);
+
 		/* Remember the first payload protocol expression to
 		 * kill it later on if made redundant by a higher layer
 		 * payload expression.
 		 */
 		if (ctx->pbase == PROTO_BASE_INVALID &&
-		    left->flags & EXPR_F_PROTOCOL)
-			payload_dependency_store(ctx, nstmt,
-						 left->payload.base);
-		else
+		    left->flags & EXPR_F_PROTOCOL) {
+			unsigned int proto = mpz_get_be16(tmp->value);
+			const struct proto_desc *desc, *next;
+			bool stacked_header = false;
+
+			desc = ctx->pctx.protocol[base].desc;
+			assert(desc);
+			if (desc) {
+				next = proto_find_upper(desc, proto);
+				stacked_header = next && next->base == base;
+			}
+
+			if (stacked_header) {
+				ctx->pctx.protocol[base].desc = next;
+				ctx->pctx.protocol[base].offset += desc->length;
+				payload_dependency_store(ctx, nstmt, base - 1);
+			} else {
+				payload_dependency_store(ctx, nstmt, base);
+			}
+		} else {
 			payload_dependency_kill(ctx, nexpr->left);
+		}
 	}
+
 	list_del(&ctx->stmt->list);
 	stmt_free(ctx->stmt);
 	ctx->stmt = NULL;
@@ -959,6 +983,12 @@ static void payload_match_expand(struct rule_pp_ctx *ctx, struct expr *expr)
 static void payload_match_postprocess(struct rule_pp_ctx *ctx,
 				      struct expr *expr)
 {
+	enum proto_bases base = expr->left->payload.base;
+	struct expr *payload = expr->left;
+
+	assert(payload->payload.offset >= ctx->pctx.protocol[base].offset);
+	payload->payload.offset -= ctx->pctx.protocol[base].offset;
+
 	switch (expr->op) {
 	case OP_EQ:
 	case OP_NEQ:
@@ -968,10 +998,10 @@ static void payload_match_postprocess(struct rule_pp_ctx *ctx,
 		}
 		/* Fall through */
 	default:
-		payload_expr_complete(expr->left, &ctx->pctx);
-		expr_set_type(expr->right, expr->left->dtype,
-			      expr->left->byteorder);
-		payload_dependency_kill(ctx, expr->left);
+		payload_expr_complete(payload, &ctx->pctx);
+		expr_set_type(expr->right, payload->dtype,
+			      payload->byteorder);
+		payload_dependency_kill(ctx, payload);
 		break;
 	}
 }
diff --git a/src/payload.c b/src/payload.c
index 5f8c4fe..bd574ee 100644
--- a/src/payload.c
+++ b/src/payload.c
@@ -340,6 +340,7 @@ void payload_expr_expand(struct list_head *list, struct expr *expr,
 		} else
 			break;
 	}
+
 raw:
 	new = payload_expr_alloc(&expr->location, NULL, 0);
 	payload_init_raw(new, expr->payload.base, expr->payload.offset,
diff --git a/src/proto.c b/src/proto.c
index 6302016..27ab4a2 100644
--- a/src/proto.c
+++ b/src/proto.c
@@ -713,6 +713,7 @@ const struct proto_desc proto_vlan = {
 	.name		= "vlan",
 	.base		= PROTO_BASE_LL_HDR,
 	.protocol_key	= VLANHDR_TYPE,
+	.length		= sizeof(struct vlan_hdr) * BITS_PER_BYTE,
 	.protocols	= {
 		PROTO_LINK(__constant_htons(ETH_P_IP),		&proto_ip),
 		PROTO_LINK(__constant_htons(ETH_P_ARP),		&proto_arp),
@@ -782,6 +783,7 @@ const struct proto_desc proto_eth = {
 	.name		= "ether",
 	.base		= PROTO_BASE_LL_HDR,
 	.protocol_key	= ETHHDR_TYPE,
+	.length		= sizeof(struct ether_header) * BITS_PER_BYTE,
 	.protocols	= {
 		PROTO_LINK(__constant_htons(ETH_P_IP),		&proto_ip),
 		PROTO_LINK(__constant_htons(ETH_P_ARP),		&proto_arp),
-- 
2.0.5

[PATCH 04/12] payload: disable payload merge if offsets are not on byte boundary

[Florian Westphal]

... because it doesn't work, we attempt to merge it into wrong
place, we would have to merge the second value at a specific location.

F.e. vlan hdr 4094 gives us

0xfe0f

Merging in the CFI should yield 0xfe1f, but the constant merging
doesn't know how to achive that; at the moment 'vlan id 4094'
and 'vlan id 4094 vlan cfi 1' give same result -- 0xfe0f.

For now just turn off the optimization step unless everything is
byte divisible (the common case).

Signed-off-by: Florian Westphal <fw@strlen.de>
---
 src/payload.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/src/payload.c b/src/payload.c
index bd574ee..372dcec 100644
--- a/src/payload.c
+++ b/src/payload.c
@@ -357,6 +357,10 @@ raw:
  */
 bool payload_is_adjacent(const struct expr *e1, const struct expr *e2)
 {
+	if (e1->payload.offset % BITS_PER_BYTE || e1->len % BITS_PER_BYTE ||
+	    e2->payload.offset % BITS_PER_BYTE || e2->len % BITS_PER_BYTE)
+		return false;
+
 	if (e1->payload.base		 == e2->payload.base &&
 	    e1->payload.offset + e1->len == e2->payload.offset)
 		return true;
-- 
2.0.5

[PATCH 05/12] src: netlink_linearize: handle sub-byte lengths

[Florian Westphal]

Currently length is expr->len / BITS_PER_BYTE, i.e. expr->len
has to be a multiple of 8.

When core asks for e.g. '9 bits', we truncate this to 8.
Round up to 16 and inject a 9-bit mask to zero out the parts we're not
interested in.

This will also need change to the delinarization step to
remove the extra op when dumping rules from kernel.

Signed-off-by: Florian Westphal <fw@strlen.de>
---
 src/netlink_linearize.c | 45 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 44 insertions(+), 1 deletion(-)

diff --git a/src/netlink_linearize.c b/src/netlink_linearize.c
index bf1e56b..0467872 100644
--- a/src/netlink_linearize.c
+++ b/src/netlink_linearize.c
@@ -99,6 +99,44 @@ static void netlink_gen_concat(struct netlink_linearize_ctx *ctx,
 	}
 }
 
+static void netlink_gen_payload_mask(struct netlink_linearize_ctx *ctx,
+				     const struct expr *expr,
+				     enum nft_registers dreg)
+{
+	struct nft_data_linearize nld, zero = {};
+	struct nft_rule_expr *nle;
+	unsigned int offset, len, masklen;
+	mpz_t mask;
+
+	offset = expr->payload.offset % BITS_PER_BYTE;
+	masklen = expr->len + offset;
+
+	if (masklen > 128)
+		BUG("expr mask length is %u (len %u, offset %u)\n",
+				masklen, expr->len, offset);
+
+	mpz_init2(mask, masklen);
+	mpz_bitmask(mask, expr->len);
+
+	if (offset)
+		mpz_lshift_ui(mask, offset);
+
+	nle = alloc_nft_expr("bitwise");
+
+	len = div_round_up(expr->len, BITS_PER_BYTE);
+
+	nft_rule_expr_set_u32(nle, NFT_EXPR_BITWISE_SREG, dreg);
+	nft_rule_expr_set_u32(nle, NFT_EXPR_BITWISE_DREG, dreg);
+	nft_rule_expr_set_u32(nle, NFT_EXPR_BITWISE_LEN, len);
+
+	netlink_gen_raw_data(mask, expr->byteorder, len, &nld);
+	nft_rule_expr_set(nle, NFT_EXPR_BITWISE_MASK, nld.value, nld.len);
+	nft_rule_expr_set(nle, NFT_EXPR_BITWISE_XOR, &zero.value, nld.len);
+
+	mpz_clear(mask);
+	nft_rule_add_expr(ctx->nlr, nle);
+}
+
 static void netlink_gen_payload(struct netlink_linearize_ctx *ctx,
 				const struct expr *expr,
 				enum nft_registers dreg)
@@ -111,9 +149,14 @@ static void netlink_gen_payload(struct netlink_linearize_ctx *ctx,
 			      expr->payload.base - 1);
 	nft_rule_expr_set_u32(nle, NFT_EXPR_PAYLOAD_OFFSET,
 			      expr->payload.offset / BITS_PER_BYTE);
+
 	nft_rule_expr_set_u32(nle, NFT_EXPR_PAYLOAD_LEN,
-			      expr->len / BITS_PER_BYTE);
+			      div_round_up(expr->len, BITS_PER_BYTE));
+
 	nft_rule_add_expr(ctx->nlr, nle);
+
+	if (expr->len % BITS_PER_BYTE)
+		netlink_gen_payload_mask(ctx, expr, dreg);
 }
 
 static void netlink_gen_exthdr(struct netlink_linearize_ctx *ctx,
-- 
2.0.5

[PATCH 06/12] src: netlink: don't truncate set key lengths

[Florian Westphal]

If key is e.g. 12 bits, pretend its 16 instead of 8.  This is needed
to make sets work with header fields with size not divisible by 8.

Signed-off-by: Florian Westphal <fw@strlen.de>
---
 src/netlink.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/netlink.c b/src/netlink.c
index 8ede8e6..b1296cf 100644
--- a/src/netlink.c
+++ b/src/netlink.c
@@ -1100,7 +1100,7 @@ static int netlink_add_set_compat(struct netlink_ctx *ctx,
 	nft_set_attr_set_u32(nls, NFT_SET_ATTR_KEY_TYPE,
 			     dtype_map_to_kernel(set->keytype));
 	nft_set_attr_set_u32(nls, NFT_SET_ATTR_KEY_LEN,
-			     set->keylen / BITS_PER_BYTE);
+			     div_round_up(set->keylen, BITS_PER_BYTE));
 	if (set->flags & NFT_SET_MAP) {
 		nft_set_attr_set_u32(nls, NFT_SET_ATTR_DATA_TYPE,
 				     dtype_map_to_kernel(set->datatype));
@@ -1135,7 +1135,7 @@ static int netlink_add_set_batch(struct netlink_ctx *ctx,
 	nft_set_attr_set_u32(nls, NFT_SET_ATTR_KEY_TYPE,
 			     dtype_map_to_kernel(set->keytype));
 	nft_set_attr_set_u32(nls, NFT_SET_ATTR_KEY_LEN,
-			     set->keylen / BITS_PER_BYTE);
+			     div_round_up(set->keylen, BITS_PER_BYTE));
 	if (set->flags & NFT_SET_MAP) {
 		nft_set_attr_set_u32(nls, NFT_SET_ATTR_DATA_TYPE,
 				     dtype_map_to_kernel(set->datatype));
-- 
2.0.5

[PATCH 07/12] nft: fill in doff and fix ihl/version template entries

[Florian Westphal]

This allows to use

nft add rule ip filter input tcp doff 8

or similar.

Furhermore, ip version looked at hdrlen and vice versa.

Signed-off-by: Florian Westphal <fw@strlen.de>
---
 src/proto.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/src/proto.c b/src/proto.c
index 27ab4a2..82cce05 100644
--- a/src/proto.c
+++ b/src/proto.c
@@ -402,8 +402,10 @@ const struct proto_desc proto_tcp = {
 		[TCPHDR_DPORT]		= INET_SERVICE("dport", struct tcphdr, dest),
 		[TCPHDR_SEQ]		= TCPHDR_FIELD("sequence", seq),
 		[TCPHDR_ACKSEQ]		= TCPHDR_FIELD("ackseq", ack_seq),
-		[TCPHDR_DOFF]		= {},
-		[TCPHDR_RESERVED]	= {},
+		[TCPHDR_DOFF]		= HDR_BITFIELD("doff", &integer_type,
+						       (12 * BITS_PER_BYTE) + 4, 4),
+		[TCPHDR_RESERVED]	= HDR_BITFIELD("reserved", &integer_type,
+						       (12 * BITS_PER_BYTE) + 0, 4),
 		[TCPHDR_FLAGS]		= HDR_BITFIELD("flags", &tcp_flag_type,
 						       13 * BITS_PER_BYTE,
 						       BITS_PER_BYTE),
@@ -501,8 +503,8 @@ const struct proto_desc proto_ip = {
 		PROTO_LINK(IPPROTO_SCTP,	&proto_sctp),
 	},
 	.templates	= {
-		[IPHDR_VERSION]		= HDR_BITFIELD("version", &integer_type, 0, 4),
-		[IPHDR_HDRLENGTH]	= HDR_BITFIELD("hdrlength", &integer_type, 4, 4),
+		[IPHDR_VERSION]		= HDR_BITFIELD("version", &integer_type, 4, 4),
+		[IPHDR_HDRLENGTH]	= HDR_BITFIELD("hdrlength", &integer_type, 0, 4),
 		[IPHDR_TOS]		= IPHDR_FIELD("tos",		tos),
 		[IPHDR_LENGTH]		= IPHDR_FIELD("length",		tot_len),
 		[IPHDR_ID]		= IPHDR_FIELD("id",		id),
-- 
2.0.5

[PATCH 08/12] netlink: cmp: shift rhs constant if lhs offset doesn't start on byte boundary

[Florian Westphal]

if we have payload(someoffset) == 42, then shift 42 in case someoffset
doesn't start on a byte boundary.

We already insert a mask instruction to only load those bits into
the register that we were interested in, but the cmp will fail without
also adjusting rhs accordingly.

Needs additional patch in reverse direction to undo the shift again
when dumping ruleset.

Signed-off-by: Florian Westphal <fw@strlen.de>
---
 src/netlink_linearize.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/src/netlink_linearize.c b/src/netlink_linearize.c
index 0467872..044bac8 100644
--- a/src/netlink_linearize.c
+++ b/src/netlink_linearize.c
@@ -278,6 +278,15 @@ static void netlink_gen_range(struct netlink_linearize_ctx *ctx,
 			      const struct expr *expr,
 			      enum nft_registers dreg);
 
+static void payload_shift_value(const struct expr *left, struct expr *right)
+{
+	if (right->ops->type != EXPR_VALUE ||
+	    left->ops->type != EXPR_PAYLOAD)
+		return;
+
+	mpz_lshift_ui(right->value, left->payload.offset % BITS_PER_BYTE);
+}
+
 static void netlink_gen_cmp(struct netlink_linearize_ctx *ctx,
 			    const struct expr *expr,
 			    enum nft_registers dreg)
@@ -326,6 +335,7 @@ static void netlink_gen_cmp(struct netlink_linearize_ctx *ctx,
 	netlink_put_register(nle, NFT_EXPR_CMP_SREG, sreg);
 	nft_rule_expr_set_u32(nle, NFT_EXPR_CMP_OP,
 			      netlink_gen_cmp_op(expr->op));
+	payload_shift_value(expr->left, right);
 	netlink_gen_data(right, &nld);
 	nft_rule_expr_set(nle, NFT_EXPR_CMP_DATA, nld.value, nld.len);
 	release_register(ctx, expr->left);
-- 
2.0.5

[PATCH 09/12] tests: add tests for ip version/hdrlength/tcp doff

[Florian Westphal]

Header fields of 4 bit lengths.  Requires implicit masks and
shifting of RHS constant.

Signed-off-by: Florian Westphal <fw@strlen.de>
---
 tests/regression/inet/tcp.t              |  2 ++
 tests/regression/inet/tcp.t.payload.inet |  8 ++++++++
 tests/regression/inet/tcp.t.payload.ip   |  8 ++++++++
 tests/regression/inet/tcp.t.payload.ip6  |  8 ++++++++
 tests/regression/ip/ip.t                 |  4 ++++
 tests/regression/ip/ip.t.payload         | 21 +++++++++++++++++++++
 tests/regression/ip/ip.t.payload.inet    | 27 +++++++++++++++++++++++++++
 7 files changed, 78 insertions(+)

diff --git a/tests/regression/inet/tcp.t b/tests/regression/inet/tcp.t
index 95267da..53a1689 100644
--- a/tests/regression/inet/tcp.t
+++ b/tests/regression/inet/tcp.t
@@ -101,3 +101,5 @@ tcp urgptr { 33, 55, 67, 88};ok
 - tcp urgptr != { 33, 55, 67, 88};ok
 tcp urgptr { 33-55};ok
 - tcp urgptr != { 33-55};ok
+
+tcp doff 8;ok
diff --git a/tests/regression/inet/tcp.t.payload.inet b/tests/regression/inet/tcp.t.payload.inet
index e1673db..21b21ab 100644
--- a/tests/regression/inet/tcp.t.payload.inet
+++ b/tests/regression/inet/tcp.t.payload.inet
@@ -498,3 +498,11 @@ inet test-inet input
   [ payload load 2b @ transport header + 18 => reg 1 ]
   [ lookup reg 1 set set%d ]
 
+# tcp doff 8
+inet test-inet input
+  [ meta load l4proto => reg 1 ]
+  [ cmp eq reg 1 0x00000006 ]
+  [ payload load 1b @ transport header + 12 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x000000f0 ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000080 ]
+
diff --git a/tests/regression/inet/tcp.t.payload.ip b/tests/regression/inet/tcp.t.payload.ip
index c21cca1..34c9714 100644
--- a/tests/regression/inet/tcp.t.payload.ip
+++ b/tests/regression/inet/tcp.t.payload.ip
@@ -498,3 +498,11 @@ ip test-ip4 input
   [ payload load 2b @ transport header + 18 => reg 1 ]
   [ lookup reg 1 set set%d ]
 
+# tcp doff 8
+ip test-ip4 input
+  [ payload load 1b @ network header + 9 => reg 1 ]
+  [ cmp eq reg 1 0x00000006 ]
+  [ payload load 1b @ transport header + 12 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x000000f0 ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000080 ]
+
diff --git a/tests/regression/inet/tcp.t.payload.ip6 b/tests/regression/inet/tcp.t.payload.ip6
index 32115ed..44decab 100644
--- a/tests/regression/inet/tcp.t.payload.ip6
+++ b/tests/regression/inet/tcp.t.payload.ip6
@@ -498,3 +498,11 @@ ip6 test-ip6 input
   [ payload load 2b @ transport header + 18 => reg 1 ]
   [ lookup reg 1 set set%d ]
 
+# tcp doff 8
+ip6 test-ip6 input
+  [ payload load 1b @ network header + 6 => reg 1 ]
+  [ cmp eq reg 1 0x00000006 ]
+  [ payload load 1b @ transport header + 12 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x000000f0 ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000080 ]
+
diff --git a/tests/regression/ip/ip.t b/tests/regression/ip/ip.t
index 6e29ffb..0339c2a 100644
--- a/tests/regression/ip/ip.t
+++ b/tests/regression/ip/ip.t
@@ -111,3 +111,7 @@ ip saddr \& 0.0.0.255 \< 0.0.0.127;ok;ip saddr & 0.0.0.255 < 0.0.0.127
 
 ip saddr \& 0xffff0000 == 0xffff0000;ok;ip saddr 255.255.0.0/16
 
+ip version 4 ip hdrlength 5;ok
+ip hdrlength 0;ok
+ip hdrlength 15;ok
+ip hdrlength 16;fail
diff --git a/tests/regression/ip/ip.t.payload b/tests/regression/ip/ip.t.payload
index 7a77dc4..31a57a2 100644
--- a/tests/regression/ip/ip.t.payload
+++ b/tests/regression/ip/ip.t.payload
@@ -353,3 +353,24 @@ ip test-ip4 input
   [ bitwise reg 1 = (reg=1 & 0x0000ffff ) ^ 0x00000000 ]
   [ cmp eq reg 1 0x0000ffff ]
 
+# ip version 4 ip hdrlength 5
+ip test-ip4 input
+  [ payload load 1b @ network header + 0 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x000000f0 ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000040 ]
+  [ payload load 1b @ network header + 0 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000000f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000005 ]
+
+# ip hdrlength 0
+ip test-ip4 input
+  [ payload load 1b @ network header + 0 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000000f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000000 ]
+
+# ip hdrlength 15
+ip test-ip4 input
+  [ payload load 1b @ network header + 0 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000000f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x0000000f ]
+
diff --git a/tests/regression/ip/ip.t.payload.inet b/tests/regression/ip/ip.t.payload.inet
index dbc7852..8c31349 100644
--- a/tests/regression/ip/ip.t.payload.inet
+++ b/tests/regression/ip/ip.t.payload.inet
@@ -465,3 +465,30 @@ inet test-inet input
   [ bitwise reg 1 = (reg=1 & 0x0000ffff ) ^ 0x00000000 ]
   [ cmp eq reg 1 0x0000ffff ]
 
+# ip version 4 ip hdrlength 5
+inet test-inet input
+  [ meta load nfproto => reg 1 ]
+  [ cmp eq reg 1 0x00000002 ]
+  [ payload load 1b @ network header + 0 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x000000f0 ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000040 ]
+  [ payload load 1b @ network header + 0 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000000f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000005 ]
+
+# ip hdrlength 0
+inet test-inet input
+  [ meta load nfproto => reg 1 ]
+  [ cmp eq reg 1 0x00000002 ]
+  [ payload load 1b @ network header + 0 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000000f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000000 ]
+
+# ip hdrlength 15
+inet test-inet input
+  [ meta load nfproto => reg 1 ]
+  [ cmp eq reg 1 0x00000002 ]
+  [ payload load 1b @ network header + 0 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000000f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x0000000f ]
+
-- 
2.0.5

[PATCH 10/12] nft: support listing expressions that use non-byte header fields

[Florian Westphal]

This allows to list rules that check fields that are not aligned on byte
boundary, and allows decoding vlan headers, which reside on top of eth header
and need the expr->payload.offset adjusted accordingly.

Signed-off-by: Florian Westphal <fw@strlen.de>
---
 include/payload.h         |  3 ++
 src/evaluate.c            |  6 ++++
 src/netlink_delinearize.c | 80 ++++++++++++++++++++++++++++++++++++++------
 src/payload.c             | 84 +++++++++++++++++++++++++++++++++++++++++++++--
 4 files changed, 160 insertions(+), 13 deletions(-)

diff --git a/include/payload.h b/include/payload.h
index 95364af..ca9b013 100644
--- a/include/payload.h
+++ b/include/payload.h
@@ -19,7 +19,10 @@ extern bool payload_is_adjacent(const struct expr *e1, const struct expr *e2);
 extern struct expr *payload_expr_join(const struct expr *e1,
 				      const struct expr *e2);
 
+bool payload_expr_trim(struct expr *expr, struct expr *mask,
+		       const struct proto_ctx *ctx);
 extern void payload_expr_expand(struct list_head *list, struct expr *expr,
+				struct expr *mask,
 				const struct proto_ctx *ctx);
 extern void payload_expr_complete(struct expr *expr,
 				  const struct proto_ctx *ctx);
diff --git a/src/evaluate.c b/src/evaluate.c
index c95d154..7ff94c0 100644
--- a/src/evaluate.c
+++ b/src/evaluate.c
@@ -332,8 +332,13 @@ static bool resolve_protocol_conflict(struct eval_ctx *ctx,
 		const struct proto_desc *next = ctx->pctx.protocol[base + 1].desc;
 
 		if (payload->payload.desc == next) {
+			ctx->pctx.protocol[base + 1].desc = NULL;
+			ctx->pctx.protocol[base].desc = next;
+			ctx->pctx.protocol[base].offset += desc->length;
 			payload->payload.offset += desc->length;
 			return true;
+		} else if (next) {
+			return false;
 		}
 	}
 
@@ -343,6 +348,7 @@ static bool resolve_protocol_conflict(struct eval_ctx *ctx,
 
 	payload->payload.offset += ctx->pctx.protocol[base].offset;
 	list_add_tail(&nstmt->list, &ctx->stmt->list);
+	ctx->pctx.protocol[base + 1].desc = NULL;
 
 	return true;
 }
diff --git a/src/netlink_delinearize.c b/src/netlink_delinearize.c
index 76c07c5..537e34c 100644
--- a/src/netlink_delinearize.c
+++ b/src/netlink_delinearize.c
@@ -918,16 +918,20 @@ static void integer_type_postprocess(struct expr *expr)
 	}
 }
 
-static void payload_match_expand(struct rule_pp_ctx *ctx, struct expr *expr)
+static void payload_match_expand(struct rule_pp_ctx *ctx,
+				 struct expr *expr,
+				 struct expr *payload,
+				 struct expr *mask)
 {
-	struct expr *left = expr->left, *right = expr->right, *tmp;
+	struct expr *left = payload, *right = expr->right, *tmp;
 	struct list_head list = LIST_HEAD_INIT(list);
 	struct stmt *nstmt;
 	struct expr *nexpr = NULL;
 	enum proto_bases base = left->payload.base;
 	const struct expr_ops *payload_ops = left->ops;
 
-	payload_expr_expand(&list, left, &ctx->pctx);
+	payload_expr_expand(&list, left, mask, &ctx->pctx);
+
 	list_for_each_entry(left, &list, list) {
 		tmp = constant_expr_splice(right, left->len);
 		expr_set_type(tmp, left->dtype, left->byteorder);
@@ -981,10 +985,11 @@ static void payload_match_expand(struct rule_pp_ctx *ctx, struct expr *expr)
 }
 
 static void payload_match_postprocess(struct rule_pp_ctx *ctx,
-				      struct expr *expr)
+				      struct expr *expr,
+				      struct expr *payload,
+				      struct expr *mask)
 {
-	enum proto_bases base = expr->left->payload.base;
-	struct expr *payload = expr->left;
+	enum proto_bases base = payload->payload.base;
 
 	assert(payload->payload.offset >= ctx->pctx.protocol[base].offset);
 	payload->payload.offset -= ctx->pctx.protocol[base].offset;
@@ -993,7 +998,7 @@ static void payload_match_postprocess(struct rule_pp_ctx *ctx,
 	case OP_EQ:
 	case OP_NEQ:
 		if (expr->right->ops->type == EXPR_VALUE) {
-			payload_match_expand(ctx, expr);
+			payload_match_expand(ctx, expr, payload, mask);
 			break;
 		}
 		/* Fall through */
@@ -1074,7 +1079,7 @@ static struct expr *binop_tree_to_list(struct expr *list, struct expr *expr)
 	return list;
 }
 
-static void relational_binop_postprocess(struct expr *expr)
+static void relational_binop_postprocess(struct rule_pp_ctx *ctx, struct expr *expr)
 {
 	struct expr *binop = expr->left, *value = expr->right;
 
@@ -1102,6 +1107,61 @@ static void relational_binop_postprocess(struct expr *expr)
 						expr_mask_to_prefix(binop->right));
 		expr_free(value);
 		expr_free(binop);
+	} else if (binop->op == OP_AND &&
+		   binop->left->ops->type == EXPR_PAYLOAD &&
+		   binop->right->ops->type == EXPR_VALUE) {
+		struct expr *payload = expr->left->left;
+		struct expr *mask = expr->left->right;
+
+		/*
+		 * This *might* be a payload match testing header fields that
+		 * have non byte divisible offsets and/or bit lengths.
+		 *
+		 * Thus we need to deal with two different cases.
+		 *
+		 * 1 the simple version:
+		 *        relation
+		 * payload        value|setlookup
+		 *
+		 * expr: relation, left: payload, right: value, e.g.  tcp dport == 22.
+		 *
+		 * 2. The '&' version (this is what we're looking at now).
+		 *            relation
+		 *     binop          value1|setlookup
+		 * payload  value2
+		 *
+		 * expr: relation, left: binop, right: value, e.g.
+		 * ip saddr 10.0.0.0/8
+		 *
+		 * payload_expr_trim will figure out if the mask is needed to match
+		 * templates.
+		 */
+		if (payload_expr_trim(payload, mask, &ctx->pctx)) {
+			/* mask is implicit, binop needs to be removed.
+			 *
+			 * Fix all values of the expression according to the mask
+			 * and then process the payload instruction using the real
+			 * sizes and offsets we're interested in.
+			 *
+			 * Finally, convert the expression to 1) by replacing
+			 * the binop with the binop payload expr.
+			 */
+			if (value->ops->type == EXPR_VALUE) {
+				assert(value->len >= expr->left->right->len);
+				value->len = mask->len;
+			}
+
+			payload->len = mask->len;
+			payload->payload.offset += mpz_scan1(mask->value, 0);
+
+			payload_match_postprocess(ctx, expr, payload, mask);
+
+			assert(expr->left->ops->type == EXPR_BINOP);
+
+			assert(binop->left == payload);
+			expr->left = payload;
+			expr_free(binop);
+		}
 	}
 }
 
@@ -1160,7 +1220,7 @@ static void expr_postprocess(struct rule_pp_ctx *ctx, struct expr **exprp)
 	case EXPR_RELATIONAL:
 		switch (expr->left->ops->type) {
 		case EXPR_PAYLOAD:
-			payload_match_postprocess(ctx, expr);
+			payload_match_postprocess(ctx, expr, expr->left, NULL);
 			return;
 		default:
 			expr_postprocess(ctx, &expr->left);
@@ -1175,7 +1235,7 @@ static void expr_postprocess(struct rule_pp_ctx *ctx, struct expr **exprp)
 			meta_match_postprocess(ctx, expr);
 			break;
 		case EXPR_BINOP:
-			relational_binop_postprocess(expr);
+			relational_binop_postprocess(ctx, expr);
 			break;
 		default:
 			break;
diff --git a/src/payload.c b/src/payload.c
index 372dcec..2ede199 100644
--- a/src/payload.c
+++ b/src/payload.c
@@ -296,12 +296,87 @@ void payload_expr_complete(struct expr *expr, const struct proto_ctx *ctx)
 	}
 }
 
+static unsigned int mask_to_offset(const struct expr *mask)
+{
+	return mask ? mpz_scan1(mask->value, 0) : 0;
+}
+
+static unsigned int mask_length(const struct expr *mask)
+{
+	unsigned long off;
+
+        off = mask_to_offset(mask);
+
+	return mpz_scan0(mask->value, off + 1);
+}
+
+/**
+ * payload_expr_trim - trim payload expression according to mask
+ *
+ * @expr:	the payload expression
+ * @mask:	mask to use when searching templates
+ * @ctx:	protocol context
+ *
+ * Walk the template list and determine if a match can be found without
+ * using the provided mask.
+ *
+ * If the mask has to be used, trim the mask length accordingly
+ * and return true to let the caller know that the mask is a dependency.
+ */
+bool payload_expr_trim(struct expr *expr, struct expr *mask,
+		       const struct proto_ctx *ctx)
+{
+	unsigned int payload_offset = expr->payload.offset + mask_to_offset(mask);
+	unsigned int mask_len = mask_length(mask);
+	const struct proto_hdr_template *tmpl;
+	unsigned int payload_len = expr->len;
+	const struct proto_desc *desc;
+	unsigned int i, matched_len = mask_to_offset(mask);
+
+	assert(expr->ops->type == EXPR_PAYLOAD);
+
+	desc = ctx->protocol[expr->payload.base].desc;
+	if (desc == NULL)
+		return false;
+
+	assert(desc->base == expr->payload.base);
+
+	if (ctx->protocol[expr->payload.base].offset) {
+		assert(payload_offset >= ctx->protocol[expr->payload.base].offset);
+		payload_offset -= ctx->protocol[expr->payload.base].offset;
+	}
+
+	for (i = 1; i < array_size(desc->templates); i++) {
+		tmpl = &desc->templates[i];
+		if (tmpl->offset != payload_offset)
+			continue;
+
+		if (tmpl->len > payload_len)
+			return false;
+
+		payload_len -= tmpl->len;
+		matched_len += tmpl->len;
+		payload_offset += tmpl->len;
+		if (payload_len == 0)
+			return false;
+
+		if (matched_len == mask_len) {
+			assert(mask->len >= mask_len);
+			mask->len = mask_len;
+			return true;
+		}
+	}
+
+	return false;
+}
+
 /**
  * payload_expr_expand - expand raw merged adjacent payload expressions into its
  * 			 original components
  *
  * @list:	list to append expanded payload expressions to
  * @expr:	the payload expression to expand
+ * @mask:	optional/implicit mask to use when searching templates
  * @ctx:	protocol context
  *
  * Expand a merged adjacent payload expression into its original components
@@ -311,18 +386,21 @@ void payload_expr_complete(struct expr *expr, const struct proto_ctx *ctx)
  * 	 offset order.
  */
 void payload_expr_expand(struct list_head *list, struct expr *expr,
-			 const struct proto_ctx *ctx)
+			 struct expr *mask, const struct proto_ctx *ctx)
 {
-	const struct proto_desc *desc;
+	unsigned int off = mask_to_offset(mask);
 	const struct proto_hdr_template *tmpl;
+	const struct proto_desc *desc;
 	struct expr *new;
 	unsigned int i;
 
 	assert(expr->ops->type == EXPR_PAYLOAD);
+	assert(!mask || mask->len != 0);
 
 	desc = ctx->protocol[expr->payload.base].desc;
 	if (desc == NULL)
 		goto raw;
+
 	assert(desc->base == expr->payload.base);
 
 	for (i = 1; i < array_size(desc->templates); i++) {
@@ -335,7 +413,7 @@ void payload_expr_expand(struct list_head *list, struct expr *expr,
 			list_add_tail(&new->list, list);
 			expr->len	     -= tmpl->len;
 			expr->payload.offset += tmpl->len;
-			if (expr->len == 0)
+			if (expr->len == off)
 				return;
 		} else
 			break;
-- 
2.0.5

[PATCH 11/12] tests: vlan tests

[Florian Westphal]

add a few regression tests that match vlan id/dei/pcp fields
of the vlan header.

Signed-off-by: Florian Westphal <fw@strlen.de>
---
 tests/regression/bridge/vlan.t         |  33 ++++++
 tests/regression/bridge/vlan.t.payload | 190 +++++++++++++++++++++++++++++++++
 2 files changed, 223 insertions(+)
 create mode 100644 tests/regression/bridge/vlan.t
 create mode 100644 tests/regression/bridge/vlan.t.payload

diff --git a/tests/regression/bridge/vlan.t b/tests/regression/bridge/vlan.t
new file mode 100644
index 0000000..e80ea50
--- /dev/null
+++ b/tests/regression/bridge/vlan.t
@@ -0,0 +1,33 @@
+*bridge;test-bridge
+:input;type filter hook input priority 0
+
+vlan id 4094;ok
+vlan id 0;ok
+# bad vlan id
+vlan id 4096;fail
+vlan id 4094 vlan cfi 0;ok
+vlan id 4094 vlan cfi 1;ok
+# bad cfi
+vlan id 4094 vlan cfi 2;fail
+vlan id 4094 vlan cfi 1 vlan pcp 8;fail
+vlan id 4094 vlan cfi 1 vlan pcp 7;ok
+vlan id 4094 vlan cfi 1 vlan pcp 3;ok
+
+ether type vlan vlan id 4094;ok;vlan id 4094
+ether type vlan vlan id 0;ok;vlan id 0
+ether type vlan vlan id 4094 vlan cfi 0;ok;vlan id 4094 vlan cfi 0
+ether type vlan vlan id 4094 vlan cfi 1;ok;vlan id 4094 vlan cfi 1
+ether type vlan vlan id 4094 vlan cfi 2;fail
+
+vlan id 4094 tcp dport 22;ok
+vlan id 1 ip saddr 10.0.0.1;ok
+vlan id 1 ip saddr 10.0.0.0/23;ok
+vlan id 1 ip saddr 10.0.0.0/23 udp dport 53;ok
+ether type vlan vlan id 1 ip saddr 10.0.0.0/23 udp dport 53;ok;vlan id 1 ip saddr 10.0.0.0/23 udp dport 53
+
+vlan id { 1, 2, 4, 100, 4095 } vlan pcp 1-3;ok
+vlan id { 1, 2, 4, 100, 4096 };fail
+
+# illegal dependencies
+ether type ip vlan id 1;fail
+ether type ip vlan id 1 ip saddr 10.0.0.1;fail
diff --git a/tests/regression/bridge/vlan.t.payload b/tests/regression/bridge/vlan.t.payload
new file mode 100644
index 0000000..4a5714a
--- /dev/null
+++ b/tests/regression/bridge/vlan.t.payload
@@ -0,0 +1,190 @@
+# vlan id 4094
+bridge test-bridge input
+  [ payload load 2b @ link header + 12 => reg 1 ]
+  [ cmp eq reg 1 0x00000081 ]
+  [ payload load 2b @ link header + 14 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000ff0f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x0000fe0f ]
+
+# vlan id 0
+bridge test-bridge input
+  [ payload load 2b @ link header + 12 => reg 1 ]
+  [ cmp eq reg 1 0x00000081 ]
+  [ payload load 2b @ link header + 14 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000ff0f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000000 ]
+
+# vlan id 4094 vlan cfi 0
+bridge test-bridge input
+  [ payload load 2b @ link header + 12 => reg 1 ]
+  [ cmp eq reg 1 0x00000081 ]
+  [ payload load 2b @ link header + 14 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000ff0f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x0000fe0f ]
+  [ payload load 1b @ link header + 15 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x00000010 ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000000 ]
+
+# vlan id 4094 vlan cfi 1
+bridge test-bridge input
+  [ payload load 2b @ link header + 12 => reg 1 ]
+  [ cmp eq reg 1 0x00000081 ]
+  [ payload load 2b @ link header + 14 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000ff0f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x0000fe0f ]
+  [ payload load 1b @ link header + 15 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x00000010 ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000010 ]
+
+# ether type vlan vlan id 4094
+bridge test-bridge input
+  [ payload load 2b @ link header + 12 => reg 1 ]
+  [ cmp eq reg 1 0x00000081 ]
+  [ payload load 2b @ link header + 14 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000ff0f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x0000fe0f ]
+
+# ether type vlan vlan id 0
+bridge test-bridge input
+  [ payload load 2b @ link header + 12 => reg 1 ]
+  [ cmp eq reg 1 0x00000081 ]
+  [ payload load 2b @ link header + 14 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000ff0f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000000 ]
+
+# ether type vlan vlan id 4094 vlan cfi 0
+bridge test-bridge input
+  [ payload load 2b @ link header + 12 => reg 1 ]
+  [ cmp eq reg 1 0x00000081 ]
+  [ payload load 2b @ link header + 14 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000ff0f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x0000fe0f ]
+  [ payload load 1b @ link header + 15 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x00000010 ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000000 ]
+
+# ether type vlan vlan id 4094 vlan cfi 1
+bridge test-bridge input
+  [ payload load 2b @ link header + 12 => reg 1 ]
+  [ cmp eq reg 1 0x00000081 ]
+  [ payload load 2b @ link header + 14 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000ff0f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x0000fe0f ]
+  [ payload load 1b @ link header + 15 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x00000010 ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000010 ]
+
+# vlan id 4094 tcp dport 22
+bridge test-bridge input
+  [ payload load 2b @ link header + 12 => reg 1 ]
+  [ cmp eq reg 1 0x00000081 ]
+  [ payload load 2b @ link header + 14 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000ff0f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x0000fe0f ]
+  [ meta load l4proto => reg 1 ]
+  [ cmp eq reg 1 0x00000006 ]
+  [ payload load 2b @ transport header + 2 => reg 1 ]
+  [ cmp eq reg 1 0x00001600 ]
+
+# vlan id 1 ip saddr 10.0.0.1
+bridge test-bridge input
+  [ payload load 2b @ link header + 12 => reg 1 ]
+  [ cmp eq reg 1 0x00000081 ]
+  [ payload load 2b @ link header + 14 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000ff0f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000100 ]
+  [ payload load 2b @ link header + 16 => reg 1 ]
+  [ cmp eq reg 1 0x00000008 ]
+  [ payload load 4b @ network header + 12 => reg 1 ]
+  [ cmp eq reg 1 0x0100000a ]
+
+# vlan id 1 ip saddr 10.0.0.0/23
+bridge test-bridge input
+  [ payload load 2b @ link header + 12 => reg 1 ]
+  [ cmp eq reg 1 0x00000081 ]
+  [ payload load 2b @ link header + 14 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000ff0f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000100 ]
+  [ payload load 2b @ link header + 16 => reg 1 ]
+  [ cmp eq reg 1 0x00000008 ]
+  [ payload load 4b @ network header + 12 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x00feffff ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x0000000a ]
+
+# vlan id 1 ip saddr 10.0.0.0/23 udp dport 53
+bridge test-bridge input
+  [ payload load 2b @ link header + 12 => reg 1 ]
+  [ cmp eq reg 1 0x00000081 ]
+  [ payload load 2b @ link header + 14 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000ff0f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000100 ]
+  [ payload load 2b @ link header + 16 => reg 1 ]
+  [ cmp eq reg 1 0x00000008 ]
+  [ payload load 4b @ network header + 12 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x00feffff ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x0000000a ]
+  [ payload load 1b @ network header + 9 => reg 1 ]
+  [ cmp eq reg 1 0x00000011 ]
+  [ payload load 2b @ transport header + 2 => reg 1 ]
+  [ cmp eq reg 1 0x00003500 ]
+
+# ether type vlan vlan id 1 ip saddr 10.0.0.0/23 udp dport 53
+bridge test-bridge input
+  [ payload load 2b @ link header + 12 => reg 1 ]
+  [ cmp eq reg 1 0x00000081 ]
+  [ payload load 2b @ link header + 14 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000ff0f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000100 ]
+  [ payload load 2b @ link header + 16 => reg 1 ]
+  [ cmp eq reg 1 0x00000008 ]
+  [ payload load 4b @ network header + 12 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x00feffff ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x0000000a ]
+  [ payload load 1b @ network header + 9 => reg 1 ]
+  [ cmp eq reg 1 0x00000011 ]
+  [ payload load 2b @ transport header + 2 => reg 1 ]
+  [ cmp eq reg 1 0x00003500 ]
+
+# vlan id 4094 vlan cfi 1 vlan pcp 7
+bridge test-bridge input
+  [ payload load 2b @ link header + 12 => reg 1 ]
+  [ cmp eq reg 1 0x00000081 ]
+  [ payload load 2b @ link header + 14 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000ff0f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x0000fe0f ]
+  [ payload load 1b @ link header + 15 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x00000010 ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000010 ]
+  [ payload load 1b @ link header + 15 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x000000e0 ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x000000e0 ]
+
+# vlan id 4094 vlan cfi 1 vlan pcp 3
+bridge test-bridge input
+  [ payload load 2b @ link header + 12 => reg 1 ]
+  [ cmp eq reg 1 0x00000081 ]
+  [ payload load 2b @ link header + 14 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000ff0f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x0000fe0f ]
+  [ payload load 1b @ link header + 15 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x00000010 ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000010 ]
+  [ payload load 1b @ link header + 15 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x000000e0 ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x00000060 ]
+
+# vlan id { 1, 2, 4, 100, 4095 } vlan pcp 1-3
+set%d test-bridge 3
+set%d test-bridge 0
+	element 00000100  : 0 [end]	element 00000200  : 0 [end]	element 00000400  : 0 [end]	element 00006400  : 0 [end]	element 0000ff0f  : 0 [end]
+bridge test-bridge input
+  [ payload load 2b @ link header + 12 => reg 1 ]
+  [ cmp eq reg 1 0x00000081 ]
+  [ payload load 2b @ link header + 14 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000ff0f ) ^ 0x00000000 ]
+  [ lookup reg 1 set set%d ]
+  [ payload load 1b @ link header + 15 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x000000e0 ) ^ 0x00000000 ]
+  [ cmp gte reg 1 0x00000001 ]
+  [ cmp lte reg 1 0x00000003 ]
+
-- 
2.0.5

[PATCH 12/13] vlan: make != tests work

[Florian Westphal]

---
 src/netlink_delinearize.c              |  1 +
 tests/regression/bridge/vlan.t         |  1 +
 tests/regression/bridge/vlan.t.payload | 11 +++++++++++
 3 files changed, 13 insertions(+)

diff --git a/src/netlink_delinearize.c b/src/netlink_delinearize.c
index 537e34c..2e3b090 100644
--- a/src/netlink_delinearize.c
+++ b/src/netlink_delinearize.c
@@ -1084,6 +1084,7 @@ static void relational_binop_postprocess(struct rule_pp_ctx *ctx, struct expr *e
 	struct expr *binop = expr->left, *value = expr->right;
 
 	if (binop->op == OP_AND && expr->op == OP_NEQ &&
+	    value->dtype->basetype &&
 	    value->dtype->basetype->type == TYPE_BITMASK &&
 	    !mpz_cmp_ui(value->value, 0)) {
 		/* Flag comparison: data & flags != 0
diff --git a/tests/regression/bridge/vlan.t b/tests/regression/bridge/vlan.t
index e80ea50..f86561a 100644
--- a/tests/regression/bridge/vlan.t
+++ b/tests/regression/bridge/vlan.t
@@ -6,6 +6,7 @@ vlan id 0;ok
 # bad vlan id
 vlan id 4096;fail
 vlan id 4094 vlan cfi 0;ok
+vlan id 4094 vlan cfi != 1;ok
 vlan id 4094 vlan cfi 1;ok
 # bad cfi
 vlan id 4094 vlan cfi 2;fail
diff --git a/tests/regression/bridge/vlan.t.payload b/tests/regression/bridge/vlan.t.payload
index 4a5714a..35dc437 100644
--- a/tests/regression/bridge/vlan.t.payload
+++ b/tests/regression/bridge/vlan.t.payload
@@ -25,6 +25,17 @@ bridge test-bridge input
   [ bitwise reg 1 = (reg=1 & 0x00000010 ) ^ 0x00000000 ]
   [ cmp eq reg 1 0x00000000 ]
 
+# vlan id 4094 vlan cfi != 1
+bridge test-bridge input
+  [ payload load 2b @ link header + 12 => reg 1 ]
+  [ cmp eq reg 1 0x00000081 ]
+  [ payload load 2b @ link header + 14 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x0000ff0f ) ^ 0x00000000 ]
+  [ cmp eq reg 1 0x0000fe0f ]
+  [ payload load 1b @ link header + 15 => reg 1 ]
+  [ bitwise reg 1 = (reg=1 & 0x00000010 ) ^ 0x00000000 ]
+  [ cmp neq reg 1 0x00000010 ]
+
 # vlan id 4094 vlan cfi 1
 bridge test-bridge input
   [ payload load 2b @ link header + 12 => reg 1 ]
-- 
2.0.5

Re: [PATCH nft 0/12] add support for VLAN header filtering in bridge family

[Florian Westphal]

Florian Westphal <fw@strlen.de> wrote:
> Unless there are objections I will push the first two patches soon.

Done.

Re: [PATCH nft 0/12] add support for VLAN header filtering in bridge family

[Florian Westphal]

Florian Westphal <fw@strlen.de> wrote:
> This adds the following features:
> - stacked headers: vlan sits on top of ethernet, so when vlan id 42 is requested,
>   add a dependency on eth and also adjust payload offsets by the size of ethernet
>   header.
> 
> - allow matching header fields of lengths and/or offsets that are not byte divisible,
>   e.g. vlan id, vlan pcp, ip hdrlength, etc.

pushed it to master.