* [PATCH net-next 0/2] nfp: flower: handle MTU changes
From: Jakub Kicinski @ 2018-03-29 1:50 UTC (permalink / raw)
To: davem; +Cc: netdev, oss-drivers, Jakub Kicinski
Hi!
This set improves MTU handling for flower offload. The max MTU is
correctly capped and physical port MTU is communicated to the FW
(and indirectly HW).
John Hurley (2):
nfp: app: modify app MTU setting callbacks
nfp: flower: offload phys port MTU change
drivers/net/ethernet/netronome/nfp/bpf/main.c | 4 +-
drivers/net/ethernet/netronome/nfp/flower/cmsg.c | 41 +++++++++-
drivers/net/ethernet/netronome/nfp/flower/cmsg.h | 4 +-
drivers/net/ethernet/netronome/nfp/flower/main.c | 87 +++++++++++++++++++++-
drivers/net/ethernet/netronome/nfp/flower/main.h | 19 +++++
drivers/net/ethernet/netronome/nfp/nfp_app.h | 25 +++++--
.../net/ethernet/netronome/nfp/nfp_net_common.c | 2 +-
drivers/net/ethernet/netronome/nfp/nfp_net_repr.c | 13 +++-
8 files changed, 179 insertions(+), 16 deletions(-)
--
2.16.2
^ permalink raw reply
* Re: [PATCH 1/2] af_key: Use DIV_ROUND_UP() instead of open-coded equivalent
From: Kevin Easton @ 2018-03-29 1:35 UTC (permalink / raw)
To: Steffen Klassert; +Cc: Herbert Xu, David S. Miller, netdev, linux-kernel
In-Reply-To: <20180328055925.skrjbax24he3vcpk@gauss3.secunet.de>
On Wed, Mar 28, 2018 at 07:59:25AM +0200, Steffen Klassert wrote:
> On Mon, Mar 26, 2018 at 07:39:16AM -0400, Kevin Easton wrote:
> > Several places use (x + 7) / 8 to convert from a number of bits to a number
> > of bytes. Replace those with DIV_ROUND_UP(x, 8) instead, for consistency
> > with other parts of the same file.
> >
> > Signed-off-by: Kevin Easton <kevin@guarana.org>
>
> Is this a fix or just a cleanup?
> If it is just a cleanup, please resent it based on ipsec-next.
Hi Steffen,
This patch (1/2) is just a cleanup, but it's in preparation for patch 2/2
which is a fix and modifies some of the same lines.
- Kevin
^ permalink raw reply
* [PATCH 2/2] rhashtable: improve rhashtable_walk stability when stop/start used.
From: NeilBrown @ 2018-03-29 1:19 UTC (permalink / raw)
To: Thomas Graf, Herbert Xu; +Cc: netdev, linux-kernel
In-Reply-To: <152228607974.16370.14544827502467836789.stgit@noble>
When a walk of an rhashtable is interrupted with rhastable_walk_stop()
and then rhashtable_walk_start(), the location to restart from is based
on a 'skip' count in the current hash chain, and this can be incorrect
if insertions or deletions have happened. This does not happen when
the walk is not stopped and started as iter->p is a placeholder which
is safe to use while holding the RCU read lock.
In rhashtable_walk_start() we can revalidate that 'p' is still in the
same hash chain. If it isn't then the current method is still used.
With this patch, if a rhashtable walker ensures that the current
object remains in the table over a stop/start period (possibly by
elevating the reference count if that is sufficient), it can be sure
that a walk will not miss objects that were in the hashtable for the
whole time of the walk.
rhashtable_walk_start() may not find the object even though it is
still in the hashtable if a rehash has moved it to a new table. In
this case it will (eventually) get -EAGAIN and will need to proceed
through the whole table again to be sure to see everything at least
once.
Signed-off-by: NeilBrown <neilb@suse.com>
---
lib/rhashtable.c | 44 +++++++++++++++++++++++++++++++++++++++++---
1 file changed, 41 insertions(+), 3 deletions(-)
diff --git a/lib/rhashtable.c b/lib/rhashtable.c
index 72e56f89e20c..46cf0cc6cefc 100644
--- a/lib/rhashtable.c
+++ b/lib/rhashtable.c
@@ -725,6 +725,7 @@ int rhashtable_walk_start_check(struct rhashtable_iter *iter)
__acquires(RCU)
{
struct rhashtable *ht = iter->ht;
+ bool rhlist = ht->rhlist;
rcu_read_lock();
@@ -733,13 +734,52 @@ int rhashtable_walk_start_check(struct rhashtable_iter *iter)
list_del(&iter->walker.list);
spin_unlock(&ht->lock);
- if (!iter->walker.tbl && !iter->end_of_table) {
+ if (iter->end_of_table)
+ return 0;
+ if (!iter->walker.tbl) {
iter->walker.tbl = rht_dereference_rcu(ht->tbl, ht);
iter->slot = 0;
iter->skip = 0;
return -EAGAIN;
}
+ if (iter->p && !rhlist) {
+ /*
+ * We need to validate that 'p' is still in the table, and
+ * if so, update 'skip'
+ */
+ struct rhash_head *p;
+ int skip = 0;
+ rht_for_each_rcu(p, iter->walker.tbl, iter->slot) {
+ skip++;
+ if (p == iter->p) {
+ iter->skip = skip;
+ goto found;
+ }
+ }
+ iter->p = NULL;
+ } else if (iter->p && rhlist) {
+ /* Need to validate that 'list' is still in the table, and
+ * if so, update 'skip' and 'p'.
+ */
+ struct rhash_head *p;
+ struct rhlist_head *list;
+ int skip = 0;
+ rht_for_each_rcu(p, iter->walker.tbl, iter->slot) {
+ for (list = container_of(p, struct rhlist_head, rhead);
+ list;
+ list = rcu_dereference(list->next)) {
+ skip++;
+ if (list == iter->list) {
+ iter->p = p;
+ skip = skip;
+ goto found;
+ }
+ }
+ }
+ iter->p = NULL;
+ }
+found:
return 0;
}
EXPORT_SYMBOL_GPL(rhashtable_walk_start_check);
@@ -915,8 +955,6 @@ void rhashtable_walk_stop(struct rhashtable_iter *iter)
iter->walker.tbl = NULL;
spin_unlock(&ht->lock);
- iter->p = NULL;
-
out:
rcu_read_unlock();
}
^ permalink raw reply related
* [PATCH 1/2] rhashtable: fix insertion of in rhltable when duplicate found.
From: NeilBrown @ 2018-03-29 1:19 UTC (permalink / raw)
To: Thomas Graf, Herbert Xu; +Cc: netdev, linux-kernel
In-Reply-To: <152228607974.16370.14544827502467836789.stgit@noble>
When rhltable_insert() finds an entry with the same key,
it splices the new entry at the start of a list of entries with the
same key.
It stores the address of the new object in *pprev, but in general this
is *not* the location where the match was found (though in a common
case where the hash chain has one element, it will be).
To fix this, pprev should be updated every time we find an object that
doesn't match.
This patch changes the behaviour for non-slow insertion in that now
insertion happens at the end of a chain rather than at the head.
I don't think this is an important change.
Signed-off-by: NeilBrown <neilb@suse.com>
---
include/linux/rhashtable.h | 4 +++-
lib/rhashtable.c | 4 +++-
2 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/include/linux/rhashtable.h b/include/linux/rhashtable.h
index c9df2527e0cd..668a21f04b09 100644
--- a/include/linux/rhashtable.h
+++ b/include/linux/rhashtable.h
@@ -766,8 +766,10 @@ static inline void *__rhashtable_insert_fast(
if (!key ||
(params.obj_cmpfn ?
params.obj_cmpfn(&arg, rht_obj(ht, head)) :
- rhashtable_compare(&arg, rht_obj(ht, head))))
+ rhashtable_compare(&arg, rht_obj(ht, head)))) {
+ pprev = &head->next;
continue;
+ }
data = rht_obj(ht, head);
diff --git a/lib/rhashtable.c b/lib/rhashtable.c
index 394de09a162c..72e56f89e20c 100644
--- a/lib/rhashtable.c
+++ b/lib/rhashtable.c
@@ -506,8 +506,10 @@ static void *rhashtable_lookup_one(struct rhashtable *ht,
if (!key ||
(ht->p.obj_cmpfn ?
ht->p.obj_cmpfn(&arg, rht_obj(ht, head)) :
- rhashtable_compare(&arg, rht_obj(ht, head))))
+ rhashtable_compare(&arg, rht_obj(ht, head)))) {
+ pprev = &head->next;
continue;
+ }
if (!ht->rhlist)
return rht_obj(ht, head);
^ permalink raw reply related
* [PATCH 0/2] rhashtable_walk fixes
From: NeilBrown @ 2018-03-29 1:19 UTC (permalink / raw)
To: Thomas Graf, Herbert Xu; +Cc: netdev, linux-kernel
These two patches apply on top of my previous "rhashtable: reset iter
when rhashtable_walk_start sees new table" patch.
The first fixes a bug that I found in rhltable_insert().
The second is an alternate to my "rhashtable: allow a walk of the hash
table without missing object."
This version doesn't require an API change and should be reliable for
rhltables too (my first version didn't handle these correctly).
Thanks,
NeilBrown
---
NeilBrown (2):
rhashtable: fix insertion of in rhltable when duplicate found.
rhashtable: improve rhashtable_walk stability when stop/start used.
include/linux/rhashtable.h | 4 +++-
lib/rhashtable.c | 48 ++++++++++++++++++++++++++++++++++++++++----
2 files changed, 47 insertions(+), 5 deletions(-)
--
Signature
^ permalink raw reply
* Re: [PATCH 4/6] rhashtable: allow a walk of the hash table without missing objects.
From: NeilBrown @ 2018-03-29 1:13 UTC (permalink / raw)
To: Herbert Xu; +Cc: Thomas Graf, netdev, linux-kernel
In-Reply-To: <87zi2r3mve.fsf@notabene.neil.brown.name>
[-- Attachment #1: Type: text/plain, Size: 732 bytes --]
On Thu, Mar 29 2018, NeilBrown wrote:
>
> How about storing the hash chains in order by object address?
> Then rhashtable_walk_start() can easily find it's place regardless of
> whether the old object was still present or not, using <= on the
> address.
> "Insert" would need to record an insert location and insert there rather
> than at the head of the chain.
> I might try coding that.
Unfortunately rhltables make this approach unworkable.
However while trying to write the code I found a bug :-(
I'll post a patch for the bug, and a patch to transparently make the
current interface more reliable when the caller keeps the current
object in the table.
I think this is sufficient for all current use-cases.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* [PATCH bpf-next 14/14] nfp: bpf: improve wrong FW response warnings
From: Jakub Kicinski @ 2018-03-29 0:48 UTC (permalink / raw)
To: alexei.starovoitov, daniel
Cc: netdev, oss-drivers, Jan Gossens, Jakub Kicinski
In-Reply-To: <20180329004839.4506-1-jakub.kicinski@netronome.com>
When FW responds with a message of wrong size or type make sure
the type is checked first and included in the wrong size message.
This makes it easier to figure out which FW command failed.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com>
Reviewed-by: Jiong Wang <jiong.wang@netronome.com>
---
drivers/net/ethernet/netronome/nfp/bpf/cmsg.c | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/cmsg.c b/drivers/net/ethernet/netronome/nfp/bpf/cmsg.c
index 80d3aa0fc9d3..7e298148ca26 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/cmsg.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/cmsg.c
@@ -218,17 +218,17 @@ nfp_bpf_cmsg_communicate(struct nfp_app_bpf *bpf, struct sk_buff *skb,
return skb;
hdr = (struct cmsg_hdr *)skb->data;
- /* 0 reply_size means caller will do the validation */
- if (reply_size && skb->len != reply_size) {
- cmsg_warn(bpf, "cmsg drop - wrong size %d != %d!\n",
- skb->len, reply_size);
- goto err_free;
- }
if (hdr->type != __CMSG_REPLY(type)) {
cmsg_warn(bpf, "cmsg drop - wrong type 0x%02x != 0x%02lx!\n",
hdr->type, __CMSG_REPLY(type));
goto err_free;
}
+ /* 0 reply_size means caller will do the validation */
+ if (reply_size && skb->len != reply_size) {
+ cmsg_warn(bpf, "cmsg drop - type 0x%02x wrong size %d != %d!\n",
+ type, skb->len, reply_size);
+ goto err_free;
+ }
return skb;
err_free:
--
2.16.2
^ permalink raw reply related
* [PATCH bpf-next 13/14] nfp: bpf: add support for bpf_get_prandom_u32()
From: Jakub Kicinski @ 2018-03-29 0:48 UTC (permalink / raw)
To: alexei.starovoitov, daniel
Cc: netdev, oss-drivers, Jan Gossens, Jakub Kicinski
In-Reply-To: <20180329004839.4506-1-jakub.kicinski@netronome.com>
NFP has a prng register, which we can read to obtain a u32 worth
of pseudo random data. Generate code for it.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com>
Reviewed-by: Jiong Wang <jiong.wang@netronome.com>
---
drivers/net/ethernet/netronome/nfp/bpf/fw.h | 1 +
drivers/net/ethernet/netronome/nfp/bpf/jit.c | 24 +++++++++++++++++++++--
drivers/net/ethernet/netronome/nfp/bpf/main.c | 12 ++++++++++++
drivers/net/ethernet/netronome/nfp/bpf/main.h | 4 ++++
drivers/net/ethernet/netronome/nfp/bpf/verifier.c | 7 +++++++
drivers/net/ethernet/netronome/nfp/nfp_asm.h | 1 +
6 files changed, 47 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/fw.h b/drivers/net/ethernet/netronome/nfp/bpf/fw.h
index cfcc7bcb2c67..39639ac28b01 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/fw.h
+++ b/drivers/net/ethernet/netronome/nfp/bpf/fw.h
@@ -41,6 +41,7 @@ enum bpf_cap_tlv_type {
NFP_BPF_CAP_TYPE_FUNC = 1,
NFP_BPF_CAP_TYPE_ADJUST_HEAD = 2,
NFP_BPF_CAP_TYPE_MAPS = 3,
+ NFP_BPF_CAP_TYPE_RANDOM = 4,
};
struct nfp_bpf_cap_tlv_func {
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/jit.c b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
index 62431a0aa0f5..4b631e26f199 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/jit.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
@@ -405,7 +405,7 @@ __emit_lcsr(struct nfp_prog *nfp_prog, u16 areg, u16 breg, bool wr, u16 addr,
FIELD_PREP(OP_LCSR_A_SRC, areg) |
FIELD_PREP(OP_LCSR_B_SRC, breg) |
FIELD_PREP(OP_LCSR_WRITE, wr) |
- FIELD_PREP(OP_LCSR_ADDR, addr) |
+ FIELD_PREP(OP_LCSR_ADDR, addr / 4) |
FIELD_PREP(OP_LCSR_SRC_LMEXTN, src_lmextn) |
FIELD_PREP(OP_LCSR_DST_LMEXTN, dst_lmextn);
@@ -433,10 +433,16 @@ static void emit_csr_wr(struct nfp_prog *nfp_prog, swreg src, u16 addr)
return;
}
- __emit_lcsr(nfp_prog, reg.areg, reg.breg, true, addr / 4,
+ __emit_lcsr(nfp_prog, reg.areg, reg.breg, true, addr,
false, reg.src_lmextn);
}
+/* CSR value is read in following immed[gpr, 0] */
+static void __emit_csr_rd(struct nfp_prog *nfp_prog, u16 addr)
+{
+ __emit_lcsr(nfp_prog, 0, 0, false, addr, false, false);
+}
+
static void emit_nop(struct nfp_prog *nfp_prog)
{
__emit_immed(nfp_prog, UR_REG_IMM, UR_REG_IMM, 0, 0, 0, 0, 0, 0, 0);
@@ -1398,6 +1404,18 @@ map_call_stack_common(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
return 0;
}
+static int
+nfp_get_prandom_u32(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
+{
+ __emit_csr_rd(nfp_prog, NFP_CSR_PSEUDO_RND_NUM);
+ /* CSR value is read in following immed[gpr, 0] */
+ emit_immed(nfp_prog, reg_both(0), 0,
+ IMMED_WIDTH_ALL, false, IMMED_SHIFT_0B);
+ emit_immed(nfp_prog, reg_both(1), 0,
+ IMMED_WIDTH_ALL, false, IMMED_SHIFT_0B);
+ return 0;
+}
+
/* --- Callbacks --- */
static int mov_reg64(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
{
@@ -2431,6 +2449,8 @@ static int call(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
case BPF_FUNC_map_update_elem:
case BPF_FUNC_map_delete_elem:
return map_call_stack_common(nfp_prog, meta);
+ case BPF_FUNC_get_prandom_u32:
+ return nfp_get_prandom_u32(nfp_prog, meta);
default:
WARN_ONCE(1, "verifier allowed unsupported function\n");
return -EOPNOTSUPP;
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/main.c b/drivers/net/ethernet/netronome/nfp/bpf/main.c
index a7e217c5204f..f2214101a1b5 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/main.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/main.c
@@ -315,6 +315,14 @@ nfp_bpf_parse_cap_maps(struct nfp_app_bpf *bpf, void __iomem *value, u32 length)
return 0;
}
+static int
+nfp_bpf_parse_cap_random(struct nfp_app_bpf *bpf, void __iomem *value,
+ u32 length)
+{
+ bpf->pseudo_random = true;
+ return 0;
+}
+
static int nfp_bpf_parse_capabilities(struct nfp_app *app)
{
struct nfp_cpp *cpp = app->pf->cpp;
@@ -353,6 +361,10 @@ static int nfp_bpf_parse_capabilities(struct nfp_app *app)
if (nfp_bpf_parse_cap_maps(app->priv, value, length))
goto err_release_free;
break;
+ case NFP_BPF_CAP_TYPE_RANDOM:
+ if (nfp_bpf_parse_cap_random(app->priv, value, length))
+ goto err_release_free;
+ break;
default:
nfp_dbg(cpp, "unknown BPF capability: %d\n", type);
break;
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/main.h b/drivers/net/ethernet/netronome/nfp/bpf/main.h
index a73b86c6ce52..4981c8944ca3 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/main.h
+++ b/drivers/net/ethernet/netronome/nfp/bpf/main.h
@@ -133,6 +133,8 @@ enum pkt_vec {
* @helpers.map_lookup: map lookup helper address
* @helpers.map_update: map update helper address
* @helpers.map_delete: map delete helper address
+ *
+ * @pseudo_random: FW initialized the pseudo-random machinery (CSRs)
*/
struct nfp_app_bpf {
struct nfp_app *app;
@@ -170,6 +172,8 @@ struct nfp_app_bpf {
u32 map_update;
u32 map_delete;
} helpers;
+
+ bool pseudo_random;
};
enum nfp_bpf_map_use {
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/verifier.c b/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
index 486ffd1d5913..06ad53ce4ad9 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
@@ -209,6 +209,13 @@ nfp_bpf_check_call(struct nfp_prog *nfp_prog, struct bpf_verifier_env *env,
meta->func_id ? &meta->arg2 : NULL))
return -EOPNOTSUPP;
break;
+
+ case BPF_FUNC_get_prandom_u32:
+ if (bpf->pseudo_random)
+ break;
+ pr_vlog(env, "bpf_get_prandom_u32(): FW doesn't support random number generation\n");
+ return -EOPNOTSUPP;
+
default:
pr_vlog(env, "unsupported function id: %d\n", func_id);
return -EOPNOTSUPP;
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_asm.h b/drivers/net/ethernet/netronome/nfp/nfp_asm.h
index 36524dd6021b..5f2b2f24f4fa 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_asm.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_asm.h
@@ -284,6 +284,7 @@ enum lcsr_wr_src {
#define NFP_CSR_ACT_LM_ADDR1 0x6c
#define NFP_CSR_ACT_LM_ADDR2 0x94
#define NFP_CSR_ACT_LM_ADDR3 0x9c
+#define NFP_CSR_PSEUDO_RND_NUM 0x148
/* Software register representation, independent of operand type */
#define NN_REG_TYPE GENMASK(31, 24)
--
2.16.2
^ permalink raw reply related
* [PATCH bpf-next 12/14] nfp: bpf: add support for atomic add of unknown values
From: Jakub Kicinski @ 2018-03-29 0:48 UTC (permalink / raw)
To: alexei.starovoitov, daniel
Cc: netdev, oss-drivers, Jan Gossens, Jakub Kicinski
In-Reply-To: <20180329004839.4506-1-jakub.kicinski@netronome.com>
Allow atomic add to be used even when the value is not guaranteed
to fit into a 16 bit immediate. This requires the value to be pulled
as data, and therefore use of a transfer register and a context swap.
Track the information about possible lengths of the value, if it's
guaranteed to be larger than 16bits don't generate the code for the
optimized case at all.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com>
Reviewed-by: Jiong Wang <jiong.wang@netronome.com>
---
drivers/net/ethernet/netronome/nfp/bpf/jit.c | 78 ++++++++++++++++++++---
drivers/net/ethernet/netronome/nfp/bpf/main.h | 7 ++
drivers/net/ethernet/netronome/nfp/bpf/verifier.c | 14 ++--
drivers/net/ethernet/netronome/nfp/nfp_asm.c | 1 +
drivers/net/ethernet/netronome/nfp/nfp_asm.h | 3 +
5 files changed, 88 insertions(+), 15 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/jit.c b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
index db73f56de59a..62431a0aa0f5 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/jit.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
@@ -2125,12 +2125,49 @@ static int mem_stx8(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
static int
mem_xadd(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, bool is64)
{
- swreg addra, addrb, off, prev_alu = imm_a(nfp_prog);
u8 dst_gpr = meta->insn.dst_reg * 2;
u8 src_gpr = meta->insn.src_reg * 2;
+ unsigned int full_add, out;
+ swreg addra, addrb, off;
off = ur_load_imm_any(nfp_prog, meta->insn.off, imm_b(nfp_prog));
+ /* We can fit 16 bits into command immediate, if we know the immediate
+ * is guaranteed to either always or never fit into 16 bit we only
+ * generate code to handle that particular case, otherwise generate
+ * code for both.
+ */
+ out = nfp_prog_current_offset(nfp_prog);
+ full_add = nfp_prog_current_offset(nfp_prog);
+
+ if (meta->insn.off) {
+ out += 2;
+ full_add += 2;
+ }
+ if (meta->xadd_maybe_16bit) {
+ out += 3;
+ full_add += 3;
+ }
+ if (meta->xadd_over_16bit)
+ out += 2 + is64;
+ if (meta->xadd_maybe_16bit && meta->xadd_over_16bit) {
+ out += 5;
+ full_add += 5;
+ }
+
+ /* Generate the branch for choosing add_imm vs add */
+ if (meta->xadd_maybe_16bit && meta->xadd_over_16bit) {
+ swreg max_imm = imm_a(nfp_prog);
+
+ wrp_immed(nfp_prog, max_imm, 0xffff);
+ emit_alu(nfp_prog, reg_none(),
+ max_imm, ALU_OP_SUB, reg_b(src_gpr));
+ emit_alu(nfp_prog, reg_none(),
+ reg_imm(0), ALU_OP_SUB_C, reg_b(src_gpr + 1));
+ emit_br(nfp_prog, BR_BLO, full_add, meta->insn.off ? 2 : 0);
+ /* defer for add */
+ }
+
/* If insn has an offset add to the address */
if (!meta->insn.off) {
addra = reg_a(dst_gpr);
@@ -2144,13 +2181,38 @@ mem_xadd(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, bool is64)
addrb = imma_b(nfp_prog);
}
- wrp_immed(nfp_prog, prev_alu,
- FIELD_PREP(CMD_OVE_DATA, 2) |
- CMD_OVE_LEN |
- FIELD_PREP(CMD_OV_LEN, 0x8 | is64 << 2));
- wrp_reg_or_subpart(nfp_prog, prev_alu, reg_b(src_gpr), 2, 2);
- emit_cmd_indir(nfp_prog, CMD_TGT_ADD_IMM, CMD_MODE_40b_BA, 0,
- addra, addrb, 0, CMD_CTX_NO_SWAP);
+ /* Generate the add_imm if 16 bits are possible */
+ if (meta->xadd_maybe_16bit) {
+ swreg prev_alu = imm_a(nfp_prog);
+
+ wrp_immed(nfp_prog, prev_alu,
+ FIELD_PREP(CMD_OVE_DATA, 2) |
+ CMD_OVE_LEN |
+ FIELD_PREP(CMD_OV_LEN, 0x8 | is64 << 2));
+ wrp_reg_or_subpart(nfp_prog, prev_alu, reg_b(src_gpr), 2, 2);
+ emit_cmd_indir(nfp_prog, CMD_TGT_ADD_IMM, CMD_MODE_40b_BA, 0,
+ addra, addrb, 0, CMD_CTX_NO_SWAP);
+
+ if (meta->xadd_over_16bit)
+ emit_br(nfp_prog, BR_UNC, out, 0);
+ }
+
+ if (!nfp_prog_confirm_current_offset(nfp_prog, full_add))
+ return -EINVAL;
+
+ /* Generate the add if 16 bits are not guaranteed */
+ if (meta->xadd_over_16bit) {
+ emit_cmd(nfp_prog, CMD_TGT_ADD, CMD_MODE_40b_BA, 0,
+ addra, addrb, is64 << 2,
+ is64 ? CMD_CTX_SWAP_DEFER2 : CMD_CTX_SWAP_DEFER1);
+
+ wrp_mov(nfp_prog, reg_xfer(0), reg_a(src_gpr));
+ if (is64)
+ wrp_mov(nfp_prog, reg_xfer(1), reg_a(src_gpr + 1));
+ }
+
+ if (!nfp_prog_confirm_current_offset(nfp_prog, out))
+ return -EINVAL;
return 0;
}
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/main.h b/drivers/net/ethernet/netronome/nfp/bpf/main.h
index 877be7143991..a73b86c6ce52 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/main.h
+++ b/drivers/net/ethernet/netronome/nfp/bpf/main.h
@@ -229,6 +229,8 @@ struct nfp_bpf_reg_state {
* @pkt_cache.range_start: start offset for associated packet data cache
* @pkt_cache.range_end: end offset for associated packet data cache
* @pkt_cache.do_init: this read needs to initialize packet data cache
+ * @xadd_over_16bit: 16bit immediate is not guaranteed
+ * @xadd_maybe_16bit: 16bit immediate is possible
* @jmp_dst: destination info for jump instructions
* @func_id: function id for call instructions
* @arg1: arg1 for call instructions
@@ -243,6 +245,7 @@ struct nfp_bpf_reg_state {
struct nfp_insn_meta {
struct bpf_insn insn;
union {
+ /* pointer ops (ld/st/xadd) */
struct {
struct bpf_reg_state ptr;
struct bpf_insn *paired_st;
@@ -253,8 +256,12 @@ struct nfp_insn_meta {
s16 range_end;
bool do_init;
} pkt_cache;
+ bool xadd_over_16bit;
+ bool xadd_maybe_16bit;
};
+ /* jump */
struct nfp_insn_meta *jmp_dst;
+ /* function calls */
struct {
u32 func_id;
struct bpf_reg_state arg1;
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/verifier.c b/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
index 40619efea77d..486ffd1d5913 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
@@ -414,16 +414,16 @@ nfp_bpf_check_xadd(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta,
dreg->type);
return -EOPNOTSUPP;
}
- if (sreg->type != SCALAR_VALUE ||
- sreg->var_off.value > 0xffff || sreg->var_off.mask > 0xffff) {
- char tn_buf[48];
-
- tnum_strn(tn_buf, sizeof(tn_buf), sreg->var_off);
- pr_vlog(env, "atomic add not of a small constant scalar: %s\n",
- tn_buf);
+ if (sreg->type != SCALAR_VALUE) {
+ pr_vlog(env, "atomic add not of a scalar: %d\n", sreg->type);
return -EOPNOTSUPP;
}
+ meta->xadd_over_16bit |=
+ sreg->var_off.value > 0xffff || sreg->var_off.mask > 0xffff;
+ meta->xadd_maybe_16bit |=
+ (sreg->var_off.value & ~sreg->var_off.mask) <= 0xffff;
+
return nfp_bpf_check_ptr(nfp_prog, meta, env, meta->insn.dst_reg);
}
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_asm.c b/drivers/net/ethernet/netronome/nfp/nfp_asm.c
index 3c0107ac9a2c..cc6ace2be8a9 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_asm.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_asm.c
@@ -48,6 +48,7 @@ const struct cmd_tgt_act cmd_tgt_act[__CMD_TGT_MAP_SIZE] = {
[CMD_TGT_READ32_SWAP] = { 0x02, 0x5c },
[CMD_TGT_READ_LE] = { 0x01, 0x40 },
[CMD_TGT_READ_SWAP_LE] = { 0x03, 0x40 },
+ [CMD_TGT_ADD] = { 0x00, 0x47 },
[CMD_TGT_ADD_IMM] = { 0x02, 0x47 },
};
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_asm.h b/drivers/net/ethernet/netronome/nfp/nfp_asm.h
index 185192590a17..36524dd6021b 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_asm.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_asm.h
@@ -238,6 +238,7 @@ enum cmd_tgt_map {
CMD_TGT_READ32_SWAP,
CMD_TGT_READ_LE,
CMD_TGT_READ_SWAP_LE,
+ CMD_TGT_ADD,
CMD_TGT_ADD_IMM,
__CMD_TGT_MAP_SIZE,
};
@@ -252,6 +253,8 @@ enum cmd_mode {
enum cmd_ctx_swap {
CMD_CTX_SWAP = 0,
+ CMD_CTX_SWAP_DEFER1 = 1,
+ CMD_CTX_SWAP_DEFER2 = 2,
CMD_CTX_NO_SWAP = 3,
};
--
2.16.2
^ permalink raw reply related
* [PATCH bpf-next 11/14] nfp: bpf: expose command delay slots
From: Jakub Kicinski @ 2018-03-29 0:48 UTC (permalink / raw)
To: alexei.starovoitov, daniel
Cc: netdev, oss-drivers, Jan Gossens, Jakub Kicinski
In-Reply-To: <20180329004839.4506-1-jakub.kicinski@netronome.com>
Allow callers to control the delay slots of commands, instead of
giving them just a wait/nowait choice.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com>
Reviewed-by: Jiong Wang <jiong.wang@netronome.com>
---
drivers/net/ethernet/netronome/nfp/bpf/jit.c | 53 +++++++++++++---------------
1 file changed, 24 insertions(+), 29 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/jit.c b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
index c7fdb8c7ae17..db73f56de59a 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/jit.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
@@ -103,23 +103,18 @@ nfp_prog_confirm_current_offset(struct nfp_prog *nfp_prog, unsigned int off)
/* --- Emitters --- */
static void
__emit_cmd(struct nfp_prog *nfp_prog, enum cmd_tgt_map op,
- u8 mode, u8 xfer, u8 areg, u8 breg, u8 size, bool sync, bool indir)
+ u8 mode, u8 xfer, u8 areg, u8 breg, u8 size, enum cmd_ctx_swap ctx,
+ bool indir)
{
- enum cmd_ctx_swap ctx;
u64 insn;
- if (sync)
- ctx = CMD_CTX_SWAP;
- else
- ctx = CMD_CTX_NO_SWAP;
-
insn = FIELD_PREP(OP_CMD_A_SRC, areg) |
FIELD_PREP(OP_CMD_CTX, ctx) |
FIELD_PREP(OP_CMD_B_SRC, breg) |
FIELD_PREP(OP_CMD_TOKEN, cmd_tgt_act[op].token) |
FIELD_PREP(OP_CMD_XFER, xfer) |
FIELD_PREP(OP_CMD_CNT, size) |
- FIELD_PREP(OP_CMD_SIG, sync) |
+ FIELD_PREP(OP_CMD_SIG, ctx != CMD_CTX_NO_SWAP) |
FIELD_PREP(OP_CMD_TGT_CMD, cmd_tgt_act[op].tgt_cmd) |
FIELD_PREP(OP_CMD_INDIR, indir) |
FIELD_PREP(OP_CMD_MODE, mode);
@@ -129,7 +124,7 @@ __emit_cmd(struct nfp_prog *nfp_prog, enum cmd_tgt_map op,
static void
emit_cmd_any(struct nfp_prog *nfp_prog, enum cmd_tgt_map op, u8 mode, u8 xfer,
- swreg lreg, swreg rreg, u8 size, bool sync, bool indir)
+ swreg lreg, swreg rreg, u8 size, enum cmd_ctx_swap ctx, bool indir)
{
struct nfp_insn_re_regs reg;
int err;
@@ -150,22 +145,22 @@ emit_cmd_any(struct nfp_prog *nfp_prog, enum cmd_tgt_map op, u8 mode, u8 xfer,
return;
}
- __emit_cmd(nfp_prog, op, mode, xfer, reg.areg, reg.breg, size, sync,
+ __emit_cmd(nfp_prog, op, mode, xfer, reg.areg, reg.breg, size, ctx,
indir);
}
static void
emit_cmd(struct nfp_prog *nfp_prog, enum cmd_tgt_map op, u8 mode, u8 xfer,
- swreg lreg, swreg rreg, u8 size, bool sync)
+ swreg lreg, swreg rreg, u8 size, enum cmd_ctx_swap ctx)
{
- emit_cmd_any(nfp_prog, op, mode, xfer, lreg, rreg, size, sync, false);
+ emit_cmd_any(nfp_prog, op, mode, xfer, lreg, rreg, size, ctx, false);
}
static void
emit_cmd_indir(struct nfp_prog *nfp_prog, enum cmd_tgt_map op, u8 mode, u8 xfer,
- swreg lreg, swreg rreg, u8 size, bool sync)
+ swreg lreg, swreg rreg, u8 size, enum cmd_ctx_swap ctx)
{
- emit_cmd_any(nfp_prog, op, mode, xfer, lreg, rreg, size, sync, true);
+ emit_cmd_any(nfp_prog, op, mode, xfer, lreg, rreg, size, ctx, true);
}
static void
@@ -610,7 +605,7 @@ static int nfp_cpp_memcpy(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
/* Memory read from source addr into transfer-in registers. */
emit_cmd_any(nfp_prog, CMD_TGT_READ32_SWAP,
src_40bit_addr ? CMD_MODE_40b_BA : CMD_MODE_32b, 0,
- src_base, off, xfer_num - 1, true, len > 32);
+ src_base, off, xfer_num - 1, CMD_CTX_SWAP, len > 32);
/* Move from transfer-in to transfer-out. */
for (i = 0; i < xfer_num; i++)
@@ -622,39 +617,39 @@ static int nfp_cpp_memcpy(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
/* Use single direct_ref write8. */
emit_cmd(nfp_prog, CMD_TGT_WRITE8_SWAP, CMD_MODE_32b, 0,
reg_a(meta->paired_st->dst_reg * 2), off, len - 1,
- true);
+ CMD_CTX_SWAP);
} else if (len <= 32 && IS_ALIGNED(len, 4)) {
/* Use single direct_ref write32. */
emit_cmd(nfp_prog, CMD_TGT_WRITE32_SWAP, CMD_MODE_32b, 0,
reg_a(meta->paired_st->dst_reg * 2), off, xfer_num - 1,
- true);
+ CMD_CTX_SWAP);
} else if (len <= 32) {
/* Use single indirect_ref write8. */
wrp_immed(nfp_prog, reg_none(),
CMD_OVE_LEN | FIELD_PREP(CMD_OV_LEN, len - 1));
emit_cmd_indir(nfp_prog, CMD_TGT_WRITE8_SWAP, CMD_MODE_32b, 0,
reg_a(meta->paired_st->dst_reg * 2), off,
- len - 1, true);
+ len - 1, CMD_CTX_SWAP);
} else if (IS_ALIGNED(len, 4)) {
/* Use single indirect_ref write32. */
wrp_immed(nfp_prog, reg_none(),
CMD_OVE_LEN | FIELD_PREP(CMD_OV_LEN, xfer_num - 1));
emit_cmd_indir(nfp_prog, CMD_TGT_WRITE32_SWAP, CMD_MODE_32b, 0,
reg_a(meta->paired_st->dst_reg * 2), off,
- xfer_num - 1, true);
+ xfer_num - 1, CMD_CTX_SWAP);
} else if (len <= 40) {
/* Use one direct_ref write32 to write the first 32-bytes, then
* another direct_ref write8 to write the remaining bytes.
*/
emit_cmd(nfp_prog, CMD_TGT_WRITE32_SWAP, CMD_MODE_32b, 0,
reg_a(meta->paired_st->dst_reg * 2), off, 7,
- true);
+ CMD_CTX_SWAP);
off = re_load_imm_any(nfp_prog, meta->paired_st->off + 32,
imm_b(nfp_prog));
emit_cmd(nfp_prog, CMD_TGT_WRITE8_SWAP, CMD_MODE_32b, 8,
reg_a(meta->paired_st->dst_reg * 2), off, len - 33,
- true);
+ CMD_CTX_SWAP);
} else {
/* Use one indirect_ref write32 to write 4-bytes aligned length,
* then another direct_ref write8 to write the remaining bytes.
@@ -665,12 +660,12 @@ static int nfp_cpp_memcpy(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
CMD_OVE_LEN | FIELD_PREP(CMD_OV_LEN, xfer_num - 2));
emit_cmd_indir(nfp_prog, CMD_TGT_WRITE32_SWAP, CMD_MODE_32b, 0,
reg_a(meta->paired_st->dst_reg * 2), off,
- xfer_num - 2, true);
+ xfer_num - 2, CMD_CTX_SWAP);
new_off = meta->paired_st->off + (xfer_num - 1) * 4;
off = re_load_imm_any(nfp_prog, new_off, imm_b(nfp_prog));
emit_cmd(nfp_prog, CMD_TGT_WRITE8_SWAP, CMD_MODE_32b,
xfer_num - 1, reg_a(meta->paired_st->dst_reg * 2), off,
- (len & 0x3) - 1, true);
+ (len & 0x3) - 1, CMD_CTX_SWAP);
}
/* TODO: The following extra load is to make sure data flow be identical
@@ -731,7 +726,7 @@ data_ld(struct nfp_prog *nfp_prog, swreg offset, u8 dst_gpr, int size)
shift = size < 4 ? 4 - size : 0;
emit_cmd(nfp_prog, CMD_TGT_READ8, CMD_MODE_32b, 0,
- pptr_reg(nfp_prog), offset, sz - 1, true);
+ pptr_reg(nfp_prog), offset, sz - 1, CMD_CTX_SWAP);
i = 0;
if (shift)
@@ -761,7 +756,7 @@ data_ld_host_order(struct nfp_prog *nfp_prog, u8 dst_gpr,
mask = size < 4 ? GENMASK(size - 1, 0) : 0;
emit_cmd(nfp_prog, CMD_TGT_READ32_SWAP, mode, 0,
- lreg, rreg, sz / 4 - 1, true);
+ lreg, rreg, sz / 4 - 1, CMD_CTX_SWAP);
i = 0;
if (mask)
@@ -841,7 +836,7 @@ data_stx_host_order(struct nfp_prog *nfp_prog, u8 dst_gpr, swreg offset,
wrp_mov(nfp_prog, reg_xfer(i), reg_a(src_gpr + i));
emit_cmd(nfp_prog, CMD_TGT_WRITE8_SWAP, CMD_MODE_32b, 0,
- reg_a(dst_gpr), offset, size - 1, true);
+ reg_a(dst_gpr), offset, size - 1, CMD_CTX_SWAP);
return 0;
}
@@ -855,7 +850,7 @@ data_st_host_order(struct nfp_prog *nfp_prog, u8 dst_gpr, swreg offset,
wrp_immed(nfp_prog, reg_xfer(1), imm >> 32);
emit_cmd(nfp_prog, CMD_TGT_WRITE8_SWAP, CMD_MODE_32b, 0,
- reg_a(dst_gpr), offset, size - 1, true);
+ reg_a(dst_gpr), offset, size - 1, CMD_CTX_SWAP);
return 0;
}
@@ -1876,7 +1871,7 @@ mem_ldx_data_init_pktcache(struct nfp_prog *nfp_prog,
/* Cache memory into transfer-in registers. */
emit_cmd_any(nfp_prog, CMD_TGT_READ32_SWAP, CMD_MODE_32b, 0, src_base,
- off, xfer_num - 1, true, indir);
+ off, xfer_num - 1, CMD_CTX_SWAP, indir);
}
static int
@@ -2155,7 +2150,7 @@ mem_xadd(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, bool is64)
FIELD_PREP(CMD_OV_LEN, 0x8 | is64 << 2));
wrp_reg_or_subpart(nfp_prog, prev_alu, reg_b(src_gpr), 2, 2);
emit_cmd_indir(nfp_prog, CMD_TGT_ADD_IMM, CMD_MODE_40b_BA, 0,
- addra, addrb, 0, false);
+ addra, addrb, 0, CMD_CTX_NO_SWAP);
return 0;
}
--
2.16.2
^ permalink raw reply related
* [PATCH bpf-next 10/14] nfp: bpf: add basic support for atomic adds
From: Jakub Kicinski @ 2018-03-29 0:48 UTC (permalink / raw)
To: alexei.starovoitov, daniel
Cc: netdev, oss-drivers, Jan Gossens, Jakub Kicinski
In-Reply-To: <20180329004839.4506-1-jakub.kicinski@netronome.com>
Implement atomic add operation for 32 and 64 bit values. Depend
on the verifier to ensure alignment. Values have to be kept in
big endian and swapped upon read/write. For now only support
atomic add of a constant.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com>
Reviewed-by: Jiong Wang <jiong.wang@netronome.com>
---
drivers/net/ethernet/netronome/nfp/bpf/jit.c | 45 ++++++++++
drivers/net/ethernet/netronome/nfp/bpf/main.h | 17 ++++
drivers/net/ethernet/netronome/nfp/bpf/offload.c | 45 +++++++++-
drivers/net/ethernet/netronome/nfp/bpf/verifier.c | 105 ++++++++++++++++++++++
drivers/net/ethernet/netronome/nfp/nfp_asm.c | 1 +
drivers/net/ethernet/netronome/nfp/nfp_asm.h | 2 +
6 files changed, 212 insertions(+), 3 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/jit.c b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
index d8df56087961..c7fdb8c7ae17 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/jit.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
@@ -2127,6 +2127,49 @@ static int mem_stx8(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
return mem_stx(nfp_prog, meta, 8);
}
+static int
+mem_xadd(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, bool is64)
+{
+ swreg addra, addrb, off, prev_alu = imm_a(nfp_prog);
+ u8 dst_gpr = meta->insn.dst_reg * 2;
+ u8 src_gpr = meta->insn.src_reg * 2;
+
+ off = ur_load_imm_any(nfp_prog, meta->insn.off, imm_b(nfp_prog));
+
+ /* If insn has an offset add to the address */
+ if (!meta->insn.off) {
+ addra = reg_a(dst_gpr);
+ addrb = reg_b(dst_gpr + 1);
+ } else {
+ emit_alu(nfp_prog, imma_a(nfp_prog),
+ reg_a(dst_gpr), ALU_OP_ADD, off);
+ emit_alu(nfp_prog, imma_b(nfp_prog),
+ reg_a(dst_gpr + 1), ALU_OP_ADD_C, reg_imm(0));
+ addra = imma_a(nfp_prog);
+ addrb = imma_b(nfp_prog);
+ }
+
+ wrp_immed(nfp_prog, prev_alu,
+ FIELD_PREP(CMD_OVE_DATA, 2) |
+ CMD_OVE_LEN |
+ FIELD_PREP(CMD_OV_LEN, 0x8 | is64 << 2));
+ wrp_reg_or_subpart(nfp_prog, prev_alu, reg_b(src_gpr), 2, 2);
+ emit_cmd_indir(nfp_prog, CMD_TGT_ADD_IMM, CMD_MODE_40b_BA, 0,
+ addra, addrb, 0, false);
+
+ return 0;
+}
+
+static int mem_xadd4(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
+{
+ return mem_xadd(nfp_prog, meta, false);
+}
+
+static int mem_xadd8(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
+{
+ return mem_xadd(nfp_prog, meta, true);
+}
+
static int jump(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
{
emit_br(nfp_prog, BR_UNC, meta->insn.off, 0);
@@ -2390,6 +2433,8 @@ static const instr_cb_t instr_cb[256] = {
[BPF_STX | BPF_MEM | BPF_H] = mem_stx2,
[BPF_STX | BPF_MEM | BPF_W] = mem_stx4,
[BPF_STX | BPF_MEM | BPF_DW] = mem_stx8,
+ [BPF_STX | BPF_XADD | BPF_W] = mem_xadd4,
+ [BPF_STX | BPF_XADD | BPF_DW] = mem_xadd8,
[BPF_ST | BPF_MEM | BPF_B] = mem_st1,
[BPF_ST | BPF_MEM | BPF_H] = mem_st2,
[BPF_ST | BPF_MEM | BPF_W] = mem_st4,
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/main.h b/drivers/net/ethernet/netronome/nfp/bpf/main.h
index 26bb491224b3..877be7143991 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/main.h
+++ b/drivers/net/ethernet/netronome/nfp/bpf/main.h
@@ -72,6 +72,7 @@ enum nfp_relo_type {
#define BR_OFF_RELO 15000
enum static_regs {
+ STATIC_REG_IMMA = 20, /* Bank AB */
STATIC_REG_IMM = 21, /* Bank AB */
STATIC_REG_STACK = 22, /* Bank A */
STATIC_REG_PKT_LEN = 22, /* Bank B */
@@ -91,6 +92,8 @@ enum pkt_vec {
#define pptr_reg(np) pv_ctm_ptr(np)
#define imm_a(np) reg_a(STATIC_REG_IMM)
#define imm_b(np) reg_b(STATIC_REG_IMM)
+#define imma_a(np) reg_a(STATIC_REG_IMMA)
+#define imma_b(np) reg_b(STATIC_REG_IMMA)
#define imm_both(np) reg_both(STATIC_REG_IMM)
#define NFP_BPF_ABI_FLAGS reg_imm(0)
@@ -169,18 +172,27 @@ struct nfp_app_bpf {
} helpers;
};
+enum nfp_bpf_map_use {
+ NFP_MAP_UNUSED = 0,
+ NFP_MAP_USE_READ,
+ NFP_MAP_USE_WRITE,
+ NFP_MAP_USE_ATOMIC_CNT,
+};
+
/**
* struct nfp_bpf_map - private per-map data attached to BPF maps for offload
* @offmap: pointer to the offloaded BPF map
* @bpf: back pointer to bpf app private structure
* @tid: table id identifying map on datapath
* @l: link on the nfp_app_bpf->map_list list
+ * @use_map: map of how the value is used (in 4B chunks)
*/
struct nfp_bpf_map {
struct bpf_offloaded_map *offmap;
struct nfp_app_bpf *bpf;
u32 tid;
struct list_head l;
+ enum nfp_bpf_map_use use_map[];
};
struct nfp_prog;
@@ -320,6 +332,11 @@ static inline bool is_mbpf_classic_store_pkt(const struct nfp_insn_meta *meta)
return is_mbpf_classic_store(meta) && meta->ptr.type == PTR_TO_PACKET;
}
+static inline bool is_mbpf_xadd(const struct nfp_insn_meta *meta)
+{
+ return (meta->insn.code & ~BPF_SIZE_MASK) == (BPF_STX | BPF_XADD);
+}
+
/**
* struct nfp_prog - nfp BPF program
* @bpf: backpointer to the bpf app priv structure
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/offload.c b/drivers/net/ethernet/netronome/nfp/bpf/offload.c
index 0a7732385469..42d98792bd25 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/offload.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/offload.c
@@ -164,6 +164,41 @@ static int nfp_bpf_destroy(struct nfp_net *nn, struct bpf_prog *prog)
return 0;
}
+/* Atomic engine requires values to be in big endian, we need to byte swap
+ * the value words used with xadd.
+ */
+static void nfp_map_bpf_byte_swap(struct nfp_bpf_map *nfp_map, void *value)
+{
+ u32 *word = value;
+ unsigned int i;
+
+ for (i = 0; i < DIV_ROUND_UP(nfp_map->offmap->map.value_size, 4); i++)
+ if (nfp_map->use_map[i] == NFP_MAP_USE_ATOMIC_CNT)
+ word[i] = (__force u32)cpu_to_be32(word[i]);
+}
+
+static int
+nfp_bpf_map_lookup_entry(struct bpf_offloaded_map *offmap,
+ void *key, void *value)
+{
+ int err;
+
+ err = nfp_bpf_ctrl_lookup_entry(offmap, key, value);
+ if (err)
+ return err;
+
+ nfp_map_bpf_byte_swap(offmap->dev_priv, value);
+ return 0;
+}
+
+static int
+nfp_bpf_map_update_entry(struct bpf_offloaded_map *offmap,
+ void *key, void *value, u64 flags)
+{
+ nfp_map_bpf_byte_swap(offmap->dev_priv, value);
+ return nfp_bpf_ctrl_update_entry(offmap, key, value, flags);
+}
+
static int
nfp_bpf_map_get_next_key(struct bpf_offloaded_map *offmap,
void *key, void *next_key)
@@ -183,8 +218,8 @@ nfp_bpf_map_delete_elem(struct bpf_offloaded_map *offmap, void *key)
static const struct bpf_map_dev_ops nfp_bpf_map_ops = {
.map_get_next_key = nfp_bpf_map_get_next_key,
- .map_lookup_elem = nfp_bpf_ctrl_lookup_entry,
- .map_update_elem = nfp_bpf_ctrl_update_entry,
+ .map_lookup_elem = nfp_bpf_map_lookup_entry,
+ .map_update_elem = nfp_bpf_map_update_entry,
.map_delete_elem = nfp_bpf_map_delete_elem,
};
@@ -192,6 +227,7 @@ static int
nfp_bpf_map_alloc(struct nfp_app_bpf *bpf, struct bpf_offloaded_map *offmap)
{
struct nfp_bpf_map *nfp_map;
+ unsigned int use_map_size;
long long int res;
if (!bpf->maps.types)
@@ -226,7 +262,10 @@ nfp_bpf_map_alloc(struct nfp_app_bpf *bpf, struct bpf_offloaded_map *offmap)
return -ENOMEM;
}
- nfp_map = kzalloc(sizeof(*nfp_map), GFP_USER);
+ use_map_size = DIV_ROUND_UP(offmap->map.value_size, 4) *
+ FIELD_SIZEOF(struct nfp_bpf_map, use_map[0]);
+
+ nfp_map = kzalloc(sizeof(*nfp_map) + use_map_size, GFP_USER);
if (!nfp_map)
return -ENOMEM;
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/verifier.c b/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
index 7d67ffc897dd..40619efea77d 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
@@ -285,6 +285,72 @@ nfp_bpf_check_stack_access(struct nfp_prog *nfp_prog,
return -EINVAL;
}
+static const char *nfp_bpf_map_use_name(enum nfp_bpf_map_use use)
+{
+ static const char * const names[] = {
+ [NFP_MAP_UNUSED] = "unused",
+ [NFP_MAP_USE_READ] = "read",
+ [NFP_MAP_USE_WRITE] = "write",
+ [NFP_MAP_USE_ATOMIC_CNT] = "atomic",
+ };
+
+ if (use >= ARRAY_SIZE(names) || !names[use])
+ return "unknown";
+ return names[use];
+}
+
+static int
+nfp_bpf_map_mark_used_one(struct bpf_verifier_env *env,
+ struct nfp_bpf_map *nfp_map,
+ unsigned int off, enum nfp_bpf_map_use use)
+{
+ if (nfp_map->use_map[off / 4] != NFP_MAP_UNUSED &&
+ nfp_map->use_map[off / 4] != use) {
+ pr_vlog(env, "map value use type conflict %s vs %s off: %u\n",
+ nfp_bpf_map_use_name(nfp_map->use_map[off / 4]),
+ nfp_bpf_map_use_name(use), off);
+ return -EOPNOTSUPP;
+ }
+
+ nfp_map->use_map[off / 4] = use;
+
+ return 0;
+}
+
+static int
+nfp_bpf_map_mark_used(struct bpf_verifier_env *env, struct nfp_insn_meta *meta,
+ const struct bpf_reg_state *reg,
+ enum nfp_bpf_map_use use)
+{
+ struct bpf_offloaded_map *offmap;
+ struct nfp_bpf_map *nfp_map;
+ unsigned int size, off;
+ int i, err;
+
+ if (!tnum_is_const(reg->var_off)) {
+ pr_vlog(env, "map value offset is variable\n");
+ return -EOPNOTSUPP;
+ }
+
+ off = reg->var_off.value + meta->insn.off + reg->off;
+ size = BPF_LDST_BYTES(&meta->insn);
+ offmap = map_to_offmap(reg->map_ptr);
+ nfp_map = offmap->dev_priv;
+
+ if (off + size > offmap->map.value_size) {
+ pr_vlog(env, "map value access out-of-bounds\n");
+ return -EINVAL;
+ }
+
+ for (i = 0; i < size; i += 4 - (off + i) % 4) {
+ err = nfp_bpf_map_mark_used_one(env, nfp_map, off + i, use);
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
static int
nfp_bpf_check_ptr(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta,
struct bpf_verifier_env *env, u8 reg_no)
@@ -307,10 +373,22 @@ nfp_bpf_check_ptr(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta,
}
if (reg->type == PTR_TO_MAP_VALUE) {
+ if (is_mbpf_load(meta)) {
+ err = nfp_bpf_map_mark_used(env, meta, reg,
+ NFP_MAP_USE_READ);
+ if (err)
+ return err;
+ }
if (is_mbpf_store(meta)) {
pr_vlog(env, "map writes not supported\n");
return -EOPNOTSUPP;
}
+ if (is_mbpf_xadd(meta)) {
+ err = nfp_bpf_map_mark_used(env, meta, reg,
+ NFP_MAP_USE_ATOMIC_CNT);
+ if (err)
+ return err;
+ }
}
if (meta->ptr.type != NOT_INIT && meta->ptr.type != reg->type) {
@@ -324,6 +402,31 @@ nfp_bpf_check_ptr(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta,
return 0;
}
+static int
+nfp_bpf_check_xadd(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta,
+ struct bpf_verifier_env *env)
+{
+ const struct bpf_reg_state *sreg = cur_regs(env) + meta->insn.src_reg;
+ const struct bpf_reg_state *dreg = cur_regs(env) + meta->insn.dst_reg;
+
+ if (dreg->type != PTR_TO_MAP_VALUE) {
+ pr_vlog(env, "atomic add not to a map value pointer: %d\n",
+ dreg->type);
+ return -EOPNOTSUPP;
+ }
+ if (sreg->type != SCALAR_VALUE ||
+ sreg->var_off.value > 0xffff || sreg->var_off.mask > 0xffff) {
+ char tn_buf[48];
+
+ tnum_strn(tn_buf, sizeof(tn_buf), sreg->var_off);
+ pr_vlog(env, "atomic add not of a small constant scalar: %s\n",
+ tn_buf);
+ return -EOPNOTSUPP;
+ }
+
+ return nfp_bpf_check_ptr(nfp_prog, meta, env, meta->insn.dst_reg);
+}
+
static int
nfp_verify_insn(struct bpf_verifier_env *env, int insn_idx, int prev_insn_idx)
{
@@ -356,6 +459,8 @@ nfp_verify_insn(struct bpf_verifier_env *env, int insn_idx, int prev_insn_idx)
if (is_mbpf_store(meta))
return nfp_bpf_check_ptr(nfp_prog, meta, env,
meta->insn.dst_reg);
+ if (is_mbpf_xadd(meta))
+ return nfp_bpf_check_xadd(nfp_prog, meta, env);
return 0;
}
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_asm.c b/drivers/net/ethernet/netronome/nfp/nfp_asm.c
index 1e597600c693..3c0107ac9a2c 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_asm.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_asm.c
@@ -48,6 +48,7 @@ const struct cmd_tgt_act cmd_tgt_act[__CMD_TGT_MAP_SIZE] = {
[CMD_TGT_READ32_SWAP] = { 0x02, 0x5c },
[CMD_TGT_READ_LE] = { 0x01, 0x40 },
[CMD_TGT_READ_SWAP_LE] = { 0x03, 0x40 },
+ [CMD_TGT_ADD_IMM] = { 0x02, 0x47 },
};
static bool unreg_is_imm(u16 reg)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_asm.h b/drivers/net/ethernet/netronome/nfp/nfp_asm.h
index 150d28f9cd52..185192590a17 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_asm.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_asm.h
@@ -238,6 +238,7 @@ enum cmd_tgt_map {
CMD_TGT_READ32_SWAP,
CMD_TGT_READ_LE,
CMD_TGT_READ_SWAP_LE,
+ CMD_TGT_ADD_IMM,
__CMD_TGT_MAP_SIZE,
};
@@ -254,6 +255,7 @@ enum cmd_ctx_swap {
CMD_CTX_NO_SWAP = 3,
};
+#define CMD_OVE_DATA GENMASK(5, 3)
#define CMD_OVE_LEN BIT(7)
#define CMD_OV_LEN GENMASK(12, 8)
--
2.16.2
^ permalink raw reply related
* [PATCH bpf-next 09/14] bpf: add parenthesis around argument of BPF_LDST_BYTES()
From: Jakub Kicinski @ 2018-03-29 0:48 UTC (permalink / raw)
To: alexei.starovoitov, daniel
Cc: netdev, oss-drivers, Jan Gossens, Jakub Kicinski
In-Reply-To: <20180329004839.4506-1-jakub.kicinski@netronome.com>
BPF_LDST_BYTES() does not put it's argument in parenthesis
when referencing it. This makes it impossible to pass pointers
obtained by address-of operator (e.g. BPF_LDST_BYTES(&insn)).
Add the parenthesis.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com>
---
include/linux/filter.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 109d05ccea9a..c2f167db8bd5 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -372,7 +372,7 @@ struct xdp_rxq_info;
#define BPF_LDST_BYTES(insn) \
({ \
- const int __size = bpf_size_to_bytes(BPF_SIZE(insn->code)); \
+ const int __size = bpf_size_to_bytes(BPF_SIZE((insn)->code)); \
WARN_ON(__size < 0); \
__size; \
})
--
2.16.2
^ permalink raw reply related
* [PATCH bpf-next 08/14] nfp: bpf: add map deletes from the datapath
From: Jakub Kicinski @ 2018-03-29 0:48 UTC (permalink / raw)
To: alexei.starovoitov, daniel
Cc: netdev, oss-drivers, Jan Gossens, Jakub Kicinski
In-Reply-To: <20180329004839.4506-1-jakub.kicinski@netronome.com>
Support calling map_delete_elem() FW helper from the datapath
programs. For JIT checks and code are basically equivalent
to map lookups. Similarly to other map helper key must be on
the stack. Different pointer types are left for future extension.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com>
Reviewed-by: Jiong Wang <jiong.wang@netronome.com>
---
drivers/net/ethernet/netronome/nfp/bpf/jit.c | 4 ++++
drivers/net/ethernet/netronome/nfp/bpf/main.c | 3 +++
drivers/net/ethernet/netronome/nfp/bpf/main.h | 2 ++
drivers/net/ethernet/netronome/nfp/bpf/verifier.c | 8 ++++++++
4 files changed, 17 insertions(+)
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/jit.c b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
index c63368fc28f6..d8df56087961 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/jit.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
@@ -2329,6 +2329,7 @@ static int call(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
return adjust_head(nfp_prog, meta);
case BPF_FUNC_map_lookup_elem:
case BPF_FUNC_map_update_elem:
+ case BPF_FUNC_map_delete_elem:
return map_call_stack_common(nfp_prog, meta);
default:
WARN_ONCE(1, "verifier allowed unsupported function\n");
@@ -3216,6 +3217,9 @@ void *nfp_bpf_relo_for_vnic(struct nfp_prog *nfp_prog, struct nfp_bpf_vnic *bv)
case BPF_FUNC_map_update_elem:
val = nfp_prog->bpf->helpers.map_update;
break;
+ case BPF_FUNC_map_delete_elem:
+ val = nfp_prog->bpf->helpers.map_delete;
+ break;
default:
pr_err("relocation of unknown helper %d\n",
val);
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/main.c b/drivers/net/ethernet/netronome/nfp/bpf/main.c
index db6940551b32..a7e217c5204f 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/main.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/main.c
@@ -287,6 +287,9 @@ nfp_bpf_parse_cap_func(struct nfp_app_bpf *bpf, void __iomem *value, u32 length)
case BPF_FUNC_map_update_elem:
bpf->helpers.map_update = readl(&cap->func_addr);
break;
+ case BPF_FUNC_map_delete_elem:
+ bpf->helpers.map_delete = readl(&cap->func_addr);
+ break;
}
return 0;
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/main.h b/drivers/net/ethernet/netronome/nfp/bpf/main.h
index 0246bd88bff3..26bb491224b3 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/main.h
+++ b/drivers/net/ethernet/netronome/nfp/bpf/main.h
@@ -129,6 +129,7 @@ enum pkt_vec {
* @helpers: helper addressess for various calls
* @helpers.map_lookup: map lookup helper address
* @helpers.map_update: map update helper address
+ * @helpers.map_delete: map delete helper address
*/
struct nfp_app_bpf {
struct nfp_app *app;
@@ -164,6 +165,7 @@ struct nfp_app_bpf {
struct {
u32 map_lookup;
u32 map_update;
+ u32 map_delete;
} helpers;
};
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/verifier.c b/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
index 482a0ce6e755..7d67ffc897dd 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
@@ -201,6 +201,14 @@ nfp_bpf_check_call(struct nfp_prog *nfp_prog, struct bpf_verifier_env *env,
!nfp_bpf_stack_arg_ok("map_update", env, reg3, NULL))
return -EOPNOTSUPP;
break;
+
+ case BPF_FUNC_map_delete_elem:
+ if (!nfp_bpf_map_call_ok("map_delete", env, meta,
+ bpf->helpers.map_delete, reg1) ||
+ !nfp_bpf_stack_arg_ok("map_delete", env, reg2,
+ meta->func_id ? &meta->arg2 : NULL))
+ return -EOPNOTSUPP;
+ break;
default:
pr_vlog(env, "unsupported function id: %d\n", func_id);
return -EOPNOTSUPP;
--
2.16.2
^ permalink raw reply related
* [PATCH bpf-next 07/14] nfp: bpf: add map updates from the datapath
From: Jakub Kicinski @ 2018-03-29 0:48 UTC (permalink / raw)
To: alexei.starovoitov, daniel
Cc: netdev, oss-drivers, Jan Gossens, Jakub Kicinski
In-Reply-To: <20180329004839.4506-1-jakub.kicinski@netronome.com>
Support calling map_update_elem() from the datapath programs
by calling into FW-provided helper. Value pointer is passed
in LM pointer #2. Keeping track of old state for arg3 is not
necessary, since LM pointer #2 will be always loaded in this
case, the trivial optimization for value at the bottom of the
stack can't be done here.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com>
Reviewed-by: Jiong Wang <jiong.wang@netronome.com>
---
drivers/net/ethernet/netronome/nfp/bpf/jit.c | 6 ++++++
drivers/net/ethernet/netronome/nfp/bpf/main.c | 3 +++
drivers/net/ethernet/netronome/nfp/bpf/main.h | 2 ++
drivers/net/ethernet/netronome/nfp/bpf/verifier.c | 10 ++++++++++
4 files changed, 21 insertions(+)
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/jit.c b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
index 3799ca9b9826..c63368fc28f6 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/jit.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
@@ -1372,6 +1372,8 @@ map_call_stack_common(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
/* Set LM0 to start of key */
if (load_lm_ptr)
emit_csr_wr(nfp_prog, reg_b(2 * 2), NFP_CSR_ACT_LM_ADDR0);
+ if (meta->func_id == BPF_FUNC_map_update_elem)
+ emit_csr_wr(nfp_prog, reg_b(3 * 2), NFP_CSR_ACT_LM_ADDR2);
/* Load map ID into a register, it should actually fit as an immediate
* but in case it doesn't deal with it here, not in the delay slots.
@@ -2326,6 +2328,7 @@ static int call(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
case BPF_FUNC_xdp_adjust_head:
return adjust_head(nfp_prog, meta);
case BPF_FUNC_map_lookup_elem:
+ case BPF_FUNC_map_update_elem:
return map_call_stack_common(nfp_prog, meta);
default:
WARN_ONCE(1, "verifier allowed unsupported function\n");
@@ -3210,6 +3213,9 @@ void *nfp_bpf_relo_for_vnic(struct nfp_prog *nfp_prog, struct nfp_bpf_vnic *bv)
case BPF_FUNC_map_lookup_elem:
val = nfp_prog->bpf->helpers.map_lookup;
break;
+ case BPF_FUNC_map_update_elem:
+ val = nfp_prog->bpf->helpers.map_update;
+ break;
default:
pr_err("relocation of unknown helper %d\n",
val);
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/main.c b/drivers/net/ethernet/netronome/nfp/bpf/main.c
index 34e98aa6b956..db6940551b32 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/main.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/main.c
@@ -284,6 +284,9 @@ nfp_bpf_parse_cap_func(struct nfp_app_bpf *bpf, void __iomem *value, u32 length)
case BPF_FUNC_map_lookup_elem:
bpf->helpers.map_lookup = readl(&cap->func_addr);
break;
+ case BPF_FUNC_map_update_elem:
+ bpf->helpers.map_update = readl(&cap->func_addr);
+ break;
}
return 0;
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/main.h b/drivers/net/ethernet/netronome/nfp/bpf/main.h
index 0f920bf48b01..0246bd88bff3 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/main.h
+++ b/drivers/net/ethernet/netronome/nfp/bpf/main.h
@@ -128,6 +128,7 @@ enum pkt_vec {
*
* @helpers: helper addressess for various calls
* @helpers.map_lookup: map lookup helper address
+ * @helpers.map_update: map update helper address
*/
struct nfp_app_bpf {
struct nfp_app *app;
@@ -162,6 +163,7 @@ struct nfp_app_bpf {
struct {
u32 map_lookup;
+ u32 map_update;
} helpers;
};
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/verifier.c b/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
index acfc4d798116..482a0ce6e755 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
@@ -167,6 +167,7 @@ nfp_bpf_check_call(struct nfp_prog *nfp_prog, struct bpf_verifier_env *env,
{
const struct bpf_reg_state *reg1 = cur_regs(env) + BPF_REG_1;
const struct bpf_reg_state *reg2 = cur_regs(env) + BPF_REG_2;
+ const struct bpf_reg_state *reg3 = cur_regs(env) + BPF_REG_3;
struct nfp_app_bpf *bpf = nfp_prog->bpf;
u32 func_id = meta->insn.imm;
@@ -191,6 +192,15 @@ nfp_bpf_check_call(struct nfp_prog *nfp_prog, struct bpf_verifier_env *env,
meta->func_id ? &meta->arg2 : NULL))
return -EOPNOTSUPP;
break;
+
+ case BPF_FUNC_map_update_elem:
+ if (!nfp_bpf_map_call_ok("map_update", env, meta,
+ bpf->helpers.map_update, reg1) ||
+ !nfp_bpf_stack_arg_ok("map_update", env, reg2,
+ meta->func_id ? &meta->arg2 : NULL) ||
+ !nfp_bpf_stack_arg_ok("map_update", env, reg3, NULL))
+ return -EOPNOTSUPP;
+ break;
default:
pr_vlog(env, "unsupported function id: %d\n", func_id);
return -EOPNOTSUPP;
--
2.16.2
^ permalink raw reply related
* [PATCH bpf-next 06/14] nfp: bpf: add helper for basic map call checks
From: Jakub Kicinski @ 2018-03-29 0:48 UTC (permalink / raw)
To: alexei.starovoitov, daniel
Cc: netdev, oss-drivers, Jan Gossens, Jakub Kicinski
In-Reply-To: <20180329004839.4506-1-jakub.kicinski@netronome.com>
Add a verifier helper for performing the basic state checks
before a call to a map helper.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com>
Reviewed-by: Jiong Wang <jiong.wang@netronome.com>
---
drivers/net/ethernet/netronome/nfp/bpf/verifier.c | 40 ++++++++++++++---------
1 file changed, 25 insertions(+), 15 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/verifier.c b/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
index 45aa11fcdcad..acfc4d798116 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
@@ -139,6 +139,28 @@ nfp_bpf_stack_arg_ok(const char *fname, struct bpf_verifier_env *env,
return true;
}
+static bool
+nfp_bpf_map_call_ok(const char *fname, struct bpf_verifier_env *env,
+ struct nfp_insn_meta *meta,
+ u32 helper_tgt, const struct bpf_reg_state *reg1)
+{
+ if (!helper_tgt) {
+ pr_vlog(env, "%s: not supported by FW\n", fname);
+ return false;
+ }
+
+ /* Rest of the checks is only if we re-parse the same insn */
+ if (!meta->func_id)
+ return true;
+
+ if (meta->arg1.map_ptr != reg1->map_ptr) {
+ pr_vlog(env, "%s: called for different map\n", fname);
+ return false;
+ }
+
+ return true;
+}
+
static int
nfp_bpf_check_call(struct nfp_prog *nfp_prog, struct bpf_verifier_env *env,
struct nfp_insn_meta *meta)
@@ -163,23 +185,11 @@ nfp_bpf_check_call(struct nfp_prog *nfp_prog, struct bpf_verifier_env *env,
break;
case BPF_FUNC_map_lookup_elem:
- if (!bpf->helpers.map_lookup) {
- pr_vlog(env, "map_lookup: not supported by FW\n");
- return -EOPNOTSUPP;
- }
-
- if (!nfp_bpf_stack_arg_ok("map_lookup", env, reg2,
+ if (!nfp_bpf_map_call_ok("map_lookup", env, meta,
+ bpf->helpers.map_lookup, reg1) ||
+ !nfp_bpf_stack_arg_ok("map_lookup", env, reg2,
meta->func_id ? &meta->arg2 : NULL))
return -EOPNOTSUPP;
-
- /* Rest of the checks is only if we re-parse the same insn */
- if (!meta->func_id)
- break;
-
- if (meta->arg1.map_ptr != reg1->map_ptr) {
- pr_vlog(env, "map_lookup: called for different map\n");
- return -EOPNOTSUPP;
- }
break;
default:
pr_vlog(env, "unsupported function id: %d\n", func_id);
--
2.16.2
^ permalink raw reply related
* [PATCH bpf-next 05/14] nfp: bpf: add helper for validating stack pointers
From: Jakub Kicinski @ 2018-03-29 0:48 UTC (permalink / raw)
To: alexei.starovoitov, daniel
Cc: netdev, oss-drivers, Jan Gossens, Jakub Kicinski
In-Reply-To: <20180329004839.4506-1-jakub.kicinski@netronome.com>
Our implementation has restriction on stack pointers for function
calls. Move the common checks into a helper for reuse. The state
has to be encapsulated into a structure to support parameters
other than BPF_REG_2.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com>
Reviewed-by: Jiong Wang <jiong.wang@netronome.com>
---
drivers/net/ethernet/netronome/nfp/bpf/jit.c | 4 +-
drivers/net/ethernet/netronome/nfp/bpf/main.h | 14 ++++--
drivers/net/ethernet/netronome/nfp/bpf/verifier.c | 59 ++++++++++++++---------
3 files changed, 50 insertions(+), 27 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/jit.c b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
index b37c9f86a18f..3799ca9b9826 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/jit.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
@@ -1366,8 +1366,8 @@ map_call_stack_common(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
/* We only have to reload LM0 if the key is not at start of stack */
lm_off = nfp_prog->stack_depth;
- lm_off += meta->arg2.var_off.value + meta->arg2.off;
- load_lm_ptr = meta->arg2_var_off || lm_off;
+ lm_off += meta->arg2.reg.var_off.value + meta->arg2.reg.off;
+ load_lm_ptr = meta->arg2.var_off || lm_off;
/* Set LM0 to start of key */
if (load_lm_ptr)
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/main.h b/drivers/net/ethernet/netronome/nfp/bpf/main.h
index c220ac5a734a..0f920bf48b01 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/main.h
+++ b/drivers/net/ethernet/netronome/nfp/bpf/main.h
@@ -190,6 +190,16 @@ typedef int (*instr_cb_t)(struct nfp_prog *, struct nfp_insn_meta *);
#define nfp_meta_next(meta) list_next_entry(meta, l)
#define nfp_meta_prev(meta) list_prev_entry(meta, l)
+/**
+ * struct nfp_bpf_reg_state - register state for calls
+ * @reg: BPF register state from latest path
+ * @var_off: for stack arg - changes stack offset on different paths
+ */
+struct nfp_bpf_reg_state {
+ struct bpf_reg_state reg;
+ bool var_off;
+};
+
#define FLAG_INSN_IS_JUMP_DST BIT(0)
/**
@@ -207,7 +217,6 @@ typedef int (*instr_cb_t)(struct nfp_prog *, struct nfp_insn_meta *);
* @func_id: function id for call instructions
* @arg1: arg1 for call instructions
* @arg2: arg2 for call instructions
- * @arg2_var_off: arg2 changes stack offset on different paths
* @off: index of first generated machine instruction (in nfp_prog.prog)
* @n: eBPF instruction number
* @flags: eBPF instruction extra optimization flags
@@ -233,8 +242,7 @@ struct nfp_insn_meta {
struct {
u32 func_id;
struct bpf_reg_state arg1;
- struct bpf_reg_state arg2;
- bool arg2_var_off;
+ struct nfp_bpf_reg_state arg2;
};
};
unsigned int off;
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/verifier.c b/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
index 479f602887e9..45aa11fcdcad 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/verifier.c
@@ -97,7 +97,7 @@ nfp_record_adjust_head(struct nfp_app_bpf *bpf, struct nfp_prog *nfp_prog,
if (nfp_prog->adjust_head_location != meta->n)
goto exit_set_location;
- if (meta->arg2.var_off.value != imm)
+ if (meta->arg2.reg.var_off.value != imm)
goto exit_set_location;
}
@@ -106,6 +106,39 @@ nfp_record_adjust_head(struct nfp_app_bpf *bpf, struct nfp_prog *nfp_prog,
nfp_prog->adjust_head_location = location;
}
+static int
+nfp_bpf_stack_arg_ok(const char *fname, struct bpf_verifier_env *env,
+ const struct bpf_reg_state *reg,
+ struct nfp_bpf_reg_state *old_arg)
+{
+ s64 off, old_off;
+
+ if (reg->type != PTR_TO_STACK) {
+ pr_vlog(env, "%s: unsupported ptr type %d\n",
+ fname, reg->type);
+ return false;
+ }
+ if (!tnum_is_const(reg->var_off)) {
+ pr_vlog(env, "%s: variable pointer\n", fname);
+ return false;
+ }
+
+ off = reg->var_off.value + reg->off;
+ if (-off % 4) {
+ pr_vlog(env, "%s: unaligned stack pointer %lld\n", fname, -off);
+ return false;
+ }
+
+ /* Rest of the checks is only if we re-parse the same insn */
+ if (!old_arg)
+ return true;
+
+ old_off = old_arg->reg.var_off.value + old_arg->reg.off;
+ old_arg->var_off |= off != old_off;
+
+ return true;
+}
+
static int
nfp_bpf_check_call(struct nfp_prog *nfp_prog, struct bpf_verifier_env *env,
struct nfp_insn_meta *meta)
@@ -114,7 +147,6 @@ nfp_bpf_check_call(struct nfp_prog *nfp_prog, struct bpf_verifier_env *env,
const struct bpf_reg_state *reg2 = cur_regs(env) + BPF_REG_2;
struct nfp_app_bpf *bpf = nfp_prog->bpf;
u32 func_id = meta->insn.imm;
- s64 off, old_off;
switch (func_id) {
case BPF_FUNC_xdp_adjust_head:
@@ -135,32 +167,15 @@ nfp_bpf_check_call(struct nfp_prog *nfp_prog, struct bpf_verifier_env *env,
pr_vlog(env, "map_lookup: not supported by FW\n");
return -EOPNOTSUPP;
}
- if (reg2->type != PTR_TO_STACK) {
- pr_vlog(env,
- "map_lookup: unsupported key ptr type %d\n",
- reg2->type);
- return -EOPNOTSUPP;
- }
- if (!tnum_is_const(reg2->var_off)) {
- pr_vlog(env, "map_lookup: variable key pointer\n");
- return -EOPNOTSUPP;
- }
- off = reg2->var_off.value + reg2->off;
- if (-off % 4) {
- pr_vlog(env,
- "map_lookup: unaligned stack pointer %lld\n",
- -off);
+ if (!nfp_bpf_stack_arg_ok("map_lookup", env, reg2,
+ meta->func_id ? &meta->arg2 : NULL))
return -EOPNOTSUPP;
- }
/* Rest of the checks is only if we re-parse the same insn */
if (!meta->func_id)
break;
- old_off = meta->arg2.var_off.value + meta->arg2.off;
- meta->arg2_var_off |= off != old_off;
-
if (meta->arg1.map_ptr != reg1->map_ptr) {
pr_vlog(env, "map_lookup: called for different map\n");
return -EOPNOTSUPP;
@@ -173,7 +188,7 @@ nfp_bpf_check_call(struct nfp_prog *nfp_prog, struct bpf_verifier_env *env,
meta->func_id = func_id;
meta->arg1 = *reg1;
- meta->arg2 = *reg2;
+ meta->arg2.reg = *reg2;
return 0;
}
--
2.16.2
^ permalink raw reply related
* [PATCH bpf-next 04/14] nfp: bpf: rename map_lookup_stack() to map_call_stack_common()
From: Jakub Kicinski @ 2018-03-29 0:48 UTC (permalink / raw)
To: alexei.starovoitov, daniel
Cc: netdev, oss-drivers, Jan Gossens, Jakub Kicinski
In-Reply-To: <20180329004839.4506-1-jakub.kicinski@netronome.com>
We will reuse most of map call code gen for other map calls.
Rename the lookup gen function and use meta->func_id instead
of hard-coding lookup.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com>
Reviewed-by: Jiong Wang <jiong.wang@netronome.com>
---
drivers/net/ethernet/netronome/nfp/bpf/jit.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/jit.c b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
index 0701590e8302..b37c9f86a18f 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/jit.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
@@ -1352,7 +1352,7 @@ static int adjust_head(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
}
static int
-map_lookup_stack(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
+map_call_stack_common(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
{
struct bpf_offloaded_map *offmap;
struct nfp_bpf_map *nfp_map;
@@ -1378,7 +1378,7 @@ map_lookup_stack(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
*/
tid = ur_load_imm_any(nfp_prog, nfp_map->tid, imm_a(nfp_prog));
- emit_br_relo(nfp_prog, BR_UNC, BR_OFF_RELO + BPF_FUNC_map_lookup_elem,
+ emit_br_relo(nfp_prog, BR_UNC, BR_OFF_RELO + meta->func_id,
2, RELO_BR_HELPER);
ret_tgt = nfp_prog_current_offset(nfp_prog) + 2;
@@ -2326,7 +2326,7 @@ static int call(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
case BPF_FUNC_xdp_adjust_head:
return adjust_head(nfp_prog, meta);
case BPF_FUNC_map_lookup_elem:
- return map_lookup_stack(nfp_prog, meta);
+ return map_call_stack_common(nfp_prog, meta);
default:
WARN_ONCE(1, "verifier allowed unsupported function\n");
return -EOPNOTSUPP;
--
2.16.2
^ permalink raw reply related
* [PATCH bpf-next 03/14] nfp: bpf: detect packet reads could be cached, enable the optimisation
From: Jakub Kicinski @ 2018-03-29 0:48 UTC (permalink / raw)
To: alexei.starovoitov, daniel; +Cc: netdev, oss-drivers, Jan Gossens, Jiong Wang
In-Reply-To: <20180329004839.4506-1-jakub.kicinski@netronome.com>
From: Jiong Wang <jiong.wang@netronome.com>
This patch is the front end of this optimisation, it detects and marks
those packet reads that could be cached. Then the optimisation "backend"
will be activated automatically.
Signed-off-by: Jiong Wang <jiong.wang@netronome.com>
Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
drivers/net/ethernet/netronome/nfp/bpf/jit.c | 115 ++++++++++++++++++++++++++
drivers/net/ethernet/netronome/nfp/bpf/main.h | 30 +++++++
2 files changed, 145 insertions(+)
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/jit.c b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
index 38ed2f72a4b0..0701590e8302 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/jit.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
@@ -2964,6 +2964,120 @@ static void nfp_bpf_opt_ldst_gather(struct nfp_prog *nfp_prog)
}
}
+static void nfp_bpf_opt_pkt_cache(struct nfp_prog *nfp_prog)
+{
+ struct nfp_insn_meta *meta, *range_node = NULL;
+ s16 range_start = 0, range_end = 0;
+ bool cache_avail = false;
+ struct bpf_insn *insn;
+ s32 range_ptr_off = 0;
+ u32 range_ptr_id = 0;
+
+ list_for_each_entry(meta, &nfp_prog->insns, l) {
+ if (meta->flags & FLAG_INSN_IS_JUMP_DST)
+ cache_avail = false;
+
+ if (meta->skip)
+ continue;
+
+ insn = &meta->insn;
+
+ if (is_mbpf_store_pkt(meta) ||
+ insn->code == (BPF_JMP | BPF_CALL) ||
+ is_mbpf_classic_store_pkt(meta) ||
+ is_mbpf_classic_load(meta)) {
+ cache_avail = false;
+ continue;
+ }
+
+ if (!is_mbpf_load(meta))
+ continue;
+
+ if (meta->ptr.type != PTR_TO_PACKET || meta->ldst_gather_len) {
+ cache_avail = false;
+ continue;
+ }
+
+ if (!cache_avail) {
+ cache_avail = true;
+ if (range_node)
+ goto end_current_then_start_new;
+ goto start_new;
+ }
+
+ /* Check ID to make sure two reads share the same
+ * variable offset against PTR_TO_PACKET, and check OFF
+ * to make sure they also share the same constant
+ * offset.
+ *
+ * OFFs don't really need to be the same, because they
+ * are the constant offsets against PTR_TO_PACKET, so
+ * for different OFFs, we could canonicalize them to
+ * offsets against original packet pointer. We don't
+ * support this.
+ */
+ if (meta->ptr.id == range_ptr_id &&
+ meta->ptr.off == range_ptr_off) {
+ s16 new_start = range_start;
+ s16 end, off = insn->off;
+ s16 new_end = range_end;
+ bool changed = false;
+
+ if (off < range_start) {
+ new_start = off;
+ changed = true;
+ }
+
+ end = off + BPF_LDST_BYTES(insn);
+ if (end > range_end) {
+ new_end = end;
+ changed = true;
+ }
+
+ if (!changed)
+ continue;
+
+ if (new_end - new_start <= 64) {
+ /* Install new range. */
+ range_start = new_start;
+ range_end = new_end;
+ continue;
+ }
+ }
+
+end_current_then_start_new:
+ range_node->pkt_cache.range_start = range_start;
+ range_node->pkt_cache.range_end = range_end;
+start_new:
+ range_node = meta;
+ range_node->pkt_cache.do_init = true;
+ range_ptr_id = range_node->ptr.id;
+ range_ptr_off = range_node->ptr.off;
+ range_start = insn->off;
+ range_end = insn->off + BPF_LDST_BYTES(insn);
+ }
+
+ if (range_node) {
+ range_node->pkt_cache.range_start = range_start;
+ range_node->pkt_cache.range_end = range_end;
+ }
+
+ list_for_each_entry(meta, &nfp_prog->insns, l) {
+ if (meta->skip)
+ continue;
+
+ if (is_mbpf_load_pkt(meta) && !meta->ldst_gather_len) {
+ if (meta->pkt_cache.do_init) {
+ range_start = meta->pkt_cache.range_start;
+ range_end = meta->pkt_cache.range_end;
+ } else {
+ meta->pkt_cache.range_start = range_start;
+ meta->pkt_cache.range_end = range_end;
+ }
+ }
+ }
+}
+
static int nfp_bpf_optimize(struct nfp_prog *nfp_prog)
{
nfp_bpf_opt_reg_init(nfp_prog);
@@ -2971,6 +3085,7 @@ static int nfp_bpf_optimize(struct nfp_prog *nfp_prog)
nfp_bpf_opt_ld_mask(nfp_prog);
nfp_bpf_opt_ld_shift(nfp_prog);
nfp_bpf_opt_ldst_gather(nfp_prog);
+ nfp_bpf_opt_pkt_cache(nfp_prog);
return 0;
}
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/main.h b/drivers/net/ethernet/netronome/nfp/bpf/main.h
index 861211e27ea6..c220ac5a734a 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/main.h
+++ b/drivers/net/ethernet/netronome/nfp/bpf/main.h
@@ -278,6 +278,36 @@ static inline bool is_mbpf_store(const struct nfp_insn_meta *meta)
return (meta->insn.code & ~BPF_SIZE_MASK) == (BPF_STX | BPF_MEM);
}
+static inline bool is_mbpf_load_pkt(const struct nfp_insn_meta *meta)
+{
+ return is_mbpf_load(meta) && meta->ptr.type == PTR_TO_PACKET;
+}
+
+static inline bool is_mbpf_store_pkt(const struct nfp_insn_meta *meta)
+{
+ return is_mbpf_store(meta) && meta->ptr.type == PTR_TO_PACKET;
+}
+
+static inline bool is_mbpf_classic_load(const struct nfp_insn_meta *meta)
+{
+ u8 code = meta->insn.code;
+
+ return BPF_CLASS(code) == BPF_LD &&
+ (BPF_MODE(code) == BPF_ABS || BPF_MODE(code) == BPF_IND);
+}
+
+static inline bool is_mbpf_classic_store(const struct nfp_insn_meta *meta)
+{
+ u8 code = meta->insn.code;
+
+ return BPF_CLASS(code) == BPF_ST && BPF_MODE(code) == BPF_MEM;
+}
+
+static inline bool is_mbpf_classic_store_pkt(const struct nfp_insn_meta *meta)
+{
+ return is_mbpf_classic_store(meta) && meta->ptr.type == PTR_TO_PACKET;
+}
+
/**
* struct nfp_prog - nfp BPF program
* @bpf: backpointer to the bpf app priv structure
--
2.16.2
^ permalink raw reply related
* [PATCH bpf-next 02/14] nfp: bpf: support unaligned read offset
From: Jakub Kicinski @ 2018-03-29 0:48 UTC (permalink / raw)
To: alexei.starovoitov, daniel; +Cc: netdev, oss-drivers, Jan Gossens, Jiong Wang
In-Reply-To: <20180329004839.4506-1-jakub.kicinski@netronome.com>
From: Jiong Wang <jiong.wang@netronome.com>
This patch add the support for unaligned read offset, i.e. the read offset
to the start of packet cache area is not aligned to REG_WIDTH. In this
case, the read area might across maximum three transfer-in registers.
Signed-off-by: Jiong Wang <jiong.wang@netronome.com>
Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
drivers/net/ethernet/netronome/nfp/bpf/jit.c | 73 ++++++++++++++++++++++++++--
1 file changed, 70 insertions(+), 3 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/jit.c b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
index 0a2c1d87fed2..38ed2f72a4b0 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/jit.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
@@ -553,6 +553,19 @@ wrp_reg_subpart(struct nfp_prog *nfp_prog, swreg dst, swreg src, u8 field_len,
emit_ld_field_any(nfp_prog, dst, mask, src, sc, offset * 8, true);
}
+/* wrp_reg_or_subpart() - load @field_len bytes from low end of @src, or the
+ * result to @dst from offset, there is no change on the other bits of @dst.
+ */
+static void
+wrp_reg_or_subpart(struct nfp_prog *nfp_prog, swreg dst, swreg src,
+ u8 field_len, u8 offset)
+{
+ enum shf_sc sc = offset ? SHF_SC_L_SHF : SHF_SC_NONE;
+ u8 mask = ((1 << field_len) - 1) << offset;
+
+ emit_ld_field(nfp_prog, dst, mask, src, sc, 32 - offset * 8);
+}
+
static void
addr40_offset(struct nfp_prog *nfp_prog, u8 src_gpr, swreg offset,
swreg *rega, swreg *regb)
@@ -1864,6 +1877,60 @@ mem_ldx_data_init_pktcache(struct nfp_prog *nfp_prog,
off, xfer_num - 1, true, indir);
}
+static int
+mem_ldx_data_from_pktcache_unaligned(struct nfp_prog *nfp_prog,
+ struct nfp_insn_meta *meta,
+ unsigned int size)
+{
+ s16 range_start = meta->pkt_cache.range_start;
+ s16 insn_off = meta->insn.off - range_start;
+ swreg dst_lo, dst_hi, src_lo, src_mid;
+ u8 dst_gpr = meta->insn.dst_reg * 2;
+ u8 len_lo = size, len_mid = 0;
+ u8 idx = insn_off / REG_WIDTH;
+ u8 off = insn_off % REG_WIDTH;
+
+ dst_hi = reg_both(dst_gpr + 1);
+ dst_lo = reg_both(dst_gpr);
+ src_lo = reg_xfer(idx);
+
+ /* The read length could involve as many as three registers. */
+ if (size > REG_WIDTH - off) {
+ /* Calculate the part in the second register. */
+ len_lo = REG_WIDTH - off;
+ len_mid = size - len_lo;
+
+ /* Calculate the part in the third register. */
+ if (size > 2 * REG_WIDTH - off)
+ len_mid = REG_WIDTH;
+ }
+
+ wrp_reg_subpart(nfp_prog, dst_lo, src_lo, len_lo, off);
+
+ if (!len_mid) {
+ wrp_immed(nfp_prog, dst_hi, 0);
+ return 0;
+ }
+
+ src_mid = reg_xfer(idx + 1);
+
+ if (size <= REG_WIDTH) {
+ wrp_reg_or_subpart(nfp_prog, dst_lo, src_mid, len_mid, len_lo);
+ wrp_immed(nfp_prog, dst_hi, 0);
+ } else {
+ swreg src_hi = reg_xfer(idx + 2);
+
+ wrp_reg_or_subpart(nfp_prog, dst_lo, src_mid,
+ REG_WIDTH - len_lo, len_lo);
+ wrp_reg_subpart(nfp_prog, dst_hi, src_mid, len_lo,
+ REG_WIDTH - len_lo);
+ wrp_reg_or_subpart(nfp_prog, dst_hi, src_hi, REG_WIDTH - len_lo,
+ len_lo);
+ }
+
+ return 0;
+}
+
static int
mem_ldx_data_from_pktcache_aligned(struct nfp_prog *nfp_prog,
struct nfp_insn_meta *meta,
@@ -1900,10 +1967,10 @@ mem_ldx_data_from_pktcache(struct nfp_prog *nfp_prog,
{
u8 off = meta->insn.off - meta->pkt_cache.range_start;
- if (WARN_ON_ONCE(!IS_ALIGNED(off, REG_WIDTH)))
- return -EOPNOTSUPP;
+ if (IS_ALIGNED(off, REG_WIDTH))
+ return mem_ldx_data_from_pktcache_aligned(nfp_prog, meta, size);
- return mem_ldx_data_from_pktcache_aligned(nfp_prog, meta, size);
+ return mem_ldx_data_from_pktcache_unaligned(nfp_prog, meta, size);
}
static int
--
2.16.2
^ permalink raw reply related
* [PATCH bpf-next 01/14] nfp: bpf: read from packet data cache for PTR_TO_PACKET
From: Jakub Kicinski @ 2018-03-29 0:48 UTC (permalink / raw)
To: alexei.starovoitov, daniel; +Cc: netdev, oss-drivers, Jan Gossens, Jiong Wang
In-Reply-To: <20180329004839.4506-1-jakub.kicinski@netronome.com>
From: Jiong Wang <jiong.wang@netronome.com>
This patch assumes there is a packet data cache, and would try to read
packet data from the cache instead of from memory.
This patch only implements the optimisation "backend", it doesn't build
the packet data cache, so this optimisation is not enabled.
This patch has only enabled aligned packet data read, i.e. when the read
offset to the start of cache is REG_WIDTH aligned.
Signed-off-by: Jiong Wang <jiong.wang@netronome.com>
Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
drivers/net/ethernet/netronome/nfp/bpf/jit.c | 80 ++++++++++++++++++++++++++-
drivers/net/ethernet/netronome/nfp/bpf/main.h | 9 +++
drivers/net/ethernet/netronome/nfp/nfp_asm.h | 1 +
3 files changed, 88 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/jit.c b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
index 56451edf01c2..0a2c1d87fed2 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/jit.c
+++ b/drivers/net/ethernet/netronome/nfp/bpf/jit.c
@@ -1838,6 +1838,74 @@ mem_ldx_emem(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta,
tmp_reg, meta->insn.dst_reg * 2, size);
}
+static void
+mem_ldx_data_init_pktcache(struct nfp_prog *nfp_prog,
+ struct nfp_insn_meta *meta)
+{
+ s16 range_start = meta->pkt_cache.range_start;
+ s16 range_end = meta->pkt_cache.range_end;
+ swreg src_base, off;
+ u8 xfer_num, len;
+ bool indir;
+
+ off = re_load_imm_any(nfp_prog, range_start, imm_b(nfp_prog));
+ src_base = reg_a(meta->insn.src_reg * 2);
+ len = range_end - range_start;
+ xfer_num = round_up(len, REG_WIDTH) / REG_WIDTH;
+
+ indir = len > 8 * REG_WIDTH;
+ /* Setup PREV_ALU for indirect mode. */
+ if (indir)
+ wrp_immed(nfp_prog, reg_none(),
+ CMD_OVE_LEN | FIELD_PREP(CMD_OV_LEN, xfer_num - 1));
+
+ /* Cache memory into transfer-in registers. */
+ emit_cmd_any(nfp_prog, CMD_TGT_READ32_SWAP, CMD_MODE_32b, 0, src_base,
+ off, xfer_num - 1, true, indir);
+}
+
+static int
+mem_ldx_data_from_pktcache_aligned(struct nfp_prog *nfp_prog,
+ struct nfp_insn_meta *meta,
+ unsigned int size)
+{
+ swreg dst_lo, dst_hi, src_lo;
+ u8 dst_gpr, idx;
+
+ idx = (meta->insn.off - meta->pkt_cache.range_start) / REG_WIDTH;
+ dst_gpr = meta->insn.dst_reg * 2;
+ dst_hi = reg_both(dst_gpr + 1);
+ dst_lo = reg_both(dst_gpr);
+ src_lo = reg_xfer(idx);
+
+ if (size < REG_WIDTH) {
+ wrp_reg_subpart(nfp_prog, dst_lo, src_lo, size, 0);
+ wrp_immed(nfp_prog, dst_hi, 0);
+ } else if (size == REG_WIDTH) {
+ wrp_mov(nfp_prog, dst_lo, src_lo);
+ wrp_immed(nfp_prog, dst_hi, 0);
+ } else {
+ swreg src_hi = reg_xfer(idx + 1);
+
+ wrp_mov(nfp_prog, dst_lo, src_lo);
+ wrp_mov(nfp_prog, dst_hi, src_hi);
+ }
+
+ return 0;
+}
+
+static int
+mem_ldx_data_from_pktcache(struct nfp_prog *nfp_prog,
+ struct nfp_insn_meta *meta, unsigned int size)
+{
+ u8 off = meta->insn.off - meta->pkt_cache.range_start;
+
+ if (WARN_ON_ONCE(!IS_ALIGNED(off, REG_WIDTH)))
+ return -EOPNOTSUPP;
+
+ return mem_ldx_data_from_pktcache_aligned(nfp_prog, meta, size);
+}
+
static int
mem_ldx(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta,
unsigned int size)
@@ -1852,8 +1920,16 @@ mem_ldx(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta,
return mem_ldx_skb(nfp_prog, meta, size);
}
- if (meta->ptr.type == PTR_TO_PACKET)
- return mem_ldx_data(nfp_prog, meta, size);
+ if (meta->ptr.type == PTR_TO_PACKET) {
+ if (meta->pkt_cache.range_end) {
+ if (meta->pkt_cache.do_init)
+ mem_ldx_data_init_pktcache(nfp_prog, meta);
+
+ return mem_ldx_data_from_pktcache(nfp_prog, meta, size);
+ } else {
+ return mem_ldx_data(nfp_prog, meta, size);
+ }
+ }
if (meta->ptr.type == PTR_TO_STACK)
return mem_ldx_stack(nfp_prog, meta, size,
diff --git a/drivers/net/ethernet/netronome/nfp/bpf/main.h b/drivers/net/ethernet/netronome/nfp/bpf/main.h
index 054df3dc0698..861211e27ea6 100644
--- a/drivers/net/ethernet/netronome/nfp/bpf/main.h
+++ b/drivers/net/ethernet/netronome/nfp/bpf/main.h
@@ -199,6 +199,10 @@ typedef int (*instr_cb_t)(struct nfp_prog *, struct nfp_insn_meta *);
* @ldst_gather_len: memcpy length gathered from load/store sequence
* @paired_st: the paired store insn at the head of the sequence
* @ptr_not_const: pointer is not always constant
+ * @pkt_cache: packet data cache information
+ * @pkt_cache.range_start: start offset for associated packet data cache
+ * @pkt_cache.range_end: end offset for associated packet data cache
+ * @pkt_cache.do_init: this read needs to initialize packet data cache
* @jmp_dst: destination info for jump instructions
* @func_id: function id for call instructions
* @arg1: arg1 for call instructions
@@ -219,6 +223,11 @@ struct nfp_insn_meta {
struct bpf_insn *paired_st;
s16 ldst_gather_len;
bool ptr_not_const;
+ struct {
+ s16 range_start;
+ s16 range_end;
+ bool do_init;
+ } pkt_cache;
};
struct nfp_insn_meta *jmp_dst;
struct {
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_asm.h b/drivers/net/ethernet/netronome/nfp/nfp_asm.h
index 5f9291db98e0..150d28f9cd52 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_asm.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_asm.h
@@ -39,6 +39,7 @@
#include <linux/types.h>
#define REG_NONE 0
+#define REG_WIDTH 4
#define RE_REG_NO_DST 0x020
#define RE_REG_IMM 0x020
--
2.16.2
^ permalink raw reply related
* [PATCH bpf-next 00/14] nfp: bpf: add updates, deletes, atomic ops, prandom and packet cache
From: Jakub Kicinski @ 2018-03-29 0:48 UTC (permalink / raw)
To: alexei.starovoitov, daniel
Cc: netdev, oss-drivers, Jan Gossens, Jakub Kicinski
Hi!
This set adds support for update and delete calls from the datapath,
as well as XADD instructions (32 and 64 bit) and pseudo random numbers.
The XADD support depends on verifier enforcing alignment which Daniel
recently added. XADD uses NFP's atomic engine which requires values
to be in big endian, therefore we need to keep track of which parts of
the values are used as atomics and byte swap them accordingly. Pseudo
random numbers are generated using NFP's HW pseudo random number
generator.
Jiong tackles initial implementation of packet cache, which he describes
as follows:
Memory reads on NFP would first fetch data from memory to transfer-in
registers, then move them from transfer-in to general registers.
Given NFP is rich on transfer-in registers, they could serve as memory
cache.
This patch tries to identify a sequence of packet data read (BPF_LDX) that
are executed sequentially, then the total access range of the sequence is
calculated and attached to each read instruction, the first instruction
in this sequence is marked with an cache init flag so the execution of
it would bring in the whole range of packet data for the sequence.
All later packet reads in this sequence would fetch data from transfer-in
registers directly, no need to JIT NFP memory access.
Function call, non-packet-data memory read, packet write and memcpy will
invalidate the cache and start a new cache range.
Cache invalidation could be improved in the future, for example packet
write doesn't need to invalidate the cache if the the write destination
won't be read again.
Jakub Kicinski (11):
nfp: bpf: rename map_lookup_stack() to map_call_stack_common()
nfp: bpf: add helper for validating stack pointers
nfp: bpf: add helper for basic map call checks
nfp: bpf: add map updates from the datapath
nfp: bpf: add map deletes from the datapath
bpf: add parenthesis around argument of BPF_LDST_BYTES()
nfp: bpf: add basic support for atomic adds
nfp: bpf: expose command delay slots
nfp: bpf: add support for atomic add of unknown values
nfp: bpf: add support for bpf_get_prandom_u32()
nfp: bpf: improve wrong FW response warnings
Jiong Wang (3):
nfp: bpf: read from packet data cache for PTR_TO_PACKET
nfp: bpf: support unaligned read offset
nfp: bpf: detect packet reads could be cached, enable the optimisation
drivers/net/ethernet/netronome/nfp/bpf/cmsg.c | 12 +-
drivers/net/ethernet/netronome/nfp/bpf/fw.h | 1 +
drivers/net/ethernet/netronome/nfp/bpf/jit.c | 462 ++++++++++++++++++++--
drivers/net/ethernet/netronome/nfp/bpf/main.c | 18 +
drivers/net/ethernet/netronome/nfp/bpf/main.h | 85 +++-
drivers/net/ethernet/netronome/nfp/bpf/offload.c | 45 ++-
drivers/net/ethernet/netronome/nfp/bpf/verifier.c | 217 ++++++++--
drivers/net/ethernet/netronome/nfp/nfp_asm.c | 2 +
drivers/net/ethernet/netronome/nfp/nfp_asm.h | 7 +
include/linux/filter.h | 2 +-
10 files changed, 771 insertions(+), 80 deletions(-)
--
2.16.2
^ permalink raw reply
* Re: [PATCH v8 bpf-next 0/9] bpf, tracing: introduce bpf raw tracepoints
From: Daniel Borkmann @ 2018-03-28 23:59 UTC (permalink / raw)
To: Alexei Starovoitov, davem
Cc: torvalds, peterz, rostedt, mathieu.desnoyers, netdev, kernel-team,
linux-api
In-Reply-To: <20180328190540.370956-1-ast@kernel.org>
On 03/28/2018 09:05 PM, Alexei Starovoitov wrote:
> v7->v8:
> - moved 'u32 num_args' from 'struct tracepoint' into 'struct bpf_raw_event_map'
> that increases memory overhead, but can be optimized/compressed later.
> Now it's zero changes in tracepoint.[ch]
[...]
> The first approach simply added these pointers to sched tracepoints:
> https://lkml.org/lkml/2017/12/14/753
> which Peter nacked.
> Few options were discussed and eventually the discussion converged on
> doing bpf specific tracepoint_probe_register() probe functions.
> Details here:
> https://lkml.org/lkml/2017/12/20/929
>
> Patch 1 is kernel wide cleanup of pass-struct-by-value into
> pass-struct-by-reference into tracepoints.
>
> Patches 2 and 3 are minor cleanups to address allyesconfig build
>
> Patch 4 refactor trace_iwlwifi_dev_ucode_error from 17 to 4 args
>
> Patch 5 introduces COUNT_ARGS macro
>
> Patch 6 introduces BPF_RAW_TRACEPOINT api.
> the auto-cleanup and multiple concurrent users are must have
> features of tracing api. For bpf raw tracepoints it looks like:
> // load bpf prog with BPF_PROG_TYPE_RAW_TRACEPOINT type
> prog_fd = bpf_prog_load(...);
>
> // receive anon_inode fd for given bpf_raw_tracepoint
> // and attach bpf program to it
> raw_tp_fd = bpf_raw_tracepoint_open("xdp_exception", prog_fd);
>
> Ctrl-C of tracing daemon or cmdline tool will automatically
> detach bpf program, unload it and unregister tracepoint probe.
> More details in patch 6.
>
> Patch 7 - trivial support in libbpf
> Patches 8, 9 - user space tests
>
> samples/bpf/test_overhead performance on 1 cpu:
>
> tracepoint base kprobe+bpf tracepoint+bpf raw_tracepoint+bpf
> task_rename 1.1M 769K 947K 1.0M
> urandom_read 789K 697K 750K 755K
Applied to bpf-next, thanks everyone!
^ permalink raw reply
* RE: [Resend Patch 1/3] Vmbus: Add function to report available ring buffer to write in total ring size percentage
From: Stephen Hemminger @ 2018-03-28 23:22 UTC (permalink / raw)
To: Long Li, Martin K. Petersen, Long Li
Cc: KY Srinivasan, Haiyang Zhang, James E . J . Bottomley,
devel@linuxdriverproject.org, linux-scsi@vger.kernel.org,
linux-kernel@vger.kernel.org, netdev@vger.kernel.org
In-Reply-To: <MWHPR2101MB072958DDC25845615C3B1ADCCEA30@MWHPR2101MB0729.namprd21.prod.outlook.com>
Maybe best through greg's char-misc tree since it has generic hv code and sometime updates both network and scsi.
Alternatively, put common code through one tree, and hold off the network device change till next release.
-----Original Message-----
From: Long Li
Sent: Wednesday, March 28, 2018 3:43 PM
To: Martin K. Petersen <martin.petersen@oracle.com>; Long Li <longli@linuxonhyperv.com>
Cc: KY Srinivasan <kys@microsoft.com>; Haiyang Zhang <haiyangz@microsoft.com>; Stephen Hemminger <sthemmin@microsoft.com>; James E . J . Bottomley <JBottomley@odin.com>; devel@linuxdriverproject.org; linux-scsi@vger.kernel.org; linux-kernel@vger.kernel.org; netdev@vger.kernel.org
Subject: RE: [Resend Patch 1/3] Vmbus: Add function to report available ring buffer to write in total ring size percentage
> Subject: Re: [Resend Patch 1/3] Vmbus: Add function to report available ring
> buffer to write in total ring size percentage
>
>
> Long,
>
> > Netvsc has a function to calculate how much ring buffer in percentage
> > is available to write. This function is also useful for storvsc and
> > other vmbus devices.
>
> What is the submission strategy for this series? Do you expect it to go
> through net or scsi? If the latter, I'll need an ack from davem.
Martin,
I hope this patch set goes through SCSI, because it's purpose is to improve storvsc.
If this strategy is not possible, I can resubmit the 1st two patches to net, and the 3rd patch to scsi after the 1st two are merged.
Long
>
> --
> Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply
* [PATCH] atm: iphase: fix spelling mistake: "Receiverd" -> "Received"
From: Colin King @ 2018-03-28 23:18 UTC (permalink / raw)
To: Chas Williams, linux-atm-general, netdev; +Cc: kernel-janitors, linux-kernel
From: Colin Ian King <colin.king@canonical.com>
Trivial fix to spelling mistake in message text
Signed-off-by: Colin Ian King <colin.king@canonical.com>
---
drivers/atm/iphase.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/atm/iphase.c b/drivers/atm/iphase.c
index 98a3a43484c8..44abb8a0a5e5 100644
--- a/drivers/atm/iphase.c
+++ b/drivers/atm/iphase.c
@@ -3147,7 +3147,7 @@ static int ia_proc_read(struct atm_dev *dev,loff_t *pos,char *page)
" Size of Tx Buffer : %u\n"
" Number of Rx Buffer: %u\n"
" Size of Rx Buffer : %u\n"
- " Packets Receiverd : %u\n"
+ " Packets Received : %u\n"
" Packets Transmitted: %u\n"
" Cells Received : %u\n"
" Cells Transmitted : %u\n"
--
2.15.1
^ permalink raw reply related
* Re: WARNING in refcount_dec
From: Cong Wang @ 2018-03-28 23:16 UTC (permalink / raw)
To: Byoungyoung Lee
Cc: LKML, DaeLyong Jeong, Kyungtae Kim,
Linux Kernel Network Developers, Willem de Bruijn
In-Reply-To: <CAG5TwLORDZPmfF7P0S2y9_7PscjvcuQd4WYnZTsBwCi+y47dOQ@mail.gmail.com>
(Cc'ing netdev and Willem)
On Wed, Mar 28, 2018 at 12:03 PM, Byoungyoung Lee
<byoungyoung@purdue.edu> wrote:
> Another crash patterns observed: race between (setsockopt$packet_int)
> and (bind$packet).
>
> ------------------------------
> [ 357.731597] kernel BUG at
> /home/blee/project/race-fuzzer/kernels/kernel_v4.16-rc3/net/packet/af_packet.c:3107!
> [ 357.733382] invalid opcode: 0000 [#1] SMP KASAN
> [ 357.734017] Modules linked in:
> [ 357.734662] CPU: 1 PID: 3871 Comm: repro.exe Not tainted 4.16.0-rc3 #1
> [ 357.735791] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
> [ 357.737434] RIP: 0010:packet_do_bind+0x88d/0x950
> [ 357.738121] RSP: 0018:ffff8800b2787b08 EFLAGS: 00010293
> [ 357.738906] RAX: ffff8800b2fdc780 RBX: ffff880234358cc0 RCX: ffffffff838b244c
> [ 357.739905] RDX: 0000000000000000 RSI: ffffffff838b257d RDI: 0000000000000001
> [ 357.741315] RBP: ffff8800b2787c10 R08: ffff8800b2fdc780 R09: 0000000000000000
> [ 357.743055] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88023352ecc0
> [ 357.744744] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000001d00
> [ 357.746377] FS: 00007f4b43733700(0000) GS:ffff8800b8b00000(0000)
> knlGS:0000000000000000
> [ 357.749599] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 357.752096] CR2: 0000000020058000 CR3: 00000002334b8000 CR4: 00000000000006e0
> [ 357.755045] Call Trace:
> [ 357.755822] ? compat_packet_setsockopt+0x100/0x100
> [ 357.757324] ? __sanitizer_cov_trace_const_cmp8+0x18/0x20
> [ 357.758810] packet_bind+0xa2/0xe0
> [ 357.759640] SYSC_bind+0x279/0x2f0
> [ 357.760364] ? move_addr_to_kernel.part.19+0xc0/0xc0
> [ 357.761491] ? __handle_mm_fault+0x25d0/0x25d0
> [ 357.762449] ? __sanitizer_cov_trace_const_cmp4+0x16/0x20
> [ 357.763663] ? __do_page_fault+0x417/0xba0
> [ 357.764569] ? vmalloc_fault+0x910/0x910
> [ 357.765405] ? __sanitizer_cov_trace_const_cmp4+0x16/0x20
> [ 357.766525] ? mark_held_locks+0x25/0xb0
> [ 357.767336] ? SyS_socketpair+0x4a0/0x4a0
> [ 357.768182] SyS_bind+0x24/0x30
> [ 357.768851] do_syscall_64+0x209/0x5d0
> [ 357.769650] ? syscall_return_slowpath+0x3e0/0x3e0
> [ 357.770665] ? __sanitizer_cov_trace_const_cmp4+0x16/0x20
> [ 357.771779] ? syscall_return_slowpath+0x260/0x3e0
> [ 357.772748] ? mark_held_locks+0x25/0xb0
> [ 357.773581] ? __sanitizer_cov_trace_const_cmp4+0x16/0x20
> [ 357.774720] ? retint_user+0x18/0x18
> [ 357.775493] ? trace_hardirqs_off_caller+0xb5/0x120
> [ 357.776567] ? trace_hardirqs_off_thunk+0x1a/0x1c
> [ 357.777512] entry_SYSCALL_64_after_hwframe+0x42/0xb7
> [ 357.778508] RIP: 0033:0x4503a9
> [ 357.779156] RSP: 002b:00007f4b43732ce8 EFLAGS: 00000246 ORIG_RAX:
> 0000000000000031
> [ 357.780737] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00000000004503a9
> [ 357.782169] RDX: 0000000000000014 RSI: 0000000020058000 RDI: 0000000000000003
> [ 357.783710] RBP: 00007f4b43732d10 R08: 0000000000000000 R09: 0000000000000000
> [ 357.785202] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
> [ 357.786664] R13: 0000000000000000 R14: 00007f4b437339c0 R15: 00007f4b43733700
> [ 357.788210] Code: c0 fd 48 c7 c2 00 c8 d9 84 be ab 02 00 00 48 c7
> c7 60 c8 d9 84 c6 05 e7 a2 48 02 01 e8 3f 17 af fd e9 60 fb ff ff e8
> 43 b3 c0 fd <0f> 0b e8 3c b3 c0 fd 48 8b bd 20 ff ff ff e8 60 1e e7 fd
> 4c 89
> [ 357.792260] RIP: packet_do_bind+0x88d/0x950 RSP: ffff8800b2787b08
> [ 357.793698] ---[ end trace 0c5a2539f0247369 ]---
> [ 357.794696] Kernel panic - not syncing: Fatal exception
> [ 357.795918] Kernel Offset: disabled
> [ 357.796614] Rebooting in 86400 seconds..
>
> On Wed, Mar 28, 2018 at 1:19 AM, Byoungyoung Lee <byoungyoung@purdue.edu> wrote:
>> We report the crash: WARNING in refcount_dec
>>
>> This crash has been found in v4.16-rc3 using RaceFuzzer (a modified
>> version of Syzkaller), which we describe more at the end of this
>> report. Our analysis shows that the race occurs when invoking two
>> syscalls concurrently, (setsockopt$packet_int) and
>> (setsockopt$packet_rx_ring).
>>
>> C repro code : https://kiwi.cs.purdue.edu/static/race-fuzzer/repro-refcount_dec.c
>> kernel config: https://kiwi.cs.purdue.edu/static/race-fuzzer/kernel-config-v4.16-rc3
I tried your reproducer, no luck here.
>>
>> ---------------------------------------
>> [ 305.838560] refcount_t: decrement hit 0; leaking memory.
>> [ 305.839669] WARNING: CPU: 0 PID: 3867 at
>> /home/blee/project/race-fuzzer/kernels/kernel_v4.16-rc3/lib/refcount.c:228
>> refcount_dec+0x62/0x70
>> [ 305.841441] Modules linked in:
>> [ 305.841883] CPU: 0 PID: 3867 Comm: repro.exe Not tainted 4.16.0-rc3 #1
>> [ 305.842803] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
>> BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
>> [ 305.844345] RIP: 0010:refcount_dec+0x62/0x70
>> [ 305.845005] RSP: 0018:ffff880224d374f8 EFLAGS: 00010286
>> [ 305.845802] RAX: 000000000000002c RBX: 0000000000000000 RCX: ffffffff81538991
>> [ 305.846768] RDX: 0000000000000000 RSI: ffffffff813cd761 RDI: 0000000000000005
>> [ 305.847748] RBP: ffff880224d37500 R08: ffff88023169a440 R09: 0000000000000000
>> [ 305.848748] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88023473ad40
>> [ 305.849738] R13: ffff88023473b368 R14: 0000000000000001 R15: 0000000000000000
>> [ 305.850733] FS: 0000000000c6e940(0000) GS:ffff8800b8a00000(0000)
>> knlGS:0000000000000000
>> [ 305.851837] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [ 305.852652] CR2: 00007fb120571db8 CR3: 0000000005422000 CR4: 00000000000006f0
>> [ 305.853619] Call Trace:
>> [ 305.854086] __unregister_prot_hook+0x15f/0x1d0
>> [ 305.854722] packet_release+0x77a/0x7a0
>> [ 305.855335] ? mark_held_locks+0x25/0xb0
>> [ 305.855883] ? packet_lookup_frame+0x110/0x110
>> [ 305.856576] ? __lock_is_held+0x39/0xc0
>> [ 305.857109] ? note_gp_changes+0x300/0x300
>> [ 305.857745] ? __sanitizer_cov_trace_const_cmp8+0x18/0x20
>> [ 305.858548] ? locks_remove_file+0x31b/0x420
>> [ 305.859138] ? fcntl_setlk+0xad0/0xad0
>> [ 305.859743] ? trace_event_raw_event_sched_switch+0x320/0x320
>> [ 305.860534] ? fsnotify_first_mark+0x2c0/0x2c0
>> [ 305.861234] sock_release+0x53/0xe0
>> [ 305.861711] ? sock_alloc_file+0x2c0/0x2c0
>> [ 305.862334] sock_close+0x16/0x20
>> [ 305.862801] __fput+0x246/0x4e0
>> [ 305.863310] ? fput+0x130/0x130
>> [ 305.863743] ? trace_event_raw_event_sched_switch+0x320/0x320
>> [ 305.864604] ____fput+0x15/0x20
>> [ 305.865046] task_work_run+0x1a5/0x200
>> [ 305.865636] ? kmem_cache_free+0x25c/0x2d0
>> [ 305.866194] ? task_work_cancel+0x1a0/0x1a0
>> [ 305.866829] ? switch_task_namespaces+0x9e/0xb0
>> [ 305.867458] do_exit+0xacf/0x10d0
>> [ 305.868023] ? mm_update_next_owner+0x650/0x650
>> [ 305.868642] ? __pv_queued_spin_lock_slowpath+0xbf0/0xbf0
>> [ 305.869427] ? trace_hardirqs_on_caller+0x136/0x2a0
>> [ 305.870102] ? trace_hardirqs_on+0xd/0x10
>> [ 305.870701] ? wake_up_new_task+0x41c/0x650
>> [ 305.871324] ? to_ratio+0x20/0x20
>> [ 305.871816] ? lock_release+0x530/0x530
>> [ 305.872341] ? trace_event_raw_event_sched_switch+0x320/0x320
>> [ 305.873161] ? match_held_lock+0x7e/0x360
>> [ 305.873777] ? trace_hardirqs_off+0x10/0x10
>> [ 305.874359] ? put_pid+0x111/0x140
>> [ 305.874897] ? task_active_pid_ns+0x70/0x70
>> [ 305.875500] ? find_held_lock+0xca/0xf0
>> [ 305.876118] ? do_group_exit+0x1f9/0x260
>> [ 305.876650] ? lock_downgrade+0x380/0x380
>> [ 305.877297] ? task_clear_jobctl_pending+0xb5/0xd0
>> [ 305.877951] ? __sanitizer_cov_trace_const_cmp4+0x16/0x20
>> [ 305.878725] ? do_raw_spin_unlock+0x112/0x1a0
>> [ 305.879309] ? do_raw_spin_trylock+0x100/0x100
>> [ 305.879969] ? mark_held_locks+0x25/0xb0
>> [ 305.880505] ? force_sig+0x30/0x30
>> [ 305.881054] ? _raw_spin_unlock_irq+0x27/0x50
>> [ 305.881671] ? trace_hardirqs_on_caller+0x136/0x2a0
>> [ 305.882412] do_group_exit+0xfb/0x260
>> [ 305.882945] ? SyS_exit+0x30/0x30
>> [ 305.883442] ? find_mergeable_anon_vma+0x90/0x90
>> [ 305.884103] ? SyS_read+0x1c0/0x1c0
>> [ 305.884785] ? mark_held_locks+0x25/0xb0
>> [ 305.885503] ? do_syscall_64+0xb2/0x5d0
>> [ 305.886093] ? do_group_exit+0x260/0x260
>> [ 305.886741] SyS_exit_group+0x1d/0x20
>> [ 305.887255] do_syscall_64+0x209/0x5d0
>> [ 305.887888] ? syscall_return_slowpath+0x3e0/0x3e0
>> [ 305.888611] ? __sanitizer_cov_trace_const_cmp4+0x16/0x20
>> [ 305.889420] ? syscall_return_slowpath+0x260/0x3e0
>> [ 305.890188] ? mark_held_locks+0x25/0xb0
>> [ 305.890724] ? entry_SYSCALL_64_after_hwframe+0x52/0xb7
>> [ 305.891556] ? trace_hardirqs_off_caller+0xb5/0x120
>> [ 305.892265] ? trace_hardirqs_off_thunk+0x1a/0x1c
>> [ 305.892939] entry_SYSCALL_64_after_hwframe+0x42/0xb7
>> [ 305.893676] RIP: 0033:0x44d989
>> [ 305.894100] RSP: 002b:00000000007fff38 EFLAGS: 00000202 ORIG_RAX:
>> 00000000000000e7
>> [ 305.895158] RAX: ffffffffffffffda RBX: 00000000004002c8 RCX: 000000000044d989
>> [ 305.896174] RDX: 00007fb120d739c0 RSI: 00007fb120572700 RDI: 0000000000000001
>> [ 305.897161] RBP: 00000000007fff60 R08: 00007fb120572700 R09: 0000000000000000
>> [ 305.898128] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
>> [ 305.899464] R13: 000000000040d270 R14: 000000000040d300 R15: 0000000000000000
>> [ 305.900823] Code: b6 1d 81 9a ef 03 31 ff 89 de e8 ca a3 67 ff 84
>> db 75 df e8 f1 a2 67 ff 48 c7 c7 60 8f 83 84 c6 05 61 9a ef 03 01 e8
>> ee 5f 49 ff <0f> 0b eb c3 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41
>> 57 49
>> [ 305.904324] ---[ end trace 360c084b02d93021 ]---
>> [ 305.919117] ------------[ cut here ]------------
>> [ 305.920120] refcount_t: underflow; use-after-free.
>> [ 305.921335] WARNING: CPU: 0 PID: 3867 at
>> /home/blee/project/race-fuzzer/kernels/kernel_v4.16-rc3/lib/refcount.c:187
>> refcount_sub_and_test+0x1ec/0x200
>> [ 305.923927] Modules linked in:
>> [ 305.924611] CPU: 0 PID: 3867 Comm: repro.exe Tainted: G W
>> 4.16.0-rc3 #1
>> [ 305.925987] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
>> BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
>> [ 305.928119] RIP: 0010:refcount_sub_and_test+0x1ec/0x200
>> [ 305.929124] RSP: 0018:ffff880224d374a0 EFLAGS: 00010282
>> [ 305.930161] RAX: 0000000000000026 RBX: 0000000000000000 RCX: ffffffff813c9644
>> [ 305.931504] RDX: 0000000000000000 RSI: ffffffff813cd761 RDI: ffff880224d37018
>> [ 305.932942] RBP: ffff880224d37538 R08: ffff88023169a440 R09: 0000000000000000
>> [ 305.934365] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000ffffffff
>> [ 305.935734] R13: ffff88023473adc0 R14: 0000000000000001 R15: 1ffff100449a6e96
>> [ 305.937114] FS: 0000000000c6e940(0000) GS:ffff8800b8a00000(0000)
>> knlGS:0000000000000000
>> [ 305.938668] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [ 305.939768] CR2: 00007fb120571db8 CR3: 0000000005422000 CR4: 00000000000006f0
>> [ 305.941212] Call Trace:
>> [ 305.941689] ? refcount_inc+0x70/0x70
>> [ 305.942216] ? skb_dequeue+0xa5/0xc0
>> [ 305.942713] refcount_dec_and_test+0x1a/0x20
>> [ 305.943295] packet_release+0x702/0x7a0
>> [ 305.943816] ? mark_held_locks+0x25/0xb0
>> [ 305.944378] ? packet_lookup_frame+0x110/0x110
>> [ 305.945021] ? __lock_is_held+0x39/0xc0
>> [ 305.945561] ? note_gp_changes+0x300/0x300
>> [ 305.946132] ? __sanitizer_cov_trace_const_cmp8+0x18/0x20
>> [ 305.946866] ? locks_remove_file+0x31b/0x420
>> [ 305.947464] ? fcntl_setlk+0xad0/0xad0
>> [ 305.948000] ? trace_event_raw_event_sched_switch+0x320/0x320
>> [ 305.948781] ? fsnotify_first_mark+0x2c0/0x2c0
>> [ 305.949386] sock_release+0x53/0xe0
>> [ 305.949866] ? sock_alloc_file+0x2c0/0x2c0
>> [ 305.950437] sock_close+0x16/0x20
>> [ 305.950906] __fput+0x246/0x4e0
>> [ 305.951360] ? fput+0x130/0x130
>> [ 305.951807] ? trace_event_raw_event_sched_switch+0x320/0x320
>> [ 305.952620] ____fput+0x15/0x20
>> [ 305.953071] task_work_run+0x1a5/0x200
>> [ 305.953585] ? kmem_cache_free+0x25c/0x2d0
>> [ 305.954143] ? task_work_cancel+0x1a0/0x1a0
>> [ 305.954714] ? switch_task_namespaces+0x9e/0xb0
>> [ 305.955334] do_exit+0xacf/0x10d0
>> [ 305.955801] ? mm_update_next_owner+0x650/0x650
>> [ 305.956431] ? __pv_queued_spin_lock_slowpath+0xbf0/0xbf0
>> [ 305.957157] ? trace_hardirqs_on_caller+0x136/0x2a0
>> [ 305.957811] ? trace_hardirqs_on+0xd/0x10
>> [ 305.958360] ? wake_up_new_task+0x41c/0x650
>> [ 305.958937] ? to_ratio+0x20/0x20
>> [ 305.959391] ? lock_release+0x530/0x530
>> [ 305.959924] ? trace_event_raw_event_sched_switch+0x320/0x320
>> [ 305.960693] ? match_held_lock+0x7e/0x360
>> [ 305.961244] ? trace_hardirqs_off+0x10/0x10
>> [ 305.961810] ? put_pid+0x111/0x140
>> [ 305.962277] ? task_active_pid_ns+0x70/0x70
>> [ 305.962862] ? find_held_lock+0xca/0xf0
>> [ 305.963396] ? do_group_exit+0x1f9/0x260
>> [ 305.963933] ? lock_downgrade+0x380/0x380
>> [ 305.964508] ? task_clear_jobctl_pending+0xb5/0xd0
>> [ 305.965147] ? __sanitizer_cov_trace_const_cmp4+0x16/0x20
>> [ 305.965871] ? do_raw_spin_unlock+0x112/0x1a0
>> [ 305.966459] ? do_raw_spin_trylock+0x100/0x100
>> [ 305.967060] ? mark_held_locks+0x25/0xb0
>> [ 305.967592] ? force_sig+0x30/0x30
>> [ 305.968135] ? _raw_spin_unlock_irq+0x27/0x50
>> [ 305.968741] ? trace_hardirqs_on_caller+0x136/0x2a0
>> [ 305.969470] do_group_exit+0xfb/0x260
>> [ 305.969987] ? SyS_exit+0x30/0x30
>> [ 305.970505] ? find_mergeable_anon_vma+0x90/0x90
>> [ 305.971126] ? SyS_read+0x1c0/0x1c0
>> [ 305.971718] ? mark_held_locks+0x25/0xb0
>> [ 305.972259] ? do_syscall_64+0xb2/0x5d0
>> [ 305.972843] ? do_group_exit+0x260/0x260
>> [ 305.973374] SyS_exit_group+0x1d/0x20
>> [ 305.973932] do_syscall_64+0x209/0x5d0
>> [ 305.974452] ? syscall_return_slowpath+0x3e0/0x3e0
>> [ 305.975149] ? __sanitizer_cov_trace_const_cmp4+0x16/0x20
>> [ 305.975941] ? syscall_return_slowpath+0x260/0x3e0
>> [ 305.976669] ? mark_held_locks+0x25/0xb0
>> [ 305.977206] ? entry_SYSCALL_64_after_hwframe+0x52/0xb7
>> [ 305.977978] ? trace_hardirqs_off_caller+0xb5/0x120
>> [ 305.978690] ? trace_hardirqs_off_thunk+0x1a/0x1c
>> [ 305.979381] entry_SYSCALL_64_after_hwframe+0x42/0xb7
>> [ 305.980114] RIP: 0033:0x44d989
>> [ 305.980531] RSP: 002b:00000000007fff38 EFLAGS: 00000202 ORIG_RAX:
>> 00000000000000e7
>> [ 305.981664] RAX: ffffffffffffffda RBX: 00000000004002c8 RCX: 000000000044d989
>> [ 305.982655] RDX: 00007fb120d739c0 RSI: 00007fb120572700 RDI: 0000000000000001
>> [ 305.983654] RBP: 00000000007fff60 R08: 00007fb120572700 R09: 0000000000000000
>> [ 305.984656] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
>> [ 305.985707] R13: 000000000040d270 R14: 000000000040d300 R15: 0000000000000000
>> [ 305.986724] Code: b6 1d 18 9b ef 03 31 ff 89 de e8 60 a4 67 ff 84
>> db 75 1a e8 87 a3 67 ff 48 c7 c7 00 8f 83 84 c6 05 f8 9a ef 03 01 e8
>> 84 60 49 ff <0f> 0b 31 db e9 2b ff ff ff 66 66 2e 0f 1f 84 00 00 00 00
>> 00 55
>> [ 305.990106] ---[ end trace 360c084b02d93022 ]---
>> [ 305.998636] IPVS: ftp: loaded support on port[0] = 21
>> ---------------------------------------
>>
>> = About RaceFuzzer
>>
>> RaceFuzzer is a customized version of Syzkaller, specifically tailored
>> to find race condition bugs in the Linux kernel. While we leverage
>> many different technique, the notable feature of RaceFuzzer is in
>> leveraging a custom hypervisor (QEMU/KVM) to interleave the
>> scheduling. In particular, we modified the hypervisor to intentionally
>> stall a per-core execution, which is similar to supporting per-core
>> breakpoint functionality. This allows RaceFuzzer to force the kernel
>> to deterministically trigger racy condition (which may rarely happen
>> in practice due to randomness in scheduling).
>>
>> RaceFuzzer's C repro always pinpoints two racy syscalls. Since C
>> repro's scheduling synchronization should be performed at the user
>> space, its reproducibility is limited (reproduction may take from 1
>> second to 10 minutes (or even more), depending on a bug). This is
>> because, while RaceFuzzer precisely interleaves the scheduling at the
>> kernel's instruction level when finding this bug, C repro cannot fully
>> utilize such a feature. Please disregard all code related to
>> "should_hypercall" in the C repro, as this is only for our debugging
>> purposes using our own hypervisor.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox